Symbolic Learning in Python and Julia

PySR, an open-source symbolic regression tool developed alongside the Julia library SymbolicRegression.jl, is installable via pip or conda and now offers a coding-agent skill installed with the command "npx skills add astroautomata/PySR." PySR's algorithms are documented in arXiv paper 2305.01582, and the tool targets interpretable symbolic expressions for low-dimensional datasets, with symbolic distillation of neural networks (arXiv 2006.11287) extending it to higher-dimensional problems such as N-body equations. The package exposes a scikit-learn-style PySRRegressor interface and ships Docker and Apptainer build files for containerized or cluster use.

pip install pysr If you find PySR useful, please cite the paper arXiv:2305.01582 https://arxiv.org/abs/2305.01582 . If you've finished a project with PySR, please submit a PR to showcase your work on the research showcase page https://pysr.ai/papers Contents : PySR is an open-source tool for Symbolic Regression : a machine learning task where the goal is to find an interpretable symbolic expression that optimizes some objective. Over a period of several years, PySR has been engineered from the ground up to be 1 as high-performance as possible, 2 as configurable as possible, and 3 easy to use. PySR is developed alongside the Julia library SymbolicRegression.jl https://github.com/astroautomata/SymbolicRegression.jl , which forms the powerful search engine of PySR. The details of these algorithms are described in the PySR paper https://arxiv.org/abs/2305.01582 . Symbolic regression works best on low-dimensional datasets, but one can also extend these approaches to higher-dimensional spaces by using " Symbolic Distillation " of Neural Networks, as explained in 2006.11287 https://arxiv.org/abs/2006.11287 , where we apply it to N-body problems. Here, one essentially uses symbolic regression to convert a neural net to an analytic equation. Thus, these tools simultaneously present an explicit and powerful way to interpret deep neural networks. You can install PySR with pip or conda: pip install pysr conda install -c conda-forge pysr Julia dependencies will be installed at first import. Install the PySR skill https://github.com/astroautomata/PySR/blob/master/skills/pysr/SKILL.md to provide your coding agents guidance using PySR as part of a larger agentic loop: npx skills add astroautomata/PySR Add --global to install the skill globally. You can also use the Dockerfile to install PySR in a docker container 1. Clone this repo. 2. Within the repo's directory, build the docker container: docker build -t pysr . 1. You can then start the container with an IPython execution with: docker run -it --rm pysr ipython For more details, see the docker section docker . If you are using PySR on a cluster where you do not have root access, you can use Apptainer https://apptainer.org/ to build a container instead of Docker. The Apptainer.def file is analogous to the Dockerfile , and can be built with: apptainer build --notest pysr.sif Apptainer.def and launched with apptainer run pysr.sif One issue you might run into can result in a hard crash at import with a message like " GLIBCXX ... not found". This is due to another one of the Python dependencies loading an incorrect libstdc++ library. To fix this, you should modify your LD LIBRARY PATH variable to reference the Julia libraries. For example, if the Julia version of libstdc++.so is located in $HOME/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/lib/julia/ which likely differs on your system , you could add: export LD LIBRARY PATH=$HOME/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/lib/julia/:$LD LIBRARY PATH to your .bashrc or .zshrc file. You might wish to try the interactive tutorial here https://colab.research.google.com/github/astroautomata/PySR/blob/master/examples/pysr demo.ipynb , which uses the notebook in examples/pysr demo.ipynb . In practice, I highly recommend using IPython rather than Jupyter, as the printing is much nicer. Below is a quick demo here which you can paste into a Python runtime. First, let's import numpy to generate some test data: python import numpy as np X = 2 np.random.randn 100, 5 y = 2.5382 np.cos X :, 3 + X :, 0 2 - 0.5 We have created a dataset with 100 datapoints, with 5 features each. The relation we wish to model is Now, let's create a PySR model and train it. PySR's main interface is in the style of scikit-learn: python from pysr import PySRRegressor model = PySRRegressor maxsize=20, niterations=40, < Increase me for better results binary operators= "+", " " , unary operators= "cos", "exp", "sin", "inv x = 1/x", ^ Custom operator julia syntax , extra sympy mappings={"inv": lambda x: 1 / x}, ^ Define operator for SymPy as well elementwise loss="loss prediction, target = prediction - target ^2", ^ Custom loss function julia syntax This will set up the model for 40 iterations of the search code, which contains hundreds of thousands of mutations and equation evaluations. Let's train this model on our dataset: model.fit X, y Internally, this launches a Julia process which will do a multithreaded search for equations to fit the dataset. Equations will be printed during training, and once you are satisfied, you may quit early by hitting 'q' and then <enter . After the model has been fit, you can run model.predict X to see the predictions on a given dataset using the automatically-selected expression, or, for example, model.predict X, 3 to see the predictions of the 3rd equation. You may run: print model to print the learned equations: PySRRegressor.equations = pick score equation loss complexity 0 0.000000 4.4324794 42.354317 1 1 1.255691 x0 x0 3.437307 3 2 0.011629 x0 x0 + -0.28087974 3.358285 5 3 0.897855 x0 x0 + cos x3 1.368308 6 4 0.857018 x0 x0 + cos x3 2.4566472 0.246483 8 5 inf cos x3 + -0.19699033 2.5382123 + x0 ... 0.000000 10 This arrow in the pick column indicates which equation is currently selected by your model selection strategy for prediction. You may change model selection after .fit X, y as well. model.equations is a pandas DataFrame containing all equations, including callable format lambda format , SymPy format sympy format - which you can also get with model.sympy , and even JAX and PyTorch format both of which are differentiable - which you can get with model.jax and model.pytorch . Note that PySRRegressor stores the state of the last search, and will restart from where you left off the next time you call .fit , assuming you have set warm start=True . This will cause problems if significant changes are made to the search parameters like changing the operators . You can run model.reset to reset the state. You will notice that PySR will save two files: hall of fame...csv and hall of fame...pkl . The csv file is a list of equations and their losses, and the pkl file is a saved state of the model. You may load the model from the pkl file with: model = PySRRegressor.from file "hall of fame.2022-08-10 100832.281.pkl" There are several other useful features such as denoising e.g., denoise=True , feature selection e.g., select k features=3 . For examples of these and other features, see the examples page https://pysr.ai/examples . For a detailed look at more options, see the options page https://pysr.ai/options . You can also see the full API at this page https://pysr.ai/api . There are also tips for tuning PySR on this page https://pysr.ai/tuning . The following code makes use of as many PySR features as possible. Note that is just a demonstration of features and you should not use this example as-is. For details on what each parameter does, check out the API page https://pysr.ai/api . model = PySRRegressor populations=8, ^ Assuming we have 4 cores, this means 2 populations per core, so one is always running. population size=50, ^ Slightly larger populations, for greater diversity. ncycles per iteration=500, ^ Generations between migrations. niterations=10000000, Run forever early stop condition= "stop if loss, complexity = loss < 1e-6 && complexity < 10" Stop early if we find a good and simple equation , timeout in seconds=60 60 24, ^ Alternatively, stop after 24 hours have passed. maxsize=50, ^ Allow greater complexity. maxdepth=10, ^ But, avoid deep nesting. binary operators= " ", "+", "-", "/" , unary operators= "square", "cube", "exp", "cos2 x =cos x ^2" , constraints={ "/": -1, 9 , "square": 9, "cube": 9, "exp": 9, }, ^ Limit the complexity within each argument. "inv": -1, 9 states that the numerator has no constraint, but the denominator has a max complexity of 9. "exp": 9 simply states that exp can only have an expression of complexity 9 as input. nested constraints={ "square": {"square": 1, "cube": 1, "exp": 0}, "cube": {"square": 1, "cube": 1, "exp": 0}, "exp": {"square": 1, "cube": 1, "exp": 0}, }, ^ Nesting constraints on operators. For example, "square exp x " is not allowed, since "square": {"exp": 0}. complexity of operators={"/": 2, "exp": 3}, ^ Custom complexity of particular operators. complexity of constants=2, ^ Punish constants more than variables select k features=4, ^ Train on only the 4 most important features progress=True, ^ Can set to false if printing to a file. weight randomize=0.1, ^ Randomize the tree much more frequently cluster manager=None, ^ Can be set to, e.g., "slurm", to run a slurm cluster. Just launch one script from the head node. precision=64, ^ Higher precision calculations. warm start=True, ^ Start from where left off. turbo=True, ^ Faster evaluation experimental extra sympy mappings={"cos2": lambda x: sympy.cos x 2}, extra torch mappings={sympy.cos: torch.cos}, ^ Not needed as cos already defined, but this is how you define custom torch operators. extra jax mappings={sympy.cos: "jnp.cos"}, ^ For JAX, one passes a string. You can also test out PySR in Docker, without installing it locally, by running the following command in the root directory of this repo: docker build -t pysr . This builds an image called pysr for your system's architecture, which also contains IPython. You can select a specific version of Python and Julia with: docker build -t pysr --build-arg JLVERSION=1.10.0 --build-arg PYVERSION=3.11.6 . You can then run with this dockerfile using: docker run -it --rm -v "$PWD:/data" pysr ipython which will link the current directory to the container's /data directory and then launch ipython. If you have issues building for your system's architecture, you can emulate another architecture by including --platform linux/amd64 , before the build and run commands. We are eager to welcome new contributors Check out our contributors guide https://github.com/astroautomata/PySR/blob/master/CONTRIBUTORS.md for tips 🚀. If you have an idea for a new feature, don't hesitate to share it on the issues https://github.com/astroautomata/PySR/issues or discussions https://github.com/astroautomata/PySR/discussions page.