Show HN: RL Environment for Sheep Hearding

A developer released SheepdogHerding-v0, a reinforcement-learning environment built on the Strömbom et al. (2014) shepherding model, where an AI agent controls a dog to herd sheep into a pen. The environment, available as a Python package with Gymnasium integration, supports vector and pixel observations and is designed for training RL algorithms like PPO and SAC.

A reinforcement-learning re-creation of the browser game Shepherd's Dog https://vnglst.github.io/when-ai-fails/shepards-dog/claude-fable-5/index.html . You control a sheepdog with limited speed. Moving the pointer steers the dog; clicking makes it bark , which sends nearby sheep bolting away. The flock behaves as one — it clumps, flees the dog, and avoids bushes and rocks. The pen is a fenced box with a single opening of fixed width : the flock can only get in through the gap, so it has to be funnelled, not just shoved at a wall. Your job is to herd at least 80% of the flock into the pen before nightfall . Optional wolves appear at dusk and pick off stray sheep. The sheep follow the Strömbom et al. 2014 shepherding model — the standard mathematical model for this exact behaviour cohesion + separation + flee-from-dog + inertia + noise . Every weight is exposed and documented in sheepdog env/flocking.py so you can retune it to match a specific game build. The core environment is a self-contained Python package sheepdog env with only numpy and gymnasium as dependencies. Install it as a module: git clone https://github.com/midhunharikumar/sheepdog-rl.git cd sheepdog-rl pip install -e . core env only: numpy + gymnasium Optional extras pull in what the demo/training scripts need: pip install -e ". render " pygame + imageio + pillow live window / gif export pip install -e ". train " stable-baselines3 + wandb RL training + tracking pip install -e ". test " pytest pip install -e ". all " everything above You can also install straight from a checkout for use as a library in another project pip install /path/to/sheepdog rl , or list it as a git dependency. python from sheepdog env import SheepdogHerdingEnv env = SheepdogHerdingEnv obs mode="vector" or "pixel" obs, info = env.reset seed=0 for in range 1000 : action = env.action space.sample move x, move y, bark obs, reward, terminated, truncated, info = env.step action if terminated or truncated: break It is also registered with Gymnasium: gym.make "SheepdogHerding-v0" after import sheepdog env . Interpretation is set by action mode default "polar" : "polar" — velocity, orientation, bark: | Index | Meaning | |---|---| a 0 | speed, mapped -1,1 → 0, dog max speed | a 1 | heading / orientation, mapped -1,1 → -π, π | a 2 | bark when 0 subject to a cooldown | "cartesian" — a 0 , a 1 are a velocity vector magnitude clamped to the dog's max speed and a 2 is bark. Bark scares nearby sheep into bolting radially outward from the dog's position , faster and harder than normal the classic sheepdog behaviour . So the dog steers the flock by where it stands — get behind the flock relative to the gap and bark to drive them toward it. Barking has a cooldown, so spamming does nothing extra. Set FlockParams.bark directional=True to instead drive sheep along the dog's heading — easier to funnel, less realistic. A Box action keeps the env compatible with continuous algorithms PPO, SAC, TD3 . Set per env with obs mode : default, fast to train : a flat, normalized "vector" Box containing the dog's position/velocity and bark-ready flag, the pen rectangle, the opening gap location and width , time- and penned-fractions, every sheep's position relative to the dog , velocity and status free / penned / lost , and the obstacles. Dimension for the defaults 40 sheep, 5 obstacles is 229 .: an "pixel" 84, 84, 3 uint8 top-down render — use a CNN policy. Both share identical dynamics, so you can train on vectors and evaluate on pixels or vice-versa . A single, dense, easy-to-reason-about reward: + pen reward — w pen enter default 10 each time a sheep enters the pen. + final — at episode end, w final × fraction of the flock penned , so "more sheep home at the end" is directly maximized. − entrance — per step, w entrance × the mean distance of un-penned sheep to the entrance the gap . This is the dense gradient that pulls the flock toward the opening from step one. − back — per step, w back × the fraction of un-penned sheep driven behind the pen the far side from the entrance , discouraging herding the flock around the pen instead of through the gap. − cohesion — per step, w cohesion × how spread out the free flock is mean distance of un-penned sheep to their centroid . Rewards keeping the herd together — important because the radial bark tends to fan the flock out. With the defaults a stalling agent scores ≈ −180, a clean win ≈ +400, and partial penning is positively rewarded in between — a smooth, dense gradient toward penning with no "park and stall" local optimum. All weights live in EnvConfig . The CEM planner below confirms this reward is well-aligned: maximizing it pens 90% of the flock. Success terminated : ≥ target fraction of the flock penned. Nightfall truncated : max steps reached — one full "day". Lost terminated : too few sheep remain to ever reach the target only possible with wolves enabled . Pass an EnvConfig , or override individual fields as keywords: python from sheepdog env import SheepdogHerdingEnv, EnvConfig env = SheepdogHerdingEnv n sheep=60, dog max speed=2.0, enable wolves=True, target fraction=0.8, obs mode="pixel", Notable knobs see env.py / flocking.py for the full list and defaults : width , height , n sheep , target fraction , max steps , action mode polar / cartesian , dog max speed , bark cooldown , bark duration , pen frac , pen opening side left / right / top / bottom , pen opening center , pen opening width , n bushes , n rocks , enable wolves , dusk fraction , n wolves , the reward weights w pen enter , w final , w entrance , w back , w cohesion , and the whole FlockParams block sheep speed delta , cohesion c , separation rho a , dog repulsion rho s , detection radius r s — lower lets the dog get closer before the flock bolts — fence repulsion rho w , and the bark amplification bark radius / bark speed mult / bark impulse , etc. . php python examples/demo.py scripted shepherd - demo.gif python examples/demo.py --random random policy python examples/demo.py --human live pygame window python examples/demo.py --wolves enable dusk wolves python examples/heuristic agent.py print a scripted-agent rollout python examples/cem solver.py --gif cem.gif CEM planner: solves the task by planning python examples/rollouts gif.py 2x2 montage of rollouts - rollouts.gif python examples/train sb3.py --timesteps 1000000 PPO vector obs , logs to W&B python examples/eval sb3.py --model ppo sheepdog --episodes 50 evaluate a trained model python examples/play sb3.py --model ppo sheepdog live pygame viewer of a policy python examples/play sb3.py --heuristic live viewer, no model needed python examples/play sb3.py --mouse play it yourself with the mouse python -m pytest tests/ -q tests Training logs to Weights & Biases . Run wandb login once or set WANDB API KEY ; metrics, config and periodic model checkpoints sync to the sheepdog-rl project. Use --wandb-project NAME / --wandb-entity TEAM to change the destination, --wandb-mode offline to log locally and sync later, or --no-wandb to disable tracking entirely. Training wraps the env in VecNormalize normalizes rewards, and observations for the vector env . The normalization stats are saved next to the model as <save vecnorm.pkl ; eval sb3.py , play sb3.py and inspect policy.py auto-detect and apply them override with --vecnorm PATH . If you train your own loop, remember the policy expects normalized observations at inference. Because the environment is its own fast simulator, you can plan instead of or before learning. The Cross-Entropy Method optimizes an open-loop action sequence by cloning the env and rolling candidate paths forward to score them — evaluating paths before committing — then refining the sampling distribution toward the best "elite" ones. Actions are piecewise-constant segments to keep the search low-dimensional, and all candidates in an iteration share the same cloned state and RNG common random numbers for low-variance comparison. python examples/cem solver.py --seed 0 --gif cem.gif It's a useful planning baseline / oracle, a sanity check that the dense, simple reward is well-aligned, and a source of expert trajectories for imitation warmup. With the directional bark it pens ~90% in under 100 steps; with the default radial bark — which fans the flock out — funneling through the narrow gap is much harder, so give it more samples/iterations. Bootstrap RL past the hard exploration: collect expert rollouts from CEM or the fast heuristic , keep only the good ones --min-penned , behavior-clone an SB3 policy on them, then fine-tune with PPO : python examples/cem to bc.py --source heuristic --seeds 0..15 --min-penned 20 --save bc sheepdog python examples/train sb3.py --init-from bc sheepdog PPO continues from the BC policy BC alone won't herd well covariate shift: small errors compound over a long episode , so it's intended as a warm start — --init-from copies the BC policy weights and observation-normalization stats into PPO, which then fixes the drift by exploring around the demonstrated behaviour. Demo quality is the bottleneck: the cloned policy is only as good as the rollouts you feed it. examples/heuristic agent.py is a classic collect-and-drive shepherd — a non-trivial baseline, not an expert policy . It pens a portion of the flock but does not reliably hit the 80% target: herding 40 sheep through a single narrow opening while the dog is fenced out of the pen is a hard control problem a naive point-shepherd tends to fragment the flock against the field edges . That difficulty is the point — it's what makes this a worthwhile RL benchmark rather than something a few lines of geometry can solve. sheepdog rl/ ├── sheepdog env/ the installable module │ ├── init .py exports + Gymnasium registration │ ├── env.py SheepdogHerdingEnv, EnvConfig │ ├── flocking.py Strömbom flock dynamics + FlockParams │ ├── geometry.py pen fence: build walls, repulsion, crossing collision │ ├── rendering.py NumPy renderer + optional pygame window │ └── vec env.py BatchedSheepdogVecEnv fast SB3 VecEnv, optional ├── examples/ demo, heuristic agent, CEM planner, SB3 training/eval ├── tests/ API conformance + vec-env parity tests ├── assets/ curated demo GIFs used by this README ├── ppo sheepdog.zip pretrained PPO policy + vecnorm.pkl for eval/play ├── pyproject.toml package metadata + extras ├── requirements.txt └── LICENSE The dynamics are faithful to the genre and the observed behavior, but the numeric constants are tuned approximations, not byte-for-byte copies of the original JavaScript its source couldn't be auto-extracted in this session . If you paste the game's JS — or run it with the browser dev tools open — the mapping is direct: dog speed, bark radius/strength, the day length, sheep count and the flock weights all have one-to-one counterparts in EnvConfig / FlockParams . Strömbom, D. et al. 2014 . Solving the shepherding problem: heuristics for herding autonomous, interacting agents. Journal of the Royal Society Interface.