{"slug": "show-hn-rl-environment-for-sheep-hearding", "title": "Show HN: RL Environment for Sheep Hearding", "summary": "A developer released SheepdogHerding-v0, a reinforcement-learning environment built on the Strömbom et al. (2014) shepherding model, where an AI agent controls a dog to herd sheep into a pen. The environment, available as a Python package with Gymnasium integration, supports vector and pixel observations and is designed for training RL algorithms like PPO and SAC.", "body_md": "A reinforcement-learning re-creation of the browser game\n[ Shepherd's Dog](https://vnglst.github.io/when-ai-fails/shepards-dog/claude-fable-5/index.html).\n\nYou control a sheepdog with limited speed. Moving the pointer steers the dog;\nclicking makes it **bark**, which sends nearby sheep bolting away. The flock\nbehaves as one — it clumps, flees the dog, and avoids bushes and rocks. The pen\nis a **fenced box with a single opening of fixed width**: the flock can only get\nin through the gap, so it has to be funnelled, not just shoved at a wall. Your\njob is to herd at least **80% of the flock into the pen before nightfall**.\nOptional **wolves** appear at dusk and pick off stray sheep.\n\nThe sheep follow the **Strömbom et al. (2014)** shepherding model — the standard\nmathematical model for this exact behaviour (cohesion + separation +\nflee-from-dog + inertia + noise). Every weight is exposed and documented in\n`sheepdog_env/flocking.py`\n\nso you can retune it to match a specific game build.\n\nThe core environment is a self-contained Python package (`sheepdog_env`\n\n) with\nonly **numpy** and **gymnasium** as dependencies. Install it as a module:\n\n```\ngit clone https://github.com/midhunharikumar/sheepdog-rl.git\ncd sheepdog-rl\npip install -e .                  # core env only: numpy + gymnasium\n```\n\nOptional extras pull in what the demo/training scripts need:\n\n```\npip install -e \".[render]\"        # pygame + imageio + pillow (live window / gif export)\npip install -e \".[train]\"         # stable-baselines3 + wandb (RL training + tracking)\npip install -e \".[test]\"          # pytest\npip install -e \".[all]\"           # everything above\n```\n\nYou can also install straight from a checkout for use as a library in another\nproject (`pip install /path/to/sheepdog_rl`\n\n), or list it as a git dependency.\n\n``` python\nfrom sheepdog_env import SheepdogHerdingEnv\n\nenv = SheepdogHerdingEnv(obs_mode=\"vector\")   # or \"pixel\"\nobs, info = env.reset(seed=0)\nfor _ in range(1000):\n    action = env.action_space.sample()        # [move_x, move_y, bark]\n    obs, reward, terminated, truncated, info = env.step(action)\n    if terminated or truncated:\n        break\n```\n\nIt is also registered with Gymnasium: `gym.make(\"SheepdogHerding-v0\")`\n\nafter\n`import sheepdog_env`\n\n.\n\nInterpretation is set by `action_mode`\n\n(default `\"polar\"`\n\n):\n\n** \"polar\"** — velocity, orientation, bark:\n\n| Index | Meaning |\n|---|---|\n`a[0]` |\nspeed, mapped `[-1,1] → [0, dog_max_speed]` |\n`a[1]` |\nheading / orientation, mapped `[-1,1] → [-π, π]` |\n`a[2]` |\nbark when `> 0` (subject to a cooldown) |\n\n** \"cartesian\"** —\n\n`a[0], a[1]`\n\nare a velocity vector (magnitude clamped to the\ndog's max speed) and `a[2]`\n\nis bark.**Bark** scares nearby sheep into bolting **radially outward from the dog's\nposition**, faster and harder than normal (the classic sheepdog behaviour). So\nthe dog steers the flock by *where it stands* — get behind the flock relative to\nthe gap and bark to drive them toward it. Barking has a cooldown, so spamming\ndoes nothing extra. (Set `FlockParams.bark_directional=True`\n\nto instead drive\nsheep along the dog's heading — easier to funnel, less realistic.) A `Box`\n\naction keeps the env compatible with continuous algorithms (PPO, SAC, TD3).\n\nSet per env with `obs_mode`\n\n:\n\n(default, fast to train): a flat, normalized`\"vector\"`\n\n`Box`\n\ncontaining the dog's position/velocity and bark-ready flag, the pen rectangle, the**opening (gap) location and width**, time- and penned-fractions, every sheep's position (relative to the dog), velocity and status (free / penned / lost), and the obstacles. Dimension for the defaults (40 sheep, 5 obstacles) is**229**.: an`\"pixel\"`\n\n`(84, 84, 3)`\n\n`uint8`\n\ntop-down render — use a CNN policy.\n\nBoth share identical dynamics, so you can train on vectors and evaluate on pixels (or vice-versa).\n\nA single, dense, easy-to-reason-about reward:\n\n**+ pen reward**—`w_pen_enter`\n\n(default 10) each time a sheep enters the pen.**+ final**— at episode end,`w_final`\n\n× (fraction of the flock penned), so \"more sheep home at the end\" is directly maximized.**− entrance**— per step,`w_entrance`\n\n× the mean distance of*un-penned*sheep to the**entrance**(the gap). This is the dense gradient that pulls the flock toward the opening from step one.**− back**— per step,`w_back`\n\n× the fraction of un-penned sheep driven*behind*the pen (the far side from the entrance), discouraging herding the flock around the pen instead of through the gap.**− cohesion**— per step,`w_cohesion`\n\n× how spread out the free flock is (mean distance of un-penned sheep to their centroid). Rewards keeping the herd together — important because the radial bark tends to fan the flock out.\n\nWith the defaults a stalling agent scores ≈ −180, a clean win ≈ +400, and\npartial penning is positively rewarded in between — a smooth, dense gradient\ntoward penning with no \"park and stall\" local optimum. All weights live in\n`EnvConfig`\n\n. The CEM planner (below) confirms this reward is well-aligned:\nmaximizing it pens 90% of the flock.\n\n**Success**(`terminated`\n\n): ≥`target_fraction`\n\nof the flock penned.**Nightfall**(`truncated`\n\n):`max_steps`\n\nreached — one full \"day\".**Lost**(`terminated`\n\n): too few sheep remain to ever reach the target (only possible with wolves enabled).\n\nPass an `EnvConfig`\n\n, or override individual fields as keywords:\n\n``` python\nfrom sheepdog_env import SheepdogHerdingEnv, EnvConfig\n\nenv = SheepdogHerdingEnv(\n    n_sheep=60,\n    dog_max_speed=2.0,\n    enable_wolves=True,\n    target_fraction=0.8,\n    obs_mode=\"pixel\",\n)\n```\n\nNotable knobs (see `env.py`\n\n/ `flocking.py`\n\nfor the full list and defaults):\n`width`\n\n, `height`\n\n, `n_sheep`\n\n, `target_fraction`\n\n, `max_steps`\n\n, `action_mode`\n\n(`polar`\n\n/`cartesian`\n\n), `dog_max_speed`\n\n, `bark_cooldown`\n\n, `bark_duration`\n\n,\n`pen_frac`\n\n, `pen_opening_side`\n\n(`left`\n\n/`right`\n\n/`top`\n\n/`bottom`\n\n), `pen_opening_center`\n\n, `pen_opening_width`\n\n,\n`n_bushes`\n\n, `n_rocks`\n\n, `enable_wolves`\n\n, `dusk_fraction`\n\n, `n_wolves`\n\n, the reward\nweights (`w_pen_enter`\n\n, `w_final`\n\n, `w_entrance`\n\n, `w_back`\n\n, `w_cohesion`\n\n), and the whole\n`FlockParams`\n\nblock (sheep speed `delta`\n\n, cohesion `c`\n\n, separation `rho_a`\n\n, dog\nrepulsion `rho_s`\n\n, detection radius `r_s`\n\n— lower lets the dog get closer before\nthe flock bolts — fence repulsion `rho_w`\n\n, and the bark amplification\n`bark_radius`\n\n/ `bark_speed_mult`\n\n/ `bark_impulse`\n\n, etc.).\n\n``` php\npython examples/demo.py                  # scripted shepherd -> demo.gif\npython examples/demo.py --random         # random policy\npython examples/demo.py --human          # live pygame window\npython examples/demo.py --wolves         # enable dusk wolves\npython examples/heuristic_agent.py       # print a scripted-agent rollout\npython examples/cem_solver.py --gif cem.gif   # CEM planner: solves the task by planning\npython examples/rollouts_gif.py          # 2x2 montage of rollouts -> rollouts.gif\npython examples/train_sb3.py --timesteps 1000000   # PPO (vector obs), logs to W&B\npython examples/eval_sb3.py --model ppo_sheepdog --episodes 50   # evaluate a trained model\npython examples/play_sb3.py --model ppo_sheepdog   # live pygame viewer of a policy\npython examples/play_sb3.py --heuristic            # live viewer, no model needed\npython examples/play_sb3.py --mouse                # play it yourself with the mouse\npython -m pytest tests/ -q               # tests\n```\n\nTraining logs to **Weights & Biases**. Run `wandb login`\n\nonce (or set\n`WANDB_API_KEY`\n\n); metrics, config and periodic model checkpoints sync to the\n`sheepdog-rl`\n\nproject. Use `--wandb-project NAME`\n\n/ `--wandb-entity TEAM`\n\nto\nchange the destination, `--wandb-mode offline`\n\nto log locally and sync later, or\n`--no-wandb`\n\nto disable tracking entirely.\n\nTraining wraps the env in ** VecNormalize** (normalizes rewards, and\nobservations for the vector env). The normalization stats are saved next to the\nmodel as\n\n`<save>_vecnorm.pkl`\n\n; `eval_sb3.py`\n\n, `play_sb3.py`\n\nand\n`inspect_policy.py`\n\nauto-detect and apply them (override with `--vecnorm PATH`\n\n).\nIf you train your own loop, remember the policy expects **normalized** observations at inference.\n\nBecause the environment is its own fast simulator, you can **plan** instead of\n(or before) learning. The Cross-Entropy Method optimizes an open-loop action\nsequence by *cloning* the env and rolling candidate paths forward to score them\n— evaluating paths before committing — then refining the sampling distribution\ntoward the best (\"elite\") ones. Actions are piecewise-constant segments to keep\nthe search low-dimensional, and all candidates in an iteration share the same\ncloned state and RNG (common random numbers) for low-variance comparison.\n\n```\npython examples/cem_solver.py --seed 0 --gif cem.gif\n```\n\nIt's a useful planning baseline / oracle, a sanity check that the (dense, simple)\nreward is well-aligned, and a source of expert trajectories for imitation warmup.\n(With the *directional* bark it pens ~90% in under 100 steps; with the default\n*radial* bark — which fans the flock out — funneling through the narrow gap is\nmuch harder, so give it more samples/iterations.)\n\nBootstrap RL past the hard exploration: collect expert rollouts (from CEM or the\nfast heuristic), keep only the good ones (`--min-penned`\n\n), behavior-clone an SB3\npolicy on them, then **fine-tune with PPO**:\n\n```\npython examples/cem_to_bc.py --source heuristic --seeds 0..15 --min-penned 20 --save bc_sheepdog\npython examples/train_sb3.py --init-from bc_sheepdog        # PPO continues from the BC policy\n```\n\nBC alone won't herd well (covariate shift: small errors compound over a long\nepisode), so it's intended as a *warm start* — `--init-from`\n\ncopies the BC policy\nweights and observation-normalization stats into PPO, which then fixes the drift\nby exploring around the demonstrated behaviour. Demo quality is the bottleneck:\nthe cloned policy is only as good as the rollouts you feed it.\n\n`examples/heuristic_agent.py`\n\nis a classic collect-and-drive shepherd — a\n*non-trivial baseline, not an expert policy*. It pens a portion of the flock but\ndoes **not** reliably hit the 80% target: herding 40 sheep through a single\nnarrow opening while the dog is fenced out of the pen is a hard control problem\n(a naive point-shepherd tends to fragment the flock against the field edges).\nThat difficulty is the point — it's what makes this a worthwhile RL benchmark\nrather than something a few lines of geometry can solve.\n\n```\nsheepdog_rl/\n├── sheepdog_env/            # the installable module\n│   ├── __init__.py          # exports + Gymnasium registration\n│   ├── env.py               # SheepdogHerdingEnv, EnvConfig\n│   ├── flocking.py          # Strömbom flock dynamics + FlockParams\n│   ├── geometry.py          # pen fence: build walls, repulsion, crossing collision\n│   ├── rendering.py         # NumPy renderer (+ optional pygame window)\n│   └── vec_env.py           # BatchedSheepdogVecEnv (fast SB3 VecEnv, optional)\n├── examples/                # demo, heuristic agent, CEM planner, SB3 training/eval\n├── tests/                   # API conformance + vec-env parity tests\n├── assets/                  # curated demo GIFs used by this README\n├── ppo_sheepdog.zip         # pretrained PPO policy (+ _vecnorm.pkl) for eval/play\n├── pyproject.toml           # package metadata + extras\n├── requirements.txt\n└── LICENSE\n```\n\nThe dynamics are faithful to the genre and the observed behavior, but the\nnumeric constants are tuned approximations, not byte-for-byte copies of the\noriginal JavaScript (its source couldn't be auto-extracted in this session). If\nyou paste the game's JS — or run it with the browser dev tools open — the\nmapping is direct: dog speed, bark radius/strength, the day length, sheep count\nand the flock weights all have one-to-one counterparts in `EnvConfig`\n\n/\n`FlockParams`\n\n.\n\nStrömbom, D. et al. (2014). *Solving the shepherding problem: heuristics for\nherding autonomous, interacting agents.* Journal of the Royal Society Interface.", "url": "https://wpnews.pro/news/show-hn-rl-environment-for-sheep-hearding", "canonical_source": "https://github.com/midhunharikumar/sheepdog-rl", "published_at": "2026-07-18 16:45:55+00:00", "updated_at": "2026-07-18 16:51:06.752361+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools"], "entities": ["SheepdogHerding-v0", "Gymnasium", "Strömbom et al. (2014)", "numpy", "stable-baselines3", "wandb", "pygame", "imageio"], "alternates": {"html": "https://wpnews.pro/news/show-hn-rl-environment-for-sheep-hearding", "markdown": "https://wpnews.pro/news/show-hn-rl-environment-for-sheep-hearding.md", "text": "https://wpnews.pro/news/show-hn-rl-environment-for-sheep-hearding.txt", "jsonld": "https://wpnews.pro/news/show-hn-rl-environment-for-sheep-hearding.jsonld"}}