cd /news/ai-agents/skill-for-ai-coding-agents-set-up-th… · home › topics › ai-agents › article
[ARTICLE · art-173513] src=gist.github.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Skill for AI coding agents: set up the environment, baselines and evals for an ML research paper (such as ICML), then run auto research inside it

A developer published an open-source skill, autonomous-ml-research, that instructs AI coding agents to run an ML research project end to end — from literature search through environment setup, baselines and evals, to a reproducible repo — with the method insisting agents build and validate the test harness before trying any designs. The skill requires agents to debug every component in roughly 7-second runs, tune baselines as hard as new designs, log costs and mistakes, and disclose when no human has reviewed the work; a worked example is hosted at github.com/vukrosic/llm-memory-env, which the author notes has also not been human-reviewed.

key takeaway

The skill's method requires agents to build a kill test and tune baselines as hard as new designs before running experiments, and to debug every code path in ~7-second runs so long runs don't surface bugs a short test could catch.

by read10 min views5 publishedNov 3, 2026
| name | autonomous-ml-research | 
|---|---|

| description | Run a machine learning research project on a topic the person gives you, from the first search to a repo a stranger can reproduce. Use it when someone gives you an ML research topic and asks you to research it, test ideas or prepare a conference paper such as ICML. |

The person gives you a topic. You do the work: find an open question, build a test that can reject ideas, prove that it can, then try designs under rules written before the results.

The order is the whole method: do not start by inventing designs. First set up the environment, baselines and evals, act as an ICML reviewer on them, and fix issues until none are left. Only then run auto research inside that environment.

This method produces evidence and a reproducible repo. It does not make a paper acceptable. A person must read and check the work before it is submitted anywhere.

A worked example with every script and result is here: https://github.com/vukrosic/llm-memory-env. Script names below come from it. No person has reviewed that example either.

  • Work one step at a time. Check each result before the next step.
  • Say what each run will cost before you start it. Ask the person before you spend money, rent a GPU or publish anything.
  • Report in plain sentences with numbers. If you don't know yet, say so in one sentence.
  • Say who did the work. If an AI agent did it and no person has checked it, write that in the repo.
  • Keep a log file. Write down every mistake and what it cost, and read the log before each new stage.

Debug every part in runs of about 7 seconds before you run anything longer: setup, models, baselines, tests, scoring and result files.

  • Build a tiny size first: a small model, a few steps, a few rows of data. Run it through the real scripts, so every code path runs. Its scores mean nothing. It only shows that things work.
  • Stop each debug step at 7 seconds in the code. If it takes longer, shrink it.
  • Before any longer run, ask what a few-second test could still catch, and write that test. A long run that finds a bug a short test could have found is a mistake, so log it.
  • After any code change, rerun the few-second tests before anything longer.

Move on only when a stage's check passes. When a check fails, fix that stage and rerun what depends on it.

  1. Search at least 3 different ways, then read the closest paper in full.
  2. Write the boring explanation of the effect you expect. Word frequency, text length or a simple name lookup may explain it.
  3. Write the one test whose failure ends the idea, with the number that counts as failure. Run it at a small size.
  4. Judge the idea by the venue's criteria. ICML asks about novelty, rigor, significance and presentation. Check the current reviewer form.

Check: the closest paper and the one-sentence difference from it are written down, and the kill test passed.

Build the test as a tool that any design plugs into, so a dead design costs one row of results and not a whole paper.

  • One script, several sizes. Name each size by rough cost, such as 5s, 1m, 5m, 1h and 5h. Fix the steps in each size, so every method reads the same data.
  • Published baselines inside. Port each one from its paper. Compare it with the authors' code when that exists. Write what you changed in the docstring. Print the number of weights per module.
  • Baselines tuned to their maximum. Tune every baseline at least as hard as any new design will be tuned, with the same search over settings and the same training. Agents tend to under-tune the state of the art, so a new design looks like a win when it only beat a weak baseline on simple tasks.
  • Many kinds of score, with no combined number. Use the general metric, tasks aimed at your claim, hard splits and real benchmarks, so a method can't win by being good at one thing.
  • Scores per item. Save the score of every item, so any two methods can be compared with a paired standard error.
  • Hidden items from the start. Keep part of every test out of normal runs, behind a flag such as--final .
  • Reproducible output. Append results as JSON lines with the settings. Let long jobs stop and resume.
  • Debug scripts that answer one question each. Keep them in the repo.

Check: a run of about 7 seconds per step goes end to end through the real job script, with every score present.

Do this with published methods, before any new design runs. If a design shaped the test, the test can't judge it.

Check Question
Bug check Can a method see the future? Does every weight get a gradient? Are test rows in the training data? Is the answer inside the question? ( check.py )
Repeat Do two runs of one setting give the same scores?
Published rankings Does the test reproduce published comparisons? Find the reason for each mismatch with a targeted run. ( claims.py )
Broken control Does a deliberately broken method get credit? It should not.
Cheating control Does plain memorizing get credit? If so, never read that score alone.
Score audit Does each score move during training, separate methods, and agree across settings and hidden items? Set the cut-offs before you read the table. ( audit.py )
Benchmarks Does each benchmark move on the plain model at this budget? If it stays at chance, it can't judge anything here.

After any fix to a test, rerun every baseline and keep the old rows in a separate file. Numbers from before the fix don't compare.

Then review everything as an ICML reviewer, and repeat until no issues are left:

  1. Read the environment, baselines, evals and results so far, and look for every issue a reviewer would raise: bugs, hacks, tests too simple, baselines not tuned to their maximum, unfair comparisons and missing evals.
  2. Fix each issue and rerun the few-second tests. Leave longer experiments out of this loop, so no long run is wasted on a bug a short test could have found.
  3. Review again from the start. Stop only when a full pass finds no new issue, and write down each pass and what it found.

Check: a full reviewer pass finds no new issue, and you can write one paragraph on what the test can judge and what it can't. Publish that paragraph. No new design runs before this check passes.

Put the rule in code (beat.py), with the sizes it needs, the z cut-off and the limits on size. Commit it before the first design runs.

  • Decide whether compute is capped as well as weights.
  • Write one fairness sentence for every comparison: "X and Y have the same N weights, because ___."
  • Count every design you try, and report the count.
  • If a design only gets close, the result is "no win". Report the score it lost.

For each round:

  1. Write the prediction in the log and commit it before running. Include numbers and the result that would drop the idea.
  2. Change one idea, and take it from the last result. Combining published ideas counts as one idea and is worth trying, because agents are better at combining than at inventing, and people have not tried many combinations.
  3. Iterate at the smallest size that predicts the next one, then check at that next size. Try each leading method at two or more learning rates.
  4. Write the result beside the prediction. Keep failed rounds in the log.
  5. Explain before you redesign. Switch each part off at test time and read the scores. Predictions often name the wrong part.

Before you credit a new part:

  • Add controls built only from the published parts it sits beside. Part of the gain often comes from those.
  • Measure speed one run at a time on one thread, because speeds from parallel runs can be far off.
  • Test "adding A helps" directly, with and without A at equal size.

Before any run over 10 minutes, check that it can resume, that it fits in time and disk, and that a short run of the same script passed.

  1. Run the baselines and leading designs at the largest size you can finish.
  2. Measure rank agreement between sizes on every score, using published methods.
  3. Report which leads grew and which faded. A large lead at a small size can almost vanish at a larger one.
  4. Make the largest finished size the main result, and say what has never run.

Score the final design on hidden items once, at the end. Say which designs never touched them, and whether anything was chosen after reading them.

Read every claim as a hostile reviewer would.

Claim Check
"Beats X" Name the size, training, scores won and scores lost. Say so if X is your own port.
"Best" Is the z above 2? If not, write "ties".
"No method does Y" Test it against every row of your table and every paper you found.
"New" Say which parts are published and what alone is new. Three searches don't settle it, so say that too.
"Adding A helps" Does a run compare with and without A at equal size?
A number in a post Is the model it belongs to named beside it?
"We built" Say who did the work and who reviewed it.

List the limits too: training runs per setting, untested sizes, compute cost, unread papers and missing controls.

Check: every limit is answered by a run or stated in the write-up.

Keep every script, result row, job list and debug script in the repo, and keep the log with its failed predictions. Reproduce the setup from a clean copy. Scan the repo for personal paths, emails and keys before any push.

Write these files:

  • README.md : the question, one table, one chart and links.
  • RESULTS.md : every number, with the command that made it.
  • PAPER-IDEA.md : the paper directions, what is done, and what is left before submitting.

Answer each with a number or a run.

  1. Is this general, or does it only fit this one method?
  2. Why this structure? Which assumption am I making, and could it be wrong?
  3. Are these tests too simple? What can a method do well that no score sees?
  4. Before I spend compute, is the environment ready, and how do I know?
  5. Am I measuring new ideas properly, or building ahead of the test?
  6. Is the comparison fair on weights and on compute?
  7. Could this small size make the whole result invalid?
  8. What exactly loses, and do I understand why?
  9. Did I invent this, or take it from somewhere?
  10. Can a reader tell which number belongs to what?
── more in #ai-agents 4 stories · sorted by recency
── more on @autonomous-ml-research 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/skill-for-ai-coding-…] indexed:0 read:10min 2026-11-03 · —