Most AI agents stop improving the moment you stop improving them.
Not because the models stop getting better.
Because the new workflows, architectures, repos, tools, and engineering tricks never reach your agent on their own.
They sit inside thousands of GitHub repositories. Somewhere, there is already a workflow your agent has never seen. Finding it is still usually a human job.
You open the repository. You read the README. You check the examples. You decide whether the idea is actually useful. Then you manually teach the agent.
That loop never made sense to me.
If an AI can write code, review pull requests, call tools, and follow structured workflows, why should every new capability still depend on a human manually browsing GitHub?
So I built a system that does the boring part automatically.
It scans GitHub, finds promising AI repositories, extracts reusable workflows, converts them into Agent Skills, reviews them, and opens a pull request for human approval.
The model does not retrain itself. It does something more practical. It expands the skill library around itself.
That is the difference between a static agent and a learning system.
The problem: AI agents stop improving when humans stop updating them.
The solution: An 8-stage pipeline that:
The key insight: Automation proposes. Humans approve.
**Time to build: **~2–3 weeks if you follow along.
Now let’s go deep.
This guide is for AI builders, automation engineers, backend developers, technical founders, and anyone building agents that need to improve over time.
This is for you if:
This is not a beginner guide to prompting. This is a systems design guide for turning public software knowledge into reusable agent capabilities.
The goal is not to make an agent that magically becomes smarter. The goal is to build a pipeline that continuously finds useful workflows and converts them into a format the agent can reuse.
That distinction matters.
Models learn during training. Agents improve through context, tools, memory, workflows, and skill libraries.
This system does not update the model weights. It updates the operating system around the model.
The loop looks like this:
Discover repositories→
Nothing is merged automatically. Automation proposes. Humans approve.
That is the safety boundary that makes the system usable.
At first glance, this looks like one pipeline. It is not. It is a team of small agents, each with one job.
That separation is the point.
One giant agent trying to discover, read, judge, generate, and publish everything will eventually become unpredictable. It will mix responsibilities. It will over-read code. It will approve its own bad work.
Small agents are easier to reason about. Each stage receives structured input, performs one job, and passes structured output to the next stage.
The system has eight stages:
Every agent has one job. That is what makes the whole system reliable.
Every learning system starts with one question: Where do new ideas come from?
For this system, the answer is GitHub.
The Scout agent continuously searches for repositories that may contain useful AI workflows, agent architectures, MCP servers, automation patterns, evaluation harnesses, or tool-use examples.
The Scout does not read repositories. It does not judge quality. It does not ask an LLM whether something is good. Its job is only discovery.
A typical search might look like this:
"agent framework" OR langgraph OR
The output is not prose. It is a queue.
{ "name": "awesome-mcp-server", "url": "https://github.com/example/awesome-mcp-server", "stars": 842, "language": "Python", "last_updated": "2026-07-09T14:32:00Z", "status": "QUEUED"}
The Scout should be boring. That is a feature. If discovery is noisy, every downstream stage gets expensive.
Finding repositories is easy. Finding useful repositories is not.
Most GitHub repositories are irrelevant to your agent. Some are UI libraries. Some are starter templates. Some are experiments with no reusable workflow inside. Some are abandoned.
If you send all of them into an LLM, you are not building intelligence. You are burning tokens.
The Filter removes obvious noise before the expensive stages begin. This is where deterministic rules are better than model judgment.
A kept repository might look like this:
{ "repository": "awesome-mcp-server", "category": "AI Agent", "language": "Python", "decision": "KEEP", "reason": "Contains reusable agent workflows"}
A rejected repository looks the same:
{ "repository": "css-animation-library", "category": "CSS", "language": "TypeScript", "decision": "REJECT", "reason": "Outside AI workflow scope"}
The Filter is not trying to be perfect. It is trying to remove the obvious waste. That one design choice makes the rest of the system cheaper, faster, and easier to trust.
The Reader is one of the most important agents in the system.
Most repository analyzers make the same mistake. They jump straight into source code. That is usually the slowest way to understand a project.
Good repositories explain their intent before their implementation. So the Reader follows a fixed order:
README→ docs/→ examples/→ package.json→ requirements.txt→ source code
It only moves deeper when the previous layer is not enough.
In many repositories, the README and examples already explain the workflow. Reading the entire source tree would add cost without adding understanding.
Documentation explains intent. Code explains implementation. You usually need the first before the second.
A Reader output might look like this:
{ "repository": "langgraph-agent", "context_loaded": [ "README", "docs/workflows.md", "examples/basic_agent.py" ], "source_code_loaded": false, "decision_reason": "Workflow identified before code analysis."}
This is not just a token optimization. It is a reasoning optimization.
If the model reads implementation first, it often gets buried in details. If it reads intent first, it knows what to look for.
This is the heart of the system.
The goal is not to understand the repository. The goal is to extract a reusable workflow. That is a narrower and better question.
The Extractor asks: Does this repository contain a process another agent could reuse?
If the answer is no, the repository leaves the pipeline. If the answer is yes, the workflow becomes structured data.
{ "skill_name": "github-pr-reviewer", "goal": "Review GitHub pull requests before merge", "inputs": [ "Pull Request URL", "Repository Context" ], "steps": [ "Read changed files", "Identify risky modifications", "Check coding standards", "Generate review comments" ], "outputs": [ "Review Summary", "Action Items" ], "failure_modes": [ "Missing project context", "Large architectural refactor", "Ambiguous project standards" ]}
Notice what changed. The system no longer cares about the original repository. It now cares about the workflow.
Reusable workflows outlive the repositories they came from. That is the key insight.
Not every extracted workflow deserves to become a Skill. Some are too specific. Some depend on hidden assumptions. Some require private infrastructure. Some are clever but not reusable.
Before generating anything, the Skill Scorer applies a fixed checklist. This is another place where deterministic rules beat LLM vibes.
The checks are simple:
The result is structured:
{ "skill_name": "github-pr-reviewer", "score": 0.94, "checks": { "readme": true, "examples": true, "min_steps": true, "reusable": true, "general_purpose": true, "clear_inputs_outputs": true }, "decision": "PASS"}
If the workflow fails, it stops here. That is important.
The system should not generate Skills just because it can. It should generate Skills only when the workflow clears a quality bar.
Once a workflow passes the score, it becomes an Agent Skill.
The Skill Generator transforms the extracted workflow into a standardized package another agent can install, read, and use. Every Skill follows the same structure, no matter where the workflow came from.
name: github-pr-reviewerversion: 1.0.0description: Review GitHub pull requests and generate actionable feedback.inputs: - github_pull_request - repository_contextsteps: - Read changed files - Identify risky modifications - Verify coding standards - Generate review commentsoutputs: - review_summary - review_commentstags: - github - code-review - engineering
The generated package also includes supporting files:
github-pr-reviewer/├── SKILL.md├── examples.md├── commands.md├── metadata.json└── tests.md
Standardization is what makes the library useful. A workflow extracted from a LangGraph project and one extracted from an MCP server may look completely different internally. But once they are converted into the same Skill format, the agent can use both without caring where they came from.
This is how you turn scattered public knowledge into an internal operating system.
The Reviewer has one job: Decide whether the Skill is worth publishing.
It does not rewrite the Skill. It does not add features. It does not approve its own work.
That last part matters. The agent that generates a Skill should not be the same agent that approves it. Generation and review must be separate.
The Reviewer prompt is intentionally simple:
Would an experienced engineer install this Skill without editing it?Answer only:YES or NOExplain your reasoning in one paragraph.Point out any missing assumptions or unclear steps.If NO, suggest the minimum changes required for approval.Do not rewrite the entire Skill.
A typical review result:
{ "decision": "YES", "confidence": 0.96, "reason": "The workflow is reusable, clearly documented, and can be applied outside the original repository."}
If the answer is NO, the pipeline stops. If the answer is YES, the Skill moves to publishing.
This separation is what keeps the system from becoming an automated slop generator.
The Publisher is pure automation.
Once a Skill is approved, it prepares a pull request. It does not merge. It does not bypass review. It does not touch the main branch.
The publishing flow is fixed:
Create branch→ Commit Skill package→ Open pull request→ Assign reviewer→ Wait for human approval
The pull request contains everything needed for review:
Title:Add Skill: github-pr-reviewerFiles:✓ SKILL.md✓ examples.md✓ commands.md✓ metadata.json✓ tests.md
This is the line I would not cross:
Nothing is merged automatically.
Automation can discover. Automation can extract. Automation can generate. Automation can propose.
But humans still approve what becomes part of the agent’s real capability set. That boundary keeps the system useful instead of reckless.
This kind of system needs guardrails from day one. Otherwise it will collect noise, duplicate ideas, and quietly pollute the Skill library.
I would add these controls before trusting it:
The dangerous failure mode is not that the system fails loudly. The dangerous failure mode is that it slowly fills your agent with mediocre capabilities.
Learning systems need forgetting systems too.
I would not let this system install Skills directly into production.
I would not let it merge its own pull requests.
I would not let it rewrite existing Skills without review.
I would not let it trust star count as a proxy for quality.
I would not let it ingest entire repositories by default.
And I would not pretend that this is the same thing as model training.
This is not self-training. This is capability acquisition through controlled workflow extraction.
That sounds less magical. It is also much more useful.
Here is the full system in one view:
The key pattern is simple:
Discovery is automated. Evaluation is structured. Publishing is reviewed.
That is the balance.
Today, most agents only improve when a human manually improves them. That is the bottleneck.
New workflows appear every day. New architectures appear every day. New tool-use patterns appear every day. But your agent does not automatically absorb them.
This pipeline is one way to close that gap.
It discovers repositories, filters noise, extracts reusable workflows, turns strong workflows into Skills, and asks a human before anything becomes official.
The goal is not to replace engineers. The goal is to stop wasting engineering time on repetitive discovery work. Engineers should decide which ideas matter. They should not spend hours digging through hundreds of repositories just to find them.
Today, AI can already write code. The next generation of agents will not just execute workflows. They will learn which workflows are worth adding.
This is my attempt to build that loop. Maybe you will build a better one.
This article continues the pipeline thinking from my earlier guide [“How to Build Your Own Tiny LLM From Scratch”]. That article explained the 5-stage training pipeline for LLMs. This one shows a real 8-stage pipeline for continuous agent learning.
Enjoyed this guide? Follow me for more practical AI engineering content.
If you build a version of this pipeline, share what you changed in the comments. I would love to see how other builders design learning agents.
How to Build an AI Agent That Keeps Learning From GitHub was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.