A compact, AI-optimized data transformation language for LLMs
~33% token reduction vs Python — validated with fine-tuned models on consumer GPUs
git clone https://github.com/pinku/Flow.git && cd Flow
python3 flow.py "1..10 | f:_>5 | m:_*2"
No dependencies. No setup. Just Python 3.8+.
LLMs are trained on verbose human-oriented languages, but that verbosity burns tokens without adding value for AI processing:
| Problem | Impact |
|---|---|
Keywords like function , const , return |
|
| Every one costs tokens | |
| Verbose variable names for human readability | No benefit for LLMs |
| Syntactic sugar optimized for humans | Wasted context window |
Flow strips away the human-oriented boilerplate, keeping only what the LLM needs to express the transformation. The result: 33% fewer tokens on average, tested and verified with real models.
| Python (24 tokens) | Flow (13 tokens) — 45% fewer |
|---|---|
result = [x * 2 for x in range(1, 11) if x > 5] |
|
| `1..10 | f:_>5 |
More examples in the Operator Reference.
| Feature | Description |
|---|---|
| Minimal Syntax | |
Single-character operators: f: (filter), m: (map), g: (group), a: (aggregate), etc. |
|
| Python Compatible | |
| Transpiles to clean, readable Python | |
| LLM Optimized | |
| Designed for minimal token usage without sacrificing expressiveness | |
| JSON Native | |
| Built-in JSON parsing and serialization | |
| Batch Processing | |
| Windowing and batching for large datasets | |
| Type Safety | |
| Type casting and validation | |
| Zero Dependencies | |
| Pure Python — no pip install required |
git clone https://github.com/pinku/Flow.git
cd Flow
python3 flow.py --help
python3 flow.py "1..10 | f:_>5 | m:_*2"
python3 flow.py "[{name:'Alice',age:30},{name:'Bob',age:25}] | g:_.age>=30 | a:len(v)"
python3 flow.py "S:data.json" # Save to JSON
python3 flow.py "J" # Parse JSON input
python3 flow.py
python
from flow import flow_run, flow_transpile
result = flow_run("1..10 | f:_>5 | m:_*2")
print(result) # [12, 14, 16, 18, 20]
code = flow_transpile("1..10 | f:_>5 | m:_*2")
print(code) # [_*2 for _ in range(1, 11) if _>5]
| Operator | Name | Example |
|---|---|---|
f: |
||
| Filter | `data | f:_.age > 18` |
m: |
||
| Map | `data | m:_.name.upper()` |
g: |
||
| Group | `data | g:_.category` |
a: |
||
| Aggregate | `data | g:_.cat |
s: |
||
| Sort | `data | s:-_.age` |
t: |
||
| Take | `data | t:10` |
d: |
||
| Drop | `data | d:2` |
u |
||
| Unique | `data | u` |
r |
||
| Reverse | `data | r` |
b: |
||
| Batch | `data | b:32` |
w: |
||
| Window | `data | w:3` |
i: |
||
| Split | `data | i:'\n'` |
o: |
||
| Join | `data | o:','` |
T: |
||
| Type cast | `data | T:int` |
J |
||
| Parse JSON | `data | J` |
S: |
||
| Save JSON | `data | S:output.json` |
x |
||
| Flatten | `nested | x` |
& |
||
| AND filter | `data | f:_.a>0 & _.b<10` |
| ` | ` | |
| OR filter | `data | f:_.a>0 |
Full interactive report: benchmark_report.html
| Task | Python Tokens | Flow Tokens | Savings |
|---|---|---|---|
| Filter + Map | 22 | 13 | 41% |
| Group + Aggregate | 25 | 14 | 44% |
| Sort + Take | 20 | 14 | 30% |
| JSON Parse | 15 | 4 | 73% |
| Batch Processing | 35 | 8 | 77% |
Average token saving: 33%
Flow has been validated with fine-tuned models on consumer hardware:
| Model | Parameters | VRAM | Training Time | Accuracy |
|---|---|---|---|---|
| phi-2 | 2.7B | ~5.4 GB | ~20 min | 100% |
| Qwen 3B Coder v2 | 3.7B | ~6.0 GB | ~5 min | 100% |
Both models were fine-tuned using QLoRA (4-bit quantization) on an RTX 4060.
Flow includes training scripts for fine-tuning small language models:
pip install transformers peft bitsandbytes trl
python3 generate_dataset.py
python3 train_qlora.py
python3 train_qwen_flow.py
Requirements:
- NVIDIA GPU with 8GB+ VRAM (RTX 4060 tested)
- CUDA 12.x
- Python 3.8+
Flow includes a Hermes skill for seamless integration:
cp -r .hermes/skills/flow-tool ~/.hermes/skills/
/hermes run flow-tool "1..10 | f:_>5 | m:_*2"
Flow/
├── flow.py # Main implementation (transpiler + runtime)
├── data/
│ └── flow_dataset.json # Training dataset (600 examples)
├── models/ # Fine-tuned LoRA adapters (not in repo)
│ ├── flow-lora/ # phi-2 adapter
│ └── qwen3b-flow-lora-v2/ # Qwen 3B adapter
├── benchmark_report.html # Visual benchmark report
├── train_qlora.py # phi-2 training script
├── train_qwen_flow.py # Qwen 3B training script
└── generate_dataset.py # Dataset generation script
Hermes Agent— LLM agent platform; Flow is used as a tool skillllama.cpp— Local LLM inference; models trained on Flow data run here
MIT License — free for any use.
This project explores the hypothesis that purpose-built languages for LLMs can achieve significant efficiency gains compared to traditional programming languages. The key findings:
Token efficiency is real: Flow achieves 33% token reduction on average** Small models can learn**: 2.7-3.7B parameter models achieve 100% accuracy after fine-tuning** Hardware accessible**: QLoRA enables training on consumer GPUs (RTX 4060)
For the full analysis and methodology, see benchmark_report.html.