Flow – a compact data pipeline language that cuts LLM token usage by 33%

A new open-source data pipeline language called Flow reduces LLM token usage by an average of 33% compared to Python by stripping away human-oriented boilerplate. Developed by Pinku, Flow uses single-character operators and transpiles to Python, with validated results on fine-tuned models like phi-2 and Qwen 3B Coder v2 running on consumer GPUs.

A compact, AI-optimized data transformation language for LLMs ~33% token reduction vs Python — validated with fine-tuned models on consumer GPUs git clone https://github.com/pinku/Flow.git && cd Flow python3 flow.py "1..10 | f: 5 | m: 2" → 12, 14, 16, 18, 20 No dependencies. No setup. Just Python 3.8+. LLMs are trained on verbose human-oriented languages, but that verbosity burns tokens without adding value for AI processing: | Problem | Impact | |---|---| Keywords like function , const , return | Every one costs tokens | | Verbose variable names for human readability | No benefit for LLMs | | Syntactic sugar optimized for humans | Wasted context window | Flow strips away the human-oriented boilerplate , keeping only what the LLM needs to express the transformation. The result: 33% fewer tokens on average , tested and verified with real models. | Python 24 tokens | Flow 13 tokens — 45% fewer | |---|---| result = x 2 for x in range 1, 11 if x 5 | 1..10 | f: 5 | m: 2 | More examples in the Operator Reference operators-reference . | Feature | Description | |---|---| Minimal Syntax | Single-character operators: f: filter , m: map , g: group , a: aggregate , etc. | Python Compatible | Transpiles to clean, readable Python | LLM Optimized | Designed for minimal token usage without sacrificing expressiveness | JSON Native | Built-in JSON parsing and serialization | Batch Processing | Windowing and batching for large datasets | Type Safety | Type casting and validation | Zero Dependencies | Pure Python — no pip install required | git clone https://github.com/pinku/Flow.git cd Flow python3 flow.py --help Filter and map python3 flow.py "1..10 | f: 5 | m: 2" Group and aggregate python3 flow.py " {name:'Alice',age:30},{name:'Bob',age:25} | g: .age =30 | a:len v " JSON processing python3 flow.py "S:data.json" Save to JSON python3 flow.py "J" Parse JSON input python3 flow.py 1..10 | f: 5 | m: 2 12, 14, 16, 18, 20 python from flow import flow run, flow transpile Execute directly result = flow run "1..10 | f: 5 | m: 2" print result 12, 14, 16, 18, 20 Get transpiled Python code = flow transpile "1..10 | f: 5 | m: 2" print code 2 for in range 1, 11 if 5 | Operator | Name | Example | |---|---|---| f: | Filter | data | f: .age 18 | m: | Map | data | m: .name.upper | g: | Group | data | g: .category | a: | Aggregate | data | g: .cat | a:sum v | s: | Sort | data | s:- .age | t: | Take | data | t:10 | d: | Drop | data | d:2 | u | Unique | data | u | r | Reverse | data | r | b: | Batch | data | b:32 | w: | Window | data | w:3 | i: | Split | data | i:'\n' | o: | Join | data | o:',' | T: | Type cast | data | T:int | J | Parse JSON | data | J | S: | Save JSON | data | S:output.json | x | Flatten | nested | x | & | AND filter | data | f: .a 0 & .b<10 | | | OR filter | data | f: .a 0 | .b 10 | Full interactive report: benchmark report.html /pinku/Flow/blob/master/benchmark report.html | Task | Python Tokens | Flow Tokens | Savings | |---|---|---|---| | Filter + Map | 22 | 13 | 41% | | Group + Aggregate | 25 | 14 | 44% | | Sort + Take | 20 | 14 | 30% | | JSON Parse | 15 | 4 | 73% | | Batch Processing | 35 | 8 | 77% | Average token saving: 33% Flow has been validated with fine-tuned models on consumer hardware: | Model | Parameters | VRAM | Training Time | Accuracy | |---|---|---|---|---| | phi-2 | 2.7B | ~5.4 GB | ~20 min | 100% | | Qwen 3B Coder v2 | 3.7B | ~6.0 GB | ~5 min | 100% | Both models were fine-tuned using QLoRA 4-bit quantization on an RTX 4060. Flow includes training scripts for fine-tuning small language models: Install dependencies pip install transformers peft bitsandbytes trl Generate training dataset python3 generate dataset.py Train phi-2 2.7B python3 train qlora.py Train Qwen 3B Coder python3 train qwen flow.py Requirements: - NVIDIA GPU with 8GB+ VRAM RTX 4060 tested - CUDA 12.x - Python 3.8+ Flow includes a Hermes skill for seamless integration: Add to your Hermes skills cp -r .hermes/skills/flow-tool ~/.hermes/skills/ Use in chat /hermes run flow-tool "1..10 | f: 5 | m: 2" Flow/ ├── flow.py Main implementation transpiler + runtime ├── data/ │ └── flow dataset.json Training dataset 600 examples ├── models/ Fine-tuned LoRA adapters not in repo │ ├── flow-lora/ phi-2 adapter │ └── qwen3b-flow-lora-v2/ Qwen 3B adapter ├── benchmark report.html Visual benchmark report ├── train qlora.py phi-2 training script ├── train qwen flow.py Qwen 3B training script └── generate dataset.py Dataset generation script Hermes Agent https://hermes-agent.nousresearch.com — LLM agent platform; Flow is used as a tool skill llama.cpp https://github.com/ggerganov/llama.cpp — Local LLM inference; models trained on Flow data run here MIT License — free for any use. This project explores the hypothesis that purpose-built languages for LLMs can achieve significant efficiency gains compared to traditional programming languages. The key findings: Token efficiency is real : Flow achieves 33% token reduction on average Small models can learn : 2.7-3.7B parameter models achieve 100% accuracy after fine-tuning Hardware accessible : QLoRA enables training on consumer GPUs RTX 4060 For the full analysis and methodology, see benchmark report.html /pinku/Flow/blob/master/benchmark report.html .