{"slug": "carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon", "title": "Carbon-Aware Model Training: Scheduling GPU Workloads Around Electricity Carbon Intensity", "summary": "A developer has built a carbon-aware training pipeline for PyTorch that schedules GPU workloads around real-time electricity carbon intensity, reducing CO2 emissions by delaying training until low-carbon windows are available. The pipeline uses gradient accumulation to minimize GPU memory, tracks emissions via CodeCarbon, and generates comparative reports quantifying carbon savings against a baseline run.", "body_md": "Training ML models has an environmental cost that most practitioners do not measure. A model trained during peak grid hours, when coal and gas plants are meeting high demand - can emit significantly more CO2 than the same model trained during off-peak hours when renewables dominate the grid. The carbon intensity of electricity varies by a factor of 2–5x throughout the day, but most training pipelines ignore this entirely.\n\n**Carbon-Aware Model Training Pipeline** is a PyTorch-based training pipeline that monitors real-time electricity carbon intensity, delays training until a low-carbon window is available, reduces GPU memory footprint through gradient accumulation, and tracks CO2 emissions throughout the training process using CodeCarbon - with a comparison report that quantifies the carbon savings against a baseline run.\n\n**Carbon-Aware Scheduling** - real-time carbon intensity monitoring with smart training delays until low-carbon windows are detected.\n\n**Gradient Accumulation** - reduces GPU memory footprint while maintaining effective batch size.\n\n**Emissions Tracking** - real-time CO2 monitoring via CodeCarbon with comprehensive JSON reports.\n\n**Modular Design** - YAML-based configuration with separate scheduler, tracker, and trainer modules.\n\n**GPU Optimized** - automatic CUDA detection with mixed precision training (FP16).\n\n**Comparative Analysis** - automated reporting quantifying carbon savings against a baseline run.\n\nThe pipeline runs in four stages:\n\n**Stage 1 - Carbon-Aware Scheduling**\n\nReal-time monitoring checks electricity carbon intensity via APIs. Smart delays wait for low-carbon windows before starting training. Fallback mechanisms use realistic mock data when APIs are unavailable - with diurnal patterns simulating peak intensity at 18:00 and trough at 03:00. Configurable thresholds allow customization for different regions.\n\n**Stage 2 - Gradient Accumulation**\n\nMemory optimization processes smaller micro-batches. Effective batch size is maintained with reduced memory. Configurable steps (2, 4, 8, 16) adapt to hardware constraints. Convergence preservation ensures model quality is not compromised.\n\n**Stage 3 - Emissions Tracking**\n\nCodeCarbon integration monitors CO2 emissions in real-time. Energy metrics track power consumption in Watts and energy in kWh. Comprehensive reports generate JSON summaries with all metrics. Comparative analysis quantifies carbon savings versus the baseline.\n\n**Stage 4 - GPU Optimization**\n\nMixed precision training (FP16) reduces memory and increases speed. Automatic CUDA detection uses GPU when available. Pin memory optimization enables faster data transfers. Graceful CPU fallback when GPU is unavailable.\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│                     Training Configuration                      │\n│                       (YAML Config File)                        │\n└─────────────────────────┬───────────────────────────────────────┘\n                          │\n                          ▼\n         ┌────────────────────────────────────┐\n         │   Carbon Intensity Scheduler       │\n         │   - API/Mock data fetch            │\n         │   - Threshold comparison           │\n         │   - Wait for low-carbon window     │\n         └────────────────┬───────────────────┘\n                          │\n                          ▼\n              ┌───────────────────────┐\n              │   Start Training?     │\n              │   Intensity < 300?    │\n              └─────┬─────────────┬───┘\n                    │ NO          │ YES\n                    ▼             ▼\n            ┌───────────┐   ┌──────────────┐\n            │   Wait    │   │ Start Tracker│\n            │ & Recheck │   │ (CodeCarbon) │\n            └───────────┘   └──────┬───────┘\n                                   │\n                                   ▼\n                  ┌────────────────────────────────┐\n                  │   PyTorch Training Loop        │\n                  │   - Gradient Accumulation      │\n                  │   - Mixed Precision (FP16)     │\n                  │   - Checkpointing              │\n                  └────────────────┬───────────────┘\n                                   │\n                                   ▼\n                  ┌────────────────────────────────┐\n                  │   Emissions Tracking           │\n                  │   - CO2 (kg)                   │\n                  │   - Energy (kWh)               │\n                  │   - Power (Watts)              │\n                  └────────────────┬───────────────┘\n                                   │\n                                   ▼\n              ┌───────────────────────────────────┐\n              │   Save Results                    │\n              │   - Model checkpoint              │\n              │   - Training summary (JSON)       │\n              │   - Emissions log (CSV)           │\n              └───────────────────────────────────┘\n```\n\n**Prerequisites**\n\n```\ngit clone https://github.com/dakshjain-1616/CarbonAwareModelTraining---by-NEO.git\ncd CarbonAwareModelTraining---by-NEO\n\npython3 -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\npip install -r requirements.txt\n```\n\nRequired packages: `torch>=2.0.0`\n\n, `torchvision>=0.15.0`\n\n, `codecarbon>=2.3.0`\n\n, `pyyaml>=6.0`\n\n, `numpy`\n\n.\n\n```\nsource venv/bin/activate\n\n# Run baseline training (no optimization)\nexport PYTHONPATH=\"$PWD/src:$PYTHONPATH\"\npython src/train.py configs/baseline.yaml\n\n# Run optimized training (carbon-aware + gradient accumulation)\npython src/train.py configs/optimized.yaml\n\n# Generate comparison report\npython generate_comparison.py\n```\n\nThis runs three steps: baseline training without carbon awareness, optimized training with carbon-aware scheduling and gradient accumulation, and a comparison report that quantifies carbon savings and performance metrics.\n\nConfigure carbon-aware training in `configs/optimized.yaml`\n\n:\n\n```\nscheduler:\n  enabled: true\n  carbon_threshold: 300           # gCO2/kWh\n  wait_for_low_carbon: true\n\ntraining:\n  batch_size: 16\n  gradient_accumulation_steps: 4  # Effective batch = 64\n  epochs: 3\n```\n\nRun optimized training:\n\n```\npython src/train.py configs/optimized.yaml\n```\n\nOutput:\n\n```\n============================================================\nCARBON-AWARE TRAINING STARTED\n============================================================\n\nCarbon Intensity Check:\n  Current Intensity: 420.5 gCO2/kWh\n  Threshold: 300 gCO2/kWh\n  Status: ⏳ Waiting for low-carbon window...\n\n[10 minutes later]\n  Current Intensity: 285.3 gCO2/kWh\n  Status: ✅ Starting training now!\n\nTraining Progress:\n  Epoch 1/3 - Loss: 0.324 - Accuracy: 91.2%\n  CO2 Emissions: 0.042 kg\n  Energy Consumed: 0.15 kWh\n\n============================================================\nCARBON SAVINGS vs BASELINE\n============================================================\n\nCO2 Reduction: 32.5% (0.024 kg saved)\nGPU Memory Reduction: 45.8%\nAccuracy: 93.1% (baseline: 93.4%)\n```\n\n**Carbon-Aware Scheduling Only**\n\nDisable gradient accumulation, enable scheduling:\n\n```\nscheduler:\n  enabled: true\n  carbon_threshold: 250\n\ntraining:\n  gradient_accumulation_steps: 1  # No accumulation\n```\n\n**Gradient Accumulation Only**\n\nDisable scheduling, enable memory optimization:\n\n```\nscheduler:\n  enabled: false\n\ntraining:\n  batch_size: 8\n  gradient_accumulation_steps: 8  # Effective batch = 64\n```\n\n**Real Carbon Intensity API**\n\nConfigure for production with a real API:\n\n```\nscheduler:\n  enabled: true\n  use_mock_data: false\n  api_endpoint: \"https://api.carbonintensity.org.uk/intensity\"\n  region: \"GB\"\n```\n\n**Custom Model Integration**\n\nReplace `SimpleCNN`\n\nin `src/train.py`\n\n:\n\n``` python\nfrom my_models import MyCustomModel\n\ndef prepare_model(config):\n    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n    model = MyCustomModel(\n        input_channels=config['training']['input_channels'],\n        num_classes=config['training']['num_classes']\n    ).to(device)\n    return model, device\n```\n\n**Output Format**\n\nTraining summary JSON saved to `output/summary_optimized.json`\n\n:\n\n```\n{\n  \"run_name\": \"optimized\",\n  \"training_metrics\": {\n    \"final_accuracy\": 93.1,\n    \"final_loss\": 0.124,\n    \"epochs\": 3,\n    \"total_time_seconds\": 245\n  },\n  \"carbon_metrics\": {\n    \"total_emissions_kg\": 0.042,\n    \"energy_consumed_kwh\": 0.15,\n    \"avg_power_watts\": 145.2\n  },\n  \"scheduler_metrics\": {\n    \"wait_time_seconds\": 600,\n    \"initial_intensity\": 420.5,\n    \"training_intensity\": 285.3\n  },\n  \"gpu_metrics\": {\n    \"peak_memory_mb\": 2048,\n    \"gradient_accumulation_steps\": 4,\n    \"effective_batch_size\": 64\n  }\n}\n```\n\nComparison report saved to `output/comparison_report.json`\n\n:\n\n```\n{\n  \"carbon_savings\": {\n    \"baseline_emissions_kg\": 0.074,\n    \"optimized_emissions_kg\": 0.042,\n    \"reduction_kg\": 0.032,\n    \"reduction_percentage\": 43.2\n  },\n  \"accuracy_impact\": {\n    \"baseline_accuracy\": 93.4,\n    \"optimized_accuracy\": 93.1,\n    \"degradation_percentage\": 0.3\n  },\n  \"memory_savings\": {\n    \"baseline_memory_mb\": 4096,\n    \"optimized_memory_mb\": 2048,\n    \"reduction_percentage\": 50.0\n  }\n}\n```\n\nEvaluated on MNIST training - 3 epochs, RTX 3090 GPU:\n\n**Carbon Intensity Patterns (Mock Data):**\n\nPeak hours 18:00–22:00: ~450 gCO2/kWh\n\nOff-peak hours 02:00–06:00: ~200 gCO2/kWh\n\nAverage reduction: 35–45% CO2 by scheduling during low-carbon windows\n\n**GPU Memory Savings:**\n\nGradient accumulation 2x: ~30% memory reduction\n\nGradient accumulation 4x: ~50% memory reduction\n\nGradient accumulation 8x: ~60% memory reduction\n\n**Convergence Validation:**\n\nAccuracy degradation under 1% across all tested configurations\n\nLoss convergence matches baseline within 2% tolerance\n\nNo divergence observed\n\n```\nCarbonAwareModelTraining---by-NEO/\n├── src/\n│   ├── scheduler.py                # Carbon intensity API & scheduling\n│   ├── tracker.py                  # CodeCarbon emissions tracking\n│   ├── train.py                    # Main training pipeline\n│   └── utils.py                    # Config loading & logging\n├── configs/\n│   ├── baseline.yaml               # Baseline training config\n│   └── optimized.yaml              # Carbon-aware optimized config\n├── output/\n│   ├── summary_baseline.json       # Baseline training summary\n│   ├── summary_optimized.json      # Optimized training summary\n│   ├── comparison_report.json      # Comparative analysis\n│   ├── emissions.csv               # CodeCarbon emissions log\n│   └── training_*.log              # Detailed training logs\n├── models/\n│   ├── model_baseline.pt           # Baseline model checkpoint\n│   └── model_optimized.pt          # Optimized model checkpoint\n├── data/                            # MNIST dataset (auto-downloaded)\n├── requirements.txt                 # Python dependencies\n├── generate_comparison.py          # Comparison report generator\n└── README.md\n```\n\n**Why Carbon-Aware Scheduling?**\n\nCarbon intensity varies 2–5x throughout the day. Scheduling training during low-carbon windows reduces emissions without affecting model quality. Low-carbon periods also often correlate with cheaper electricity.\n\n**Why Gradient Accumulation?**\n\nGradient accumulation enables training larger models on limited hardware by processing smaller micro-batches and updating weights less frequently. Used in BERT, GPT, and other large-scale models for the same reason.\n\n**Why CodeCarbon?**\n\nCodeCarbon uses lifecycle assessment methodologies, supports CPU, GPU, and multi-device setups, and produces transparent, community-validated calculations. It tracks energy, power, and emissions in a single library.\n\n**Why YAML Configuration?**\n\nYAML configs are version-controlled, human-readable, and separate code from experiment parameters - enabling reproducible A/B comparisons between baseline and optimized runs.\n\nValidate installation:\n\n``` python\npython -c \"import torch; print(f'PyTorch: {torch.__version__}')\"\npython -c \"import codecarbon; print('CodeCarbon: OK')\"\npython -c \"import yaml; print('PyYAML: OK')\"\npython -c \"import torch; print(f'CUDA Available: {torch.cuda.is_available()}')\"\n```\n\nRun a quick 5-minute test:\n\n```\npython src/train.py configs/test.yaml\n```\n\nValidate carbon savings:\n\n```\npython src/train.py configs/baseline.yaml\npython src/train.py configs/optimized.yaml\npython generate_comparison.py\ncat output/comparison_report.json\n```\n\n**CUDA Out of Memory**\n\nReduce `batch_size`\n\nand increase `gradient_accumulation_steps`\n\nin the config.\n\n**Carbon Intensity API Timeout**\n\nNo action needed - the pipeline automatically falls back to mock data and training proceeds.\n\n**Module Import Errors**\n\n```\nexport PYTHONPATH=\"$PWD/src:$PYTHONPATH\"\n```\n\n**CodeCarbon Tracking Fails**\n\n```\npip install --upgrade codecarbon\n```\n\nTraining continues without emissions tracking if CodeCarbon fails.\n\n**Scheduler Waits Too Long**\n\nIncrease `max_wait_seconds`\n\n, raise `carbon_threshold`\n\n, or set `wait_for_low_carbon: false`\n\nin the config.\n\nThis project was built using NEO. [NEO](https://heyneo.com/) is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.\n\nThe requirement was a PyTorch training pipeline that schedules GPU workloads based on real-time carbon intensity, reduces memory footprint through gradient accumulation, and tracks emissions with CodeCarbon - producing a side-by-side comparison report. NEO built the full implementation: the carbon intensity scheduler in `scheduler.py`\n\nwith API integration and mock fallback, the CodeCarbon emissions tracker in `tracker.py`\n\n, the main training pipeline in `train.py`\n\nwith gradient accumulation and mixed precision FP16, the config loader and logging utilities in `utils.py`\n\n, the YAML configs for baseline and optimized runs, the comparison report generator in `generate_comparison.py`\n\n, and the full output structure covering JSON summaries, emissions CSV, and model checkpoints.\n\n**Use it to measure the carbon cost of your existing training runs.**\n\nRun `python src/train.py configs/baseline.yaml`\n\non your own model and data by replacing `SimpleCNN`\n\nin `src/train.py`\n\nwith your model. The CodeCarbon tracker produces a JSON summary with CO2 in kg, energy in kWh, and average power in Watts, a baseline measurement before any optimization.\n\n**Use the comparison report to justify scheduling infrastructure.**\n\nRun both the baseline and optimized configs on the same dataset. The `comparison_report.json`\n\ngives you a concrete before and after - percentage reduction in emissions, energy, and memory, alongside accuracy degradation, that makes the case for carbon-aware scheduling with real numbers from your own hardware.\n\n**Use mock data for development and real API for production.**\n\nSet `use_mock_data: true`\n\nduring development so training always proceeds without waiting. Switch to `use_mock_data: false`\n\nwith a real `api_endpoint`\n\nfor production runs where actual carbon savings matter.\n\n**Extend the scheduler with additional carbon intensity sources.**\n\nThe scheduler in `scheduler.py`\n\nfetches from a configurable `api_endpoint`\n\n. Adding support for additional regional carbon intensity APIs - Electricity Maps, WattTime, or a custom internal source, means updating the fetch logic in `scheduler.py`\n\nwithout touching the training loop, tracker, or reporting pipeline.\n\nCarbon intensity varies throughout the day, and most training pipelines ignore it. A 43% reduction in CO2 emissions with less than 1% accuracy degradation, achieved by scheduling when the grid is cleaner and accumulating gradients to reduce memory - shows that sustainable ML is a practical engineering choice, not just an aspiration.\n\nThe code is at [https://github.com/dakshjain-1616/CarbonAwareModelTraining](https://github.com/dakshjain-1616/CarbonAwareModelTraining)\n\nYou can also build with NEO in your IDE using the [VS Code extension](https://marketplace.visualstudio.com/items?itemName=NeoResearchInc.heyneo) or [Cursor](https://open-vsx.org/extension/NeoResearchInc/heyneo).\n\nYou can use NEO MCP with Claude Code: [https://heyneo.com/claude-code](https://heyneo.com/claude-code)", "url": "https://wpnews.pro/news/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon", "canonical_source": "https://dev.to/nilofer_tweets/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon-intensity-b4b", "published_at": "2026-06-06 08:48:43+00:00", "updated_at": "2026-06-06 09:11:28.715045+00:00", "lang": "en", "topics": ["machine-learning", "ai-ethics", "mlops", "ai-infrastructure", "artificial-intelligence"], "entities": ["PyTorch", "CodeCarbon", "GPU"], "alternates": {"html": "https://wpnews.pro/news/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon", "markdown": "https://wpnews.pro/news/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon.md", "text": "https://wpnews.pro/news/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon.txt", "jsonld": "https://wpnews.pro/news/carbon-aware-model-training-scheduling-gpu-workloads-around-electricity-carbon.jsonld"}}