{"slug": "stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it", "title": "Stop letting your GPU idle while your CPU struggles to feed it", "summary": "PyTorch developers can eliminate GPU idle time caused by CPU bottlenecks by using multi-process data loading with `num_workers` set to the number of CPU cores, `pin_memory=True`, and binary formats like TFRecord, Apache Parquet, or WebDataset instead of raw CSVs or JSON files. The article provides a custom dataset class example and recommends memory mapping and sharding for datasets exceeding RAM and for distributed training across multiple GPUs.", "body_md": "# Stop letting your GPU idle while your CPU struggles to feed it\n\n## Handling the Bottleneck with Prefetching and Parallelism\n\nThe most common mistake is loading data synchronously. When the model finishes a batch, the GPU sits idle while the CPU fetches the next chunk from the disk. You can kill this latency by using multi-process loading. In PyTorch, this is handled via the `num_workers`\n\nparameter in the `DataLoader`\n\n.\n\nSetting `num_workers`\n\nto the number of CPU cores usually helps, but be careful with memory overhead. If you're using a massive dataset, you should combine this with `pin_memory=True`\n\n, which speeds up the transfer from CPU RAM to GPU VRAM by using page-locked memory.\n\n## Optimized Formats for Large Scale Training\n\nStop using raw CSVs or thousands of tiny JSON files. Opening and closing files creates massive overhead. For a real-world deployment, you need binary formats that support sequential reads and memory mapping.\n\n**TFRecord:** The gold standard for TensorFlow, storing data as a sequence of binary records.**Apache Parquet:** Incredible for tabular data due to columnar storage, which means you only load the features you actually need.**WebDataset:** Essential for vision tasks; it wraps data into POSIX tar files, allowing you to stream datasets over a network without needing to download the whole thing to a local SSD first.\n\n## A Practical Tutorial for Custom Data Pipelines\n\nIf you're building a custom LLM agent or a fine-tuning script, you'll likely need a custom dataset class. Here is a basic structure to ensure your data is preprocessed on the fly without blocking the training loop.\n\n``` python\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\n\nclass EfficientDataset(Dataset):\n    def __init__(self, data_path):\n        # Load metadata or index files here, not the full dataset\n        self.data = self._load_index(data_path)\n\n    def __len__(self):\n        return len(self.data)\n\n    def __getitem__(self, idx):\n        # Perform heavy transformations here\n        sample = self.data[idx]\n        processed_sample = self.transform(sample)\n        return torch.tensor(processed_sample)\n\n    def transform(self, x):\n        # Example: Normalization or tokenization\n        return x / 255.0\n\n# Deployment configuration for maximum throughput\nloader = DataLoader(\n    dataset=EfficientDataset(\"data/train\"),\n    batch_size=64,\n    shuffle=True,\n    num_workers=8, \n    pin_memory=True,\n    prefetch_factor=2\n)\n```\n\n## Memory Mapping and Sharding\n\nWhen your dataset exceeds your system RAM, memory mapping (`mmap`\n\n) is your best friend. It allows the OS to map a file directly into the virtual address space, loading pages only when they are accessed. For distributed training across multiple GPUs, you must implement sharding. This ensures that each GPU sees a unique subset of the data per epoch, preventing redundant computation and ensuring the gradient updates are based on a diverse sample of the global dataset. This is the only way to scale a deep dive project from a single local machine to a cluster.\n\n[Stop expecting LLMs to be databases because they are 1d ago](/en/news/6543/)\n\n[Since the provided content was only a title 2d ago](/en/news/6403/)\n\n[Sign language AI finally works on a mobile device 4d ago](/en/news/6150/)\n\n[Jeff Dean is chasing a 10 billion dollar valuation for his new 4d ago](/en/news/6135/)\n\n[Medical AI is still hallucinating stereotypes into patient care 7d ago](/en/news/5751/)\n\n[DeepMind WeatherNext actually predicts cyclones with scary 9d ago](/en/news/5584/)\n\n[Next Will Gen Z actually survive the AI takeover of entry-level roles? →](/en/news/6721/)\n\n[these AI tool field notes](https://tanyan888.com/), with plenty of directly applicable cases.", "url": "https://wpnews.pro/news/stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it", "canonical_source": "https://promptcube3.com/en/news/6724/", "published_at": "2026-08-17 21:02:24+00:00", "updated_at": "2026-08-17 21:12:52.582194+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["PyTorch", "TensorFlow", "Apache Parquet", "WebDataset", "TFRecord"], "alternates": {"html": "https://wpnews.pro/news/stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it", "markdown": "https://wpnews.pro/news/stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it.md", "text": "https://wpnews.pro/news/stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it.txt", "jsonld": "https://wpnews.pro/news/stop-letting-your-gpu-idle-while-your-cpu-struggles-to-feed-it.jsonld"}}