DeepSeek: What They Invented

DeepSeek, a Chinese AI lab, has developed a novel training technique that reduces the cost of training large language models by up to 90%, according to a technical report published on the company's website. The method, which combines sparse attention with a new parameter-sharing scheme, enables models with 100 billion parameters to be trained on 2,048 GPUs instead of the 16,384 typically required. This breakthrough could significantly lower the barrier to entry for AI research and development, potentially reshaping the competitive landscape of the AI industry.