Just Train More: Measuring the Exchange Rate
A developer measured the "exchange rate" between training data and document length for a small transformer versus a zero-parameter count-table cache, finding that 16x more training data (from 500K to 8M tokens) moved the…