You want a model to look over a day of sensor readings and flag anything odd. One reading a minute is 1,440 rows, and you write them out with Python's csv module, the obvious way. That file is 49,723 tokens. The same information fits in 17,287. The other 32,436 tokens are digits that either repeat o
n every row or mean nothing. You pay per token, a short chunk of text the model reads as one piece. Input tokens are what you send, output tokens are what it writes back, and each has its own price per million. A tokenizer is the program that does the cutting.
I used tiktoken, OpenAI's open-source one, with its o200k_base encoding. For numbers the rule is mechanical: digits are cut into groups of up to three, left to right. 19.94 is three tokens: 19, ., 94. 19.944444444444446 is seven: 19, ., 944, 444, 444, 444, 446.
That second number is what Python gives you for (67.9 - 32) * 5 / 9, a Fahrenheit reading converted to Celsius. A floating-point number is stored in binary, most decimal fractions can't be stored exactly, and Python prints the shortest form that reads back to the same value, often 17 digits. str(), f-strings, csv.writer and json.dumps all print that full form. The sensor measured to 0.1 °F, so four digits carry the measurement and thirteen are leftovers of arithmetic.
I built one synthetic day: a greenhouse sensor, one row per minute, three columns. Temperature is the converted reading above. Humidity is the average of three whole-number samples, which is where 50.666666666666664 comes from. Timestamps are isoformat() strings with microseconds, as datetime.now() produces them.
Then I counted the file four ways. What you send, 1,440 rows Tokens Per row Saved csv.writer default 49,723 34.5 — Values rounded to 2 decimals 40,327 28.0 19% Timestamps as HH:MM, values as is 26,683 18.5 46% Both 17,287 12.0 65% I expected the long floats to be the problem. They're the smaller half. The timestamp column alone is 28,799 tokens, 58% of the file.
2026-10-22T00:00:00.670487+00:00 is 19 tokens, and 16 of them are a date and timezone that repeat on every row plus seconds and microseconds that only record when the logger woke up. 00:00 is 3 tokens. That long form is ISO 8601, the standard date-time format, and it's the right choice for storage. In a prompt where every row shares one date, it's a tax.
With short timestamps in place, I tried rounding harder. One, two and three decimals all came out at exactly 17,287 tokens. That's the three-digit rule again: 19.9, 19.94 and 19.944 are each 19, . and one more token.
I checked all 1,110 digit strings of length one to three, and every one is a single token. The fourth decimal is where the next token starts. So one decimal only costs you data. Two decimals kept all 96 distinct temperatures in the file.
One decimal merged them into 53, because 0.1 °C is coarser than the sensor's 0.1 °F. The rule: round to the precision your source has, and don't expect tokens back below three decimals. To check, count distinct values before and after rounding. If the count drops, you rounded away a measurement.
Save your readings the way your code writes them and run python compact_csv.py readings.csv. It expects an ISO timestamp in the first column and numbers in the rest: import csv, sys, tiktoken # pip install tiktoken from datetime import datetime DECIMALS = 2 # the precision your data really has; 1-3 cost the same tokens src = open(sys.argv[1], encoding="utf-8").read() header, *rows = csv.reader(src.splitlines()) first = datetime.fromisoformat(rows[0][0]) lines = [f"# {first:%Y-%m-%d} {first:%Z}, time is HH:MM", ",".join(header)] for ts, *values in rows: hhmm = f"{datetime.fromisoformat(ts):%H:%M}" lines.append(",".join([hhmm] + [str(round(float(v), DECIMALS)) for v in values])) compact = "\n".join(lines) tok = tiktoken.get_encoding("o200k_base").encode print(f"{len(tok(src)):,} tokens as written, {len(tok(compact)):,} compact") print(lines[0], lines[2], sep="\n") It writes the date and timezone once in a header line, keeps hours and minutes, rounds the values and prints both counts. On my test file it prints 49,723 tokens as written, 17,302 compact; the header line is the 15 tokens over the table. The compact variable is what you send.
Now the bill. Say ten sensors, each checked every hour against its last 24 hours: 240 calls a day, each with a 41-token instruction and an answer of about 200 tokens. Prices are gpt-5.6's list prices, $2 per million input tokens and $12 per million output, from the catalog of the LLM gateway I work on, read on 23 October 2026. As written, a call costs $0.10 and the month $734.
Compact, it's $0.037 a call and $267 a month. Input is 98% of the first bill, so the answer length barely matters here. On gpt-5.6-mini, ten times cheaper, the month is $73 against $27. Rounding destroys information when the digits are real: money (keep the cents, and keep money out of floats altogether), GPS coordinates, where the fifth decimal is about a metre, or lab measurements.
HH:MM only works while the window sits inside one day and one timezone. A window across midnight needs the date where it changes, or minutes since the window start. The data is synthetic, one simulated sensor, and the counts come from tiktoken, not from an API's usage field. Claude and Gemini use their own tokenizers, so count yours with your vendor's token counter.
Sending min, max and mean computed in code is cheaper still, though then the model no longer sees each reading. The gateway sells gpt-5.6 at $1.70 / $10.20, 15% under list, which makes the two monthly bills $624 and $227. It bills the tokens you send, digits included, and rounds nothing for you. The tokenizer can't tell a measurement from arithmetic noise, and it bills both.
Write the date once and round to the precision you have, and a day of readings costs a third of what Python's defaults send.
Key Takeaways #
- •You pay per token, a short chunk of text the model reads as one piece.
- •This story was reported by Dev.to , covering developments in thedev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: