Jev speaks binary

When we wrote What if Jev spoke Arrow? in late September, the TypeSafe API spoke only JSON, so we built Jevaro to find out what an Arrow interface to Jev would feel like. A lot has happened in the few weeks since. TypeSafe released a new version of Jev that can look at images, we released four new versions of Jevaro, and MotherDuck took on one of computer vision’s most pressing questions: hot dog or not hot dog?

One detail of TypeSafe’s release stood out to us more than the rest: to send images, the API now accepts binary data, with no Base64 encoding needed. Here’s a quick tour of what’s new, and why that detail matters for Arrow.

What’s new in Jev #

Jev 1.14, released on October 14, can look at images in a request’s state, instructions, or criteria. It also adds Box, a question type that returns the bounding box of one thing in an image, and an option for longer context. All three are experimental and opt-in: each request that uses them needs a TypeSafe-Danger-Zone-* header.

What interests us most is how the images get there. The API now accepts multipart/form-data requests: a JSON part named request, then each image as a file part, byte for byte. Images Base64-encoded in the JSON work too, but TypeSafe’s docs recommend multipart, because Base64 makes each image about a third larger.

What’s new in Jevaro #

Jevaro 0.5.0 supports all of these new features. The main point of Jevaro was to model Jev’s answers in an Arrow schema, so that each question becomes a typed column that analytic tools can use directly. Box is the first new answer type since our last post, so it needed a column type of its own:

Box
  struct<
    box: struct<
      top: float64 not null,
      bottom: float64 not null,
      left: float64 not null,
      right: float64 not null
    > not null,
    confidence: float64
  >
  metadata: {}

Each edge is a fraction of the image’s height or width, and confidence is null when TypeSafe doesn’t report one. But a box like this is also one of the most common shapes in geospatial data, and GeoArrow already defines a standard Arrow type for it. So we made that an option: with "box_format": "geoarrow", the box field is a geoarrow.box instead, and tools that understand GeoArrow can read it as a rectangle.

Answers are only half of the Arrow story, though. Our last post promised Arrow input, and Jevaro now has it, so it speaks Arrow in both directions. A multipart/form-data request has the questions in a JSON part and the states in an Arrow IPC stream, one state per row. That means a table from DuckDB, Polars, or pandas can go in, and a table of answers comes back in the same row order, ready to join to it. Images fit the same model: an Arrow binary column holds one image per row. Jevaro also accepts images the two ways TypeSafe does, Base64-encoded in JSON or as multipart file parts.

Jevaro has also gotten faster. With more calls in flight over four HTTP/2 connections, it handles about 1,040 states per second at one state per call, more than twice the rate in our last post. A new rows_per_call option packs many states into each call. That sustains about 2,000 states per second with 28% fewer input tokens, and a 10,000-state job finishes in about a second, though answers shift slightly. In a test of Box questions about 2,000 small generated PNGs, Jevaro streamed answers at 1,130 images per second.

Meanwhile, at MotherDuck 🌭 #

We’re not the only ones putting Jev’s new image support to work. MotherDuck’s prompt_jev() function now works on images, which makes images just another column in MotherDuck. In Arrow terms, that’s a binary column, the same way Jevaro accepts them. On the Food-101 dataset, Jev was the fastest and cheapest model they tested. On its own it was also the least accurate, but when they kept only answers with a confidence of 0.8 or more, about 80% of them, Jev was right 97.7% of the time.

Then they built hotdogbench.com, which asks ten models whether each of 1,000 photos shows a hot dog. Every model scored between 96.9% and 98.5%, but Jev classified 250 photos per second, five times as fast as the next fastest model. In MotherDuck’s words, “Jev isn’t the top dog, but it’s the first one to the bun.”

Jev speaks binary #

Now, back to that detail of how images reach Jev. Arrow is a binary format, and until Jev 1.14, every request to the API was JSON text. Now a request can have binary parts beside its JSON. That’s the same shape Jevaro uses for Arrow input, with the questions in a JSON part and the states in an Arrow IPC stream part.

Binary parts also mean no Base64 encoding, and in the Arrow community, we’re well-known Base64 haters. Base64 makes binary data a third larger and costs time to encode and decode. When we benchmarked the same image requests sent both ways, TypeSafe processed them equally fast, but once the network uplink was full, Base64 delivered 14–16% fewer images per second. And because the API’s 32 MiB limit counts the decompressed request body, a request can have at most 24 MiB of Base64-encoded images, even with compression. Multipart avoids all of that, and because each part has its own Content-Type, it fits right into HTTP.

One thing hasn’t changed: even with images, each request to the TypeSafe API still has just one state. So bulk jobs like hotdogbench and our Box test have to be split into many separate calls, and those calls are what limit throughput. With multipart in place, though, it’s easy to see how Arrow could fit in. An Arrow IPC part could hold all of a job’s states, and a multipart/mixed response could return the answers as an Arrow IPC stream, beside a small JSON part with usage details.

An old idea, newly fast #

Multipart isn’t new. MIME introduced it in 1992, browsers have uploaded files with it since 1995, and multipart/form-data got its current specification in 2015. But its implementations have come a long way recently. Python’s standard library has only ever offered the cgi module (removed in Python 3.13) and the email package (which “is not designed for form data or large uploads at all,” as Marcel Hellkamp, who maintains the multipart library, puts it). In his benchmark, multipart and python-multipart parse uploads at about 6 to 8 GB/s, against 91 MB/s for cgi and 61 MB/s for email. When we built Jevaro’s Arrow input, we measured python-multipart at about 30 times the speed of email. In JavaScript, fetch and FormData let any browser or Node.js program send multipart in a few lines, which is how Jevaro’s JavaScript SDK sends Arrow states and images.

What’s next #

Jev speaks binary now. We have a hunch it won’t stop there. Stay tuned.

Next steps #

• Try Jevaro 0.5.0 with your own states, questions, and images, andopen an issue to tell us what you’d want from an Arrow interface to Jev.
• Check out hotdogbench.com and MotherDuck’s post onprompt_jev() for images .
• Read TypeSafe’s guide to images in Jev 1.14.