I added AI background removal to my image converter in a week, in Rust, no Python

A solo developer added AI background removal to Convertify, a free image converter, in one week using only Rust and no Python. The backend runs ONNX models natively via the ort crate, avoiding the need for a separate Python process. Challenges included handling a 171 MB model file and adapting to ort's non-Send+Sync errors and mutable session requirements.

Part of an ongoing build-in-public series on Convertify, a free image/file converter I build solo. This week: background removal. The honest version, with the walls I hit. Most "remove image background" tutorials end with pip install rembg and a happy screenshot. Mine started with a constraint: my whole backend is Rust, and I did not want to bolt a Python process onto it just to run one model. Here is how the week went. The good parts, and the three or four times I stared at a compiler error wondering if the constraint was worth it. Convertify is a free image converter. The backend is Rust + Axum + libvips , the model has to run CPU-only on a modest VPS , and there is no GPU anywhere in the budget. The obvious path for background removal is rembg , which is excellent, but it is Python and ships as a separate server process. Adding it would mean a second runtime, a second thing to deploy, a second thing to crash at 3am. So the question for the week was simple: can I run the same models rembg uses, but natively in Rust? Short answer: yes. rembg is, under the hood, just ONNX models plus some image pre and post processing. The models u2net, isnet, silueta are all .onnx files. If I can run ONNX in Rust and do the image work in libvips which I already have , there is no Python in the picture at all. The pipeline for background removal is not magic, it is five boring steps: Steps 1, 4, 5 are libvips, which I already use everywhere. Step 3 is ONNX Runtime via the ort https://github.com/pykeio/ort crate. Step 2 is a tight Rust loop. No Python anywhere. dependencies ort = { version = "=2.0.0-rc.12", features = "download-binaries" } The download-binaries feature pulls a CPU build of ONNX Runtime at build time, so there is nothing to install on the box. That alone deleted half the "deploy a Python service" anxiety. I grabbed isnet-general-use.onnx from the rembg releases, expecting ~44 MB. What landed was 171 MB . My first thought was a broken download or an HTML error page renamed to .onnx . Quick check: file models/isnet-general-use.onnx head -c 200 models/isnet-general-use.onnx | xxd | head The header showed a real pytorch 1.13.1 signature and tensor names like input image and conv in.weight . So it was a valid ONNX model, just heavier than the name suggested. Lesson: verify the file is actually what you think before you spend an hour debugging "why is RAM so high." ort errors are not Send + Sync First compile against anyhow and I get hit with this: the trait Sync is not implemented for NonNull<OrtSessionOptions required for anyhow::Error to implement From<ort::Error<SessionBuilder anyhow::Error wants Send + Sync . The ort error type holds raw pointers into the ONNX Runtime C++ session, which are not Sync . So ? straight into anyhow does not compile. The fix is to stringify the error at the boundary. Display gives you a String , and String is Send + Sync : js let session = build model path, intra threads .map err |e| anyhow "ort session init: {e}" ?; The pointer never leaves, only the message does. Once I understood why , the pattern was mechanical: every ort ? that crosses into anyhow gets a .map err |e| anyhow "...: {e}" ? . run takes &mut self This one actually changed my architecture. In this ort version, Session::run takes &mut self . I had the session behind an Arc in my Axum app state so it could be shared. You cannot get &mut through an Arc . cannot borrow self.session as mutable, as it is behind a & reference Options were a session pool, or a Mutex . Since my traffic is low and I gate inference to one at a time anyway, I wrapped the session in a Mutex : pub struct BgRemover { session: Mutex<Session , } remove &self stays &self , so Arc<BgRemover still works in app state. The Mutex hands out the &mut for the single inference call. With a one-permit semaphore in front, the mutex never even contends. When traffic grows, the upgrade path is a pool of sessions, but that is a future-me problem. mut VipsImage is not Send libvips image pointers are not Send , which means they cannot be held across an .await . If I ran inference directly in the async handler, the borrow checker would stop me, and even if it did not, a multi-second CPU inference on an async worker thread would freeze the whole runtime. The answer is spawn blocking . The entire libvips + inference chain runs on a dedicated blocking thread and returns finished PNG bytes which are Send : js let png = tokio::task::spawn blocking move || { let permit = permit; // hold the semaphore for the whole job remover.remove &bytes } .await??; Every VipsImage is created and dropped inside that closure, never crossing an await point. The async runtime stays free to serve everything else while one image is being cut out. Because the handler returns the PNG straight in the HTTP response, the image is never written to disk . It comes in as multipart bytes, gets processed in memory, and the result streams back. Nothing is stored, nothing is queued, nothing to clean up. I did not plan that as a feature, it fell out of the architecture. But "your photo is processed in memory and never saved" is a genuinely strong thing to be able to say, and it is true, not marketing. Yes. First real test through Postman with a HEIC photo: 200 OK, transparent PNG out. The model is ISNet the IS-Net dichotomous segmentation architecture , and on clean subjects, products, people, logos, the cutout is sharp. rembg is "just" ONNX + image ops. If you already have an image library, you can skip the Python entirely with ort . ort 2.0 API churns between rc versions. Pin the exact version and expect to fix one or two method names. spawn blocking is not optional for CPU-heavy, non- Send work. It is the whole reason the server stays responsive.If you want to see the result, background removal is live and free no signup, no watermark on Convertify https://convertifyapp.net/remove-background . Upload a photo, get a transparent PNG. It runs the exact pipeline above. Next week: turning one tool into a set of use-case pages passport photos, product shots without drowning in duplicate content. That one is more SEO than Rust, but the build-in-public log continues. What would you have done differently on the &mut self session problem? A pool, a mutex, something smarter? Curious how others handle shared ONNX sessions under load.