# From 101 S to 27 S: Five Ways to Boot a Heavy Python ML Service Faster on Kubernetes

> Source: <https://pub.towardsai.net/from-101-s-to-27-s-five-ways-to-boot-a-heavy-python-ml-service-faster-on-kubernetes-5246a8b6ef0b?source=rss----98111c9905da---4>
> Published: 2026-11-19 18:01:05+00:00

**Recommendation:** Precompile Python bytecode at build time and ship model weights inside the image rather than fetching them at container start; the source cut new-node boot from 101 seconds to 27 seconds.
More in the playbook "Running AI in production": <https://wpnews.pro/playbooks/production-ops>

On Kubernetes, your app does not live on one fixed server. Kubernetes starts copies of it on machines in the cluster, and starts more copies when traffic grows. Every new copy has to boot before it can answer a single request. If booting takes a minute and a half, your users wait a minute and a half.

Our example is an OCR service (it reads text from images) built on PaddleOCR, a deep-learning library, running on a GPU. It is a typical heavy Python ML app: about 7 GB of libraries and a small model.

We packaged the same app in five different ways and timed each one. This post goes from the slowest to the fastest, with the Dockerfile for each, so you can try the same on your own app.

The boot has two parts, and it helps to time them separately:

On a new node, the total is roughly the pull plus the start. Each of the five approaches below makes a different part of this faster or slower.

**How we measured.** Same app, same GPU (an NVIDIA L4), one server process per pod. We started most setups three times and report the middle value; the base + app setup was measured once. A “new node” is one that did not have the image yet. The total on a new node is the time until the container started on such a node, plus the typical container start → Ready time.

First, the app itself. It is deliberately tiny: it loads the OCR model once, then takes an image crop and returns the text it reads.

``` python
import osimport cv2import numpy as npfrom fastapi import FastAPI, UploadFilefrom paddleocr import PaddleOCR
MODEL_DIR = os.environ["MODEL_DIR"]# Loaded once at startup. The False flags skip PaddleOCR's optional page-fixing models.ocr = PaddleOCR(    text_detection_model_name="PP-OCRv6_tiny_det", text_detection_model_dir=f"{MODEL_DIR}/det",    text_recognition_model_name="PP-OCRv6_tiny_rec", text_recognition_model_dir=f"{MODEL_DIR}/rec",    use_doc_orientation_classify=False, use_doc_unwarping=False, use_textline_orientation=False,)app = FastAPI()
python
@app.get("/ready")def ready():    return {"ready": True}
python
@app.post("/ocr")def read_text(image: UploadFile):    img = cv2.imdecode(np.frombuffer(image.file.read(), np.uint8), cv2.IMREAD_COLOR)    return {"lines": ocr.predict(img)[0]["rec_texts"]}
```

It needs two groups of libraries: small web packages, and the heavy PaddleOCR libraries (several gigabytes, mostly GPU code).

```
fastapiuvicornpython-multipart  # lets FastAPI accept file uploadsboto3
# Paddle's GPU wheels (CUDA 12.6) live on Paddle's own package index--extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/paddlepaddle-gpu==3.3.1paddleocr==3.7.0
```

The usual starting point: install all the libraries into the image with pip. The model weights are not in the image; a small script downloads them from S3 every time the container starts.

```
FROM python:3.10-slim# System libraries that Paddle and OpenCV needRUN apt-get update && apt-get install -y --no-install-recommends libgomp1 libgl1 libglib2.0-0 \    && rm -rf /var/lib/apt/lists/*# Python won't write .pyc files while it runs; any .pyc must be made at build timeENV PYTHONDONTWRITEBYTECODE=1 MODEL_DIR=/models/ocrWORKDIR /appCOPY requirements-web.txt requirements-ml.txt ./RUN pip install --no-cache-dir --no-compile -r requirements-web.txt -r requirements-ml.txtCOPY app.py download_weights.py start.sh ./CMD ["./start.sh"]
bash
#!/bin/sh# Steps 1-3: fetch the model weights from S3, then replace this shell with the web server.python download_weights.py && exec uvicorn app:app --host 0.0.0.0 --port 8080
```

Two settings in this Dockerfile turn off .pyc files completely: --no-compile tells pip not to write them, and PYTHONDONTWRITEBYTECODE=1 tells Python not to save them at runtime. Many images do this to stay small and clean. The cost is that Python compiles every library file again on every start.

**Result**

Once the image is on the node, the app starts quickly: everything is on the node’s own disk. The problem is the first start on a new node, where downloading a multi-gigabyte image takes most of the time.

One extra line compiles every library file at build time, so the .pyc files ship inside the image:

```
RUN pip install --no-cache-dir --no-compile -r requirements-web.txt -r requirements-ml.txt# New: compile every library file once, at build timeRUN python -m compileall -q /usr/local/lib/python3.10/site-packages
```

When Python imports a module now, it finds a ready-made .pyc next to the .py file, checks that it still matches, and skips the compile step.

Every start gets faster. On a new node, though, the slightly bigger download eats that gain, so the total barely moves. The image pull is still the problem.

The next two approaches use EFS, so here is what it is. **EFS** (Amazon Elastic File System) is a shared network drive that AWS runs for you. It speaks NFS, the standard network file system on Linux, so a machine mounts it like any other folder, and many machines can mount the same drive at the same time. You do not choose a size: it grows as you add files, and you pay for what you use. In Kubernetes, the EFS CSI driver turns it into a volume that pods on every node can mount at once (ReadWriteMany). That is what makes it useful here: put the libraries on it once, and every pod can read them without downloading anything. The catch is that every file operation, such as opening a file or checking that it exists, goes over the network, so it is slower than the node's own disk.

So what if the image did not carry the heavy libraries at all? **EFS** (Amazon Elastic File System) is a network drive that every node in the cluster can use at the same time. We install the PaddleOCR libraries (the multi-gigabyte part) and the model weights onto EFS once, and every pod reads them from there. The image keeps only Python, a few small web packages and our code.

```
FROM python:3.10-slim# System libraries that Paddle and OpenCV needRUN apt-get update && apt-get install -y --no-install-recommends libgomp1 libgl1 libglib2.0-0 \    && rm -rf /var/lib/apt/lists/*# Only the small web libraries go in the image; Paddle and PaddleOCR are read from EFSCOPY requirements-web.txt /tmp/RUN pip install --no-cache-dir -r /tmp/requirements-web.txtENV PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=/mnt/efs/python/site-packages MODEL_DIR=/mnt/efs/models/ocrWORKDIR /appCOPY app.py .CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
```

The image installs only the small web packages. PYTHONPATH tells Python to also look for libraries on EFS, which is where PaddleOCR comes from. You put those libraries on EFS once, from any machine that has the drive mounted:

``` bash
#!/bin/sh# Run once per release, from a machine or pod that mounts the same EFS volume# read-write at /mnt/efs. Use Python 3.10 on Linux x86_64, like the pods.set -e# No .pyc here, so step 4 matches what we measured; step 5 adds them.pip install --no-cache-dir --no-compile --target /mnt/efs/python/site-packages -r requirements-ml.txtaws s3 cp --recursive s3://my-bucket/ocr-models/ /mnt/efs/models/ocr/
```

And you attach the drive to your pods:

```
# A shared EFS volume (EFS CSI driver, storage class "efs-sc").apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: ocr-efsspec:  accessModes: [ReadWriteMany]  # many pods can mount it at once  storageClassName: efs-sc  resources:    requests:      storage: 20Gi  # EFS ignores the size, but Kubernetes requires one---# Only the relevant part of the Deployment (labels, selector, GPU resources left out).apiVersion: apps/v1kind: Deploymentmetadata:  name: ocrspec:  template:    spec:      containers:        - name: ocr          image: my-registry/ocr-efs:v1          readinessProbe:            httpGet: {path: /ready, port: 8080}          volumeMounts:            - name: efs              mountPath: /mnt/efs              readOnly: true      volumes:        - name: efs          persistentVolumeClaim:            claimName: ocr-efs
```

The download problem is gone, but the start got much slower: importing the libraries took 12.8 s, against 2 s from the image. Why?

The PaddleOCR libraries are 27,967 files, and most of them are small. To import a module, Python looks for the file, opens it and reads it. On a local disk that is instant. On a network drive, every one of those steps is a trip over the network. Thousands of small trips add up to many seconds. EFS is fast at reading one big file and slow at reading many small ones.

If EFS is fast at one big file and slow at many small ones, the fix is to turn the many small files into one big file.

When we put the libraries on EFS, we also build one archive, the **hot set**. It holds every file smaller than 4 MB, plus a compiled .pyc file for every module. When a pod starts, it:

The archive is built once, every time the libraries change:

And a short start script unpacks it before the app starts:

``` bash
#!/bin/shset -e# One sequential read of the tar from EFS onto local disk. tar keeps each file's mtime,# so the .pyc inside still match their .py. Big files arrive as symlinks back to EFS.mkdir -p /tmp/hot-set && tar -xf /mnt/efs/python/hot-set.tar -C /tmp/hot-set# The local copy first; EFS second, as a fallback for anything not in the hot set.export PYTHONPATH=/tmp/hot-set:/mnt/efs/python/site-packagesexec uvicorn app:app --host 0.0.0.0 --port 8080
FROM python:3.10-slim# System libraries that Paddle and OpenCV needRUN apt-get update && apt-get install -y --no-install-recommends libgomp1 libgl1 libglib2.0-0 \    && rm -rf /var/lib/apt/lists/*# As in step 4, but PYTHONPATH is set by start.sh, after it unpacks the hot setCOPY requirements-web.txt /tmp/RUN pip install --no-cache-dir -r /tmp/requirements-web.txtENV PYTHONDONTWRITEBYTECODE=1 MODEL_DIR=/mnt/efs/models/ocrWORKDIR /appCOPY app.py .COPY 05-start.sh start.shCMD ["./start.sh"]
```

The small image stays, and the imports are back to local-disk speed. The only new cost is a few seconds to unpack one archive.

On a node that already has the image, the image setups start fastest (8.5 s with .pyc), because everything is on the local disk. But Kubernetes adds nodes exactly when traffic grows, and on a new node the 4.3 GB download dominates: every image setup needs 100.5 to 110.5 seconds before it can answer a request.

EFS removes the download but pays for it with slow imports (42.7 s on a new node). EFS plus the hot set keeps the small image and brings the imports back to local-disk speed: **27.4 s on a new node**, about a quarter of the image setups.

[From 101 S to 27 S: Five Ways to Boot a Heavy Python ML Service Faster on Kubernetes](https://pub.towardsai.net/from-101-s-to-27-s-five-ways-to-boot-a-heavy-python-ml-service-faster-on-kubernetes-5246a8b6ef0b) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.
