The “Local LLM” revolution has given us incredible tools like llama.cpp and the GGUF format, allowing us to run high-performance Vision models on modest hardware. But there is a massive leap between running an OCR model in a local terminal and serving it as a production-grade, auto-scaling API on the cloud.
When you need to move beyond “it works on my machine,” Amazon SageMaker is the gold standard. However, SageMaker’s pre-built environments don’t always support the cutting-edge efficiency of quantized GGUF vision models.
To bridge this gap, we must go BYOC (Bring Your Own Container).
By wrapping the raw speed of llama.cpp inside the modern, asynchronous architecture of FastAPI, we can create a custom inference engine that SageMaker treats like any other managed endpoint. This approach gives you the best of both worlds: the extreme memory efficiency of 4-bit quantization and the industrial-strength reliability of AWS.
In this end-to-end guide, we will build a custom OCR solution that handles complex visual reasoning without the “GPU tax” of unquantized models. We’ll cover:
First, verify Docker is running and install the Hugging Face Hub library to pull our model weights.
!docker version!pip install huggingface-hub
We’ll dynamically pull your AWS account details to ensure this script is portable across regions.
Note: Ensure you replace "your-bucket-name"with an actual S3 bucket in your account.
import subprocessS3_BUCKET_NAME = "bucket_name" # Please change the Bucketname when you running this cellECR_REPOSITORY_NAME = "ecr_repo_name"AWS_ACCOUNT_ID = subprocess.check_output("aws sts get-caller-identity --query Account --output text", shell=True).decode().strip()AWS_REGION = subprocess.check_output("aws configure get region", shell=True).decode().strip()ECR_URI=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com"ECR_REPOSITORY_URI=f"{ECR_URI}/{ECR_REPOSITORY_NAME}"IMAGE_TAG="latest"MODEL_NAME="Nanonets-OCR-s-Q4_K_M.gguf"
2. Down GGUF Model Files
We need two files: the quantized weights and the multimodal projector (the “eyes” of the model). We’ll use the Nanonets-OCR-s-GGUF, which is highly efficient for document text extraction.
from huggingface_hub import hf_hub_downloadimport osGGUF_NAME = "Nanonets-OCR-s-Q4_K_M.gguf"MM_PROJ_NAME = "mmproj-F16.gguf"MODEL_PATH = hf_hub_download( repo_id="unsloth/Nanonets-OCR-s-GGUF", filename=GGUF_NAME, local_dir="~/SageMaker/workspace/", local_dir_use_symlinks=False)MMPROJ_PATH = hf_hub_download( repo_id="unsloth/Nanonets-OCR-s-GGUF", filename=MMPROJ_NAME, local_dir=r"/home/ec2-user/SageMaker/workspace", local_dir_use_symlinks=False,)print("✅ Model:", MODEL_PATH)print("📦 Size:", round(os.path.getsize(MODEL_PATH)/1024/1024, 2), "MB")
3. Implementing the FastAPI Inference Server
Our main.py handles the SageMaker lifecycle. We use **Lazy ** for the model to ensure the container starts quickly, though the very first request will bear the "Cold Start" load time.
%%writefile main.pyimport os, base64, io, loggingfrom fastapi import FastAPI, Request, Responsefrom PIL import Image, ImageOps, ImageEnhancefrom llama_cpp import Llamafrom llama_cpp.llama_chat_format import MiniCPMv26ChatHandler# Setup logging to see output in SageMaker/Docker logslogging.basicConfig(level=logging.INFO)logger = logging.getLogger(__name__)app = FastAPI()# --- DIAGNOSTIC LOGGING BLOCK ---MODEL_DIR = "/opt/ml/model"logger.info(f"Checking directory: {MODEL_DIR}")if os.path.exists(MODEL_DIR): # List all files and subdirectories found in the model path contents = [] for root, dirs, files in os.walk(MODEL_DIR): for name in files: contents.append(os.path.join(root, name)) if contents: logger.info(f"✅ FOUND {len(contents)} files in {MODEL_DIR}:") for f in contents: # Print file size to ensure it's not a 0-byte corrupted download size_mb = os.path.getsize(f) / (1024 * 1024) logger.info(f" - {f} ({size_mb:.2f} MB)") else: logger.error(f"❌ {MODEL_DIR} exists but is EMPTY.")else: logger.error(f"❌ {MODEL_DIR} directory DOES NOT EXIST.")# SageMaker unpacks S3 model.tar.gz into /opt/ml/modelMODEL_PATH = "/opt/ml/model/Nanonets-OCR-s-Q4_K_M.gguf"MMPROJ_PATH = "/opt/ml/model/mmproj-F16.gguf"# Global model variable for lazy llm = Nonedef load_model(): global llm if llm is None: chat_handler = MiniCPMv26ChatHandler(clip_model_path=MMPROJ_PATH, verbose=False) llm = Llama( model_path=MODEL_PATH, chat_handler=chat_handler, n_ctx=2096, #max for t4 is 128000 (128k) , we have limited to 2096 (2k context) n_gpu_layers=-1, verbose=False )@app.get("/ping")async def ping(): return Response(status_code=200)@app.post("/invocations")async def invocations(request: Request): load_model() payload = await request.json() # 1. Standardize input to a list of tasks # This supports {"image": "..."} OR {"images": ["...", "..."]} raw_images = payload.get("images", []) if not raw_images and "image" in payload: raw_images = [payload["image"]] prompt = payload.get("prompt", "Extract all text from this image") max_tokens = int(payload.get("max_tokens", 512)) temperature = float(payload.get("temperature", 0.0)) if not raw_images: return Response(content="Missing 'image' or 'images' in payload", status_code=400) results = [] # 2. Process the batch sequentially for i, image_b64 in enumerate(raw_images): try: out = llm.create_chat_completion( messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}, {"type": "text", "text": prompt}, ], }], max_tokens=max_tokens, temperature=temperature, ) results.append({ "index": i, "text": out["choices"][0]["message"]["content"].strip(), "finish_reason": out["choices"][0].get("finish_reason"), "usage": out.get("usage") }) except Exception as e: results.append({"index": i, "error": str(e)}) # 3. Return the batch result return {"results": results}
4. Building the Docker Image (BYOC)
We use a multi-stage build to keep the runtime image lean. We pull a pre-compiled llama-cpp-python wheel for CUDA 12.4 to avoid lengthy compilation times during the build.
%%writefile DockerfileFROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builderENV DEBIAN_FRONTEND=noninteractiveRUN apt-get update && apt-get install -y python3-pip python3-devRUN pip install --no-cache-dir --upgrade pipRUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124RUN pip install --no-cache-dir fastapi uvicorn pillowFROM nvidia/cuda:12.4.1-runtime-ubuntu22.04RUN apt-get update && apt-get install -y python3 python3-pip && rm -rf /var/lib/apt/lists/*WORKDIR /appCOPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packagesCOPY --from=builder /usr/local/bin /usr/local/binCOPY main.py .RUN mkdir -p /opt/ml/modelEXPOSE 8080ENTRYPOINT ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
5. Packaging and Pushing
Crucial Step: When creating the model.tar.gz, ensure the files are at the root of the archive so SageMaker finds them at /opt/ml/model/.
Building and pushing image to ECR
!docker build -t "$ECR_REPOSITORY_URI":"$IMAGE_TAG" .!docker push "$ECR_REPOSITORY_URI":"$IMAGE_TAG"
6. Deployment to SageMaker Endpoint
We deploy to an ml.g4dn.xlarge instance. Even though GGUF is efficient on CPUs, using the NVIDIA T4 GPU ensures low-latency OCR performance.
import sagemakerfrom sagemaker.model import Modelrole = sagemaker.get_execution_role()container_uri = f"{ECR_REPOSITORY_URI}:{IMAGE_TAG}"model_s3_url = f"s3://{S3_BUCKET_NAME}/nanonets-ocr/model.tar.gz"ocr_model = Model( image_uri=container_uri, model_data=model_s3_url, role=role, name="nanonets-ocr-gguf-model")predictor = ocr_model.deploy( initial_instance_count=1, instance_type="ml.g4dn.xlarge", endpoint_name="vision-ocr-endpoint", container_startup_health_check_timeout=600)
7. Conclusion
By combining llama.cpp and SageMaker BYOC, you’ve built a production-grade OCR pipeline that is both high-performance and cost-aware. You are no longer restricted by pre-built cloud containers and can leverage the latest quantized multimodal models as soon as they drop on Hugging Face.
Don’t forget to clean up:
sagemaker_c = boto3.client('sagemaker')sagemaker_c.delete_endpoint(EndpointName=endpoint_name)sagemaker_c.get_waiter('endpoint_deleted').wait(EndpointName=endpoint_name)print("Endpoint + model deleted!")
Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.