Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp

A new end-to-end guide shows how to deploy quantized GGUF vision models, specifically the Nanonets-OCR-s-Q4_K_M.gguf weights paired with an mmproj-F16.gguf multimodal projector from the unsloth/Nanonets-OCR-s-GGUF repository, to Amazon SageMaker using a bring-your-own-container approach. The guide wraps llama.cpp in a FastAPI inference server with lazy model loading to speed container startup, and pushes the image to Amazon ECR before creating the SageMaker endpoint. The approach targets the memory efficiency of 4-bit quantization for OCR and visual reasoning workloads that SageMaker's pre-built environments do not natively support.

The “Local LLM” revolution has given us incredible tools like llama.cpp and the GGUF format, allowing us to run high-performance Vision models on modest hardware. But there is a massive leap between running an OCR model in a local terminal and serving it as a production-grade, auto-scaling API on the cloud. When you need to move beyond “it works on my machine,” Amazon SageMaker is the gold standard. However, SageMaker’s pre-built environments don’t always support the cutting-edge efficiency of quantized GGUF vision models. To bridge this gap, we must go BYOC Bring Your Own Container . By wrapping the raw speed of llama.cpp inside the modern, asynchronous architecture of FastAPI , we can create a custom inference engine that SageMaker treats like any other managed endpoint. This approach gives you the best of both worlds: the extreme memory efficiency of 4-bit quantization and the industrial-strength reliability of AWS. In this end-to-end guide, we will build a custom OCR solution that handles complex visual reasoning without the “GPU tax” of unquantized models. We’ll cover: First, verify Docker is running and install the Hugging Face Hub library to pull our model weights. docker version pip install huggingface-hub We’ll dynamically pull your AWS account details to ensure this script is portable across regions. Note: Ensure you replace "your-bucket-name"with an actual S3 bucket in your account. python import subprocessS3 BUCKET NAME = "bucket name" Please change the Bucketname when you running this cellECR REPOSITORY NAME = "ecr repo name"AWS ACCOUNT ID = subprocess.check output "aws sts get-caller-identity --query Account --output text", shell=True .decode .strip AWS REGION = subprocess.check output "aws configure get region", shell=True .decode .strip ECR URI=f"{AWS ACCOUNT ID}.dkr.ecr.{AWS REGION}.amazonaws.com"ECR REPOSITORY URI=f"{ECR URI}/{ECR REPOSITORY NAME}"IMAGE TAG="latest"MODEL NAME="Nanonets-OCR-s-Q4 K M.gguf" 2. Downloading GGUF Model Files We need two files: the quantized weights and the multimodal projector the “eyes” of the model . We’ll use the Nanonets-OCR-s-GGUF, which is highly efficient for document text extraction. python from huggingface hub import hf hub downloadimport osGGUF NAME = "Nanonets-OCR-s-Q4 K M.gguf"MM PROJ NAME = "mmproj-F16.gguf"MODEL PATH = hf hub download repo id="unsloth/Nanonets-OCR-s-GGUF", filename=GGUF NAME, local dir="~/SageMaker/workspace/", local dir use symlinks=False MMPROJ PATH = hf hub download repo id="unsloth/Nanonets-OCR-s-GGUF", filename=MMPROJ NAME, local dir=r"/home/ec2-user/SageMaker/workspace", local dir use symlinks=False, print "✅ Model:", MODEL PATH print "📦 Size:", round os.path.getsize MODEL PATH /1024/1024, 2 , "MB" 3. Implementing the FastAPI Inference Server Our main.py handles the SageMaker lifecycle. We use Lazy Loading for the model to ensure the container starts quickly, though the very first request will bear the "Cold Start" load time. python %%writefile main.pyimport os, base64, io, loggingfrom fastapi import FastAPI, Request, Responsefrom PIL import Image, ImageOps, ImageEnhancefrom llama cpp import Llamafrom llama cpp.llama chat format import MiniCPMv26ChatHandler Setup logging to see output in SageMaker/Docker logslogging.basicConfig level=logging.INFO logger = logging.getLogger name app = FastAPI --- DIAGNOSTIC LOGGING BLOCK ---MODEL DIR = "/opt/ml/model"logger.info f"Checking directory: {MODEL DIR}" if os.path.exists MODEL DIR : List all files and subdirectories found in the model path contents = for root, dirs, files in os.walk MODEL DIR : for name in files: contents.append os.path.join root, name if contents: logger.info f"✅ FOUND {len contents } files in {MODEL DIR}:" for f in contents: Print file size to ensure it's not a 0-byte corrupted download size mb = os.path.getsize f / 1024 1024 logger.info f" - {f} {size mb:.2f} MB " else: logger.error f"❌ {MODEL DIR} exists but is EMPTY." else: logger.error f"❌ {MODEL DIR} directory DOES NOT EXIST." SageMaker unpacks S3 model.tar.gz into /opt/ml/modelMODEL PATH = "/opt/ml/model/Nanonets-OCR-s-Q4 K M.gguf"MMPROJ PATH = "/opt/ml/model/mmproj-F16.gguf" Global model variable for lazy loadingllm = Nonedef load model : global llm if llm is None: chat handler = MiniCPMv26ChatHandler clip model path=MMPROJ PATH, verbose=False llm = Llama model path=MODEL PATH, chat handler=chat handler, n ctx=2096, max for t4 is 128000 128k , we have limited to 2096 2k context n gpu layers=-1, verbose=False @app.get "/ping" async def ping : return Response status code=200 @app.post "/invocations" async def invocations request: Request : load model payload = await request.json 1. Standardize input to a list of tasks This supports {"image": "..."} OR {"images": "...", "..." } raw images = payload.get "images", if not raw images and "image" in payload: raw images = payload "image" prompt = payload.get "prompt", "Extract all text from this image" max tokens = int payload.get "max tokens", 512 temperature = float payload.get "temperature", 0.0 if not raw images: return Response content="Missing 'image' or 'images' in payload", status code=400 results = 2. Process the batch sequentially for i, image b64 in enumerate raw images : try: out = llm.create chat completion messages= { "role": "user", "content": {"type": "image url", "image url": {"url": f"data:image/jpeg;base64,{image b64}"}}, {"type": "text", "text": prompt}, , } , max tokens=max tokens, temperature=temperature, results.append { "index": i, "text": out "choices" 0 "message" "content" .strip , "finish reason": out "choices" 0 .get "finish reason" , "usage": out.get "usage" } except Exception as e: results.append {"index": i, "error": str e } 3. Return the batch result return {"results": results} 4. Building the Docker Image BYOC We use a multi-stage build to keep the runtime image lean. We pull a pre-compiled llama-cpp-python wheel for CUDA 12.4 to avoid lengthy compilation times during the build. %%writefile DockerfileFROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builderENV DEBIAN FRONTEND=noninteractiveRUN apt-get update && apt-get install -y python3-pip python3-devRUN pip install --no-cache-dir --upgrade pipRUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124RUN pip install --no-cache-dir fastapi uvicorn pillowFROM nvidia/cuda:12.4.1-runtime-ubuntu22.04RUN apt-get update && apt-get install -y python3 python3-pip && rm -rf /var/lib/apt/lists/ WORKDIR /appCOPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packagesCOPY --from=builder /usr/local/bin /usr/local/binCOPY main.py .RUN mkdir -p /opt/ml/modelEXPOSE 8080ENTRYPOINT "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080" 5. Packaging and Pushing Crucial Step: When creating the model.tar.gz, ensure the files are at the root of the archive so SageMaker finds them at /opt/ml/model/. Package artifacts tar -cvzf model.tar.gz Nanonets-OCR-s-Q4 K M.gguf mmproj-F16.gguf Upload to S3 aws s3 cp model.tar.gz s3://{S3 BUCKET NAME}/nanonets-ocr/model.tar.gz Build and Push Container aws ecr get-login-password --region {AWS REGION} | docker login --username AWS --password-stdin {ECR URI} docker build -t "$ECR REPOSITORY NAME" . docker tag "$ECR REPOSITORY NAME" "$ECR REPOSITORY URI":"$IMAGE TAG" docker push "$ECR REPOSITORY URI":"$IMAGE TAG"Upload to S3 bucket Building and pushing image to ECR docker build -t "$ECR REPOSITORY URI":"$IMAGE TAG" . docker push "$ECR REPOSITORY URI":"$IMAGE TAG" 6. Deployment to SageMaker Endpoint We deploy to an ml.g4dn.xlarge instance. Even though GGUF is efficient on CPUs, using the NVIDIA T4 GPU ensures low-latency OCR performance. python import sagemakerfrom sagemaker.model import Modelrole = sagemaker.get execution role container uri = f"{ECR REPOSITORY URI}:{IMAGE TAG}"model s3 url = f"s3://{S3 BUCKET NAME}/nanonets-ocr/model.tar.gz"ocr model = Model image uri=container uri, model data=model s3 url, role=role, name="nanonets-ocr-gguf-model" predictor = ocr model.deploy initial instance count=1, instance type="ml.g4dn.xlarge", endpoint name="vision-ocr-endpoint", container startup health check timeout=600 7. Conclusion By combining llama.cpp and SageMaker BYOC , you’ve built a production-grade OCR pipeline that is both high-performance and cost-aware. You are no longer restricted by pre-built cloud containers and can leverage the latest quantized multimodal models as soon as they drop on Hugging Face. Don’t forget to clean up: sagemaker c = boto3.client 'sagemaker' sagemaker c.delete endpoint EndpointName=endpoint name sagemaker c.get waiter 'endpoint deleted' .wait EndpointName=endpoint name print "Endpoint + model deleted " Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp https://pub.towardsai.net/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp-fb6948e79e10 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.