{"slug": "deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp", "title": "Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp", "summary": "A new end-to-end guide shows how to deploy quantized GGUF vision models, specifically the Nanonets-OCR-s-Q4_K_M.gguf weights paired with an mmproj-F16.gguf multimodal projector from the unsloth/Nanonets-OCR-s-GGUF repository, to Amazon SageMaker using a bring-your-own-container approach. The guide wraps llama.cpp in a FastAPI inference server with lazy model loading to speed container startup, and pushes the image to Amazon ECR before creating the SageMaker endpoint. The approach targets the memory efficiency of 4-bit quantization for OCR and visual reasoning workloads that SageMaker's pre-built environments do not natively support.", "body_md": "The “Local LLM” revolution has given us incredible tools like llama.cpp and the GGUF format, allowing us to run high-performance Vision models on modest hardware. But there is a massive leap between running an OCR model in a local terminal and serving it as a production-grade, auto-scaling API on the cloud.\n\nWhen you need to move beyond “it works on my machine,” **Amazon SageMaker** is the gold standard. However, SageMaker’s pre-built environments don’t always support the cutting-edge efficiency of quantized GGUF vision models.\n\nTo bridge this gap, we must go **BYOC (Bring Your Own Container)**.\n\nBy wrapping the raw speed of llama.cpp inside the modern, asynchronous architecture of **FastAPI**, we can create a custom inference engine that SageMaker treats like any other managed endpoint. This approach gives you the best of both worlds: the extreme memory efficiency of 4-bit quantization and the industrial-strength reliability of AWS.\n\nIn this end-to-end guide, we will build a custom OCR solution that handles complex visual reasoning without the “GPU tax” of unquantized models. We’ll cover:\n\nFirst, verify Docker is running and install the Hugging Face Hub library to pull our model weights.\n\n```\n!docker version!pip install huggingface-hub\n```\n\nWe’ll dynamically pull your AWS account details to ensure this script is portable across regions.\n\n***Note:*** *Ensure you replace* *\"your-bucket-name\"with an actual S3 bucket in your account.*\n\n``` python\nimport subprocessS3_BUCKET_NAME = \"bucket_name\" # Please change the Bucketname when you running this cellECR_REPOSITORY_NAME = \"ecr_repo_name\"AWS_ACCOUNT_ID = subprocess.check_output(\"aws sts get-caller-identity --query Account --output text\", shell=True).decode().strip()AWS_REGION = subprocess.check_output(\"aws configure get region\", shell=True).decode().strip()ECR_URI=f\"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com\"ECR_REPOSITORY_URI=f\"{ECR_URI}/{ECR_REPOSITORY_NAME}\"IMAGE_TAG=\"latest\"MODEL_NAME=\"Nanonets-OCR-s-Q4_K_M.gguf\"\n```\n\n**2. Downloading GGUF Model Files**\n\nWe need two files: the **quantized weights** and the **multimodal projector** (the “eyes” of the model). We’ll use the Nanonets-OCR-s-GGUF, which is highly efficient for document text extraction.\n\n``` python\nfrom huggingface_hub import hf_hub_downloadimport osGGUF_NAME = \"Nanonets-OCR-s-Q4_K_M.gguf\"MM_PROJ_NAME = \"mmproj-F16.gguf\"MODEL_PATH = hf_hub_download(    repo_id=\"unsloth/Nanonets-OCR-s-GGUF\",    filename=GGUF_NAME,    local_dir=\"~/SageMaker/workspace/\",    local_dir_use_symlinks=False)MMPROJ_PATH = hf_hub_download(    repo_id=\"unsloth/Nanonets-OCR-s-GGUF\",    filename=MMPROJ_NAME,    local_dir=r\"/home/ec2-user/SageMaker/workspace\",    local_dir_use_symlinks=False,)print(\"✅ Model:\", MODEL_PATH)print(\"📦 Size:\", round(os.path.getsize(MODEL_PATH)/1024/1024, 2), \"MB\")\n```\n\n**3. Implementing the FastAPI Inference Server**\n\nOur main.py handles the SageMaker lifecycle. We use **Lazy Loading** for the model to ensure the container starts quickly, though the very first request will bear the \"Cold Start\" load time.\n\n``` python\n%%writefile main.pyimport os, base64, io, loggingfrom fastapi import FastAPI, Request, Responsefrom PIL import Image, ImageOps, ImageEnhancefrom llama_cpp import Llamafrom llama_cpp.llama_chat_format import MiniCPMv26ChatHandler# Setup logging to see output in SageMaker/Docker logslogging.basicConfig(level=logging.INFO)logger = logging.getLogger(__name__)app = FastAPI()# --- DIAGNOSTIC LOGGING BLOCK ---MODEL_DIR = \"/opt/ml/model\"logger.info(f\"Checking directory: {MODEL_DIR}\")if os.path.exists(MODEL_DIR):    # List all files and subdirectories found in the model path    contents = []    for root, dirs, files in os.walk(MODEL_DIR):        for name in files:            contents.append(os.path.join(root, name))        if contents:        logger.info(f\"✅ FOUND {len(contents)} files in {MODEL_DIR}:\")        for f in contents:            # Print file size to ensure it's not a 0-byte corrupted download            size_mb = os.path.getsize(f) / (1024 * 1024)            logger.info(f\" - {f} ({size_mb:.2f} MB)\")    else:        logger.error(f\"❌ {MODEL_DIR} exists but is EMPTY.\")else:    logger.error(f\"❌ {MODEL_DIR} directory DOES NOT EXIST.\")# SageMaker unpacks S3 model.tar.gz into /opt/ml/modelMODEL_PATH = \"/opt/ml/model/Nanonets-OCR-s-Q4_K_M.gguf\"MMPROJ_PATH = \"/opt/ml/model/mmproj-F16.gguf\"# Global model variable for lazy loadingllm = Nonedef load_model():    global llm    if llm is None:        chat_handler = MiniCPMv26ChatHandler(clip_model_path=MMPROJ_PATH, verbose=False)        llm = Llama(            model_path=MODEL_PATH,            chat_handler=chat_handler,            n_ctx=2096, #max for t4 is 128000 (128k) , we have limited to 2096 (2k context)            n_gpu_layers=-1,             verbose=False        )@app.get(\"/ping\")async def ping():    return Response(status_code=200)@app.post(\"/invocations\")async def invocations(request: Request):    load_model()         payload = await request.json()        # 1. Standardize input to a list of tasks    # This supports {\"image\": \"...\"} OR {\"images\": [\"...\", \"...\"]}    raw_images = payload.get(\"images\", [])    if not raw_images and \"image\" in payload:        raw_images = [payload[\"image\"]]            prompt = payload.get(\"prompt\", \"Extract all text from this image\")    max_tokens = int(payload.get(\"max_tokens\", 512))    temperature = float(payload.get(\"temperature\", 0.0))        if not raw_images:        return Response(content=\"Missing 'image' or 'images' in payload\", status_code=400)    results = []    # 2. Process the batch sequentially    for i, image_b64 in enumerate(raw_images):        try:            out = llm.create_chat_completion(                messages=[{                    \"role\": \"user\",                    \"content\": [                        {\"type\": \"image_url\", \"image_url\": {\"url\": f\"data:image/jpeg;base64,{image_b64}\"}},                        {\"type\": \"text\", \"text\": prompt},                    ],                }],                max_tokens=max_tokens,                temperature=temperature,            )                        results.append({                \"index\": i,                \"text\": out[\"choices\"][0][\"message\"][\"content\"].strip(),                \"finish_reason\": out[\"choices\"][0].get(\"finish_reason\"),                \"usage\": out.get(\"usage\")            })        except Exception as e:            results.append({\"index\": i, \"error\": str(e)})        # 3. Return the batch result    return {\"results\": results}\n```\n\n**4. Building the Docker Image (BYOC)**\n\nWe use a multi-stage build to keep the runtime image lean. We pull a pre-compiled llama-cpp-python wheel for **CUDA 12.4** to avoid lengthy compilation times during the build.\n\n```\n%%writefile DockerfileFROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builderENV DEBIAN_FRONTEND=noninteractiveRUN apt-get update && apt-get install -y python3-pip python3-devRUN pip install --no-cache-dir --upgrade pipRUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124RUN pip install --no-cache-dir fastapi uvicorn pillowFROM nvidia/cuda:12.4.1-runtime-ubuntu22.04RUN apt-get update && apt-get install -y python3 python3-pip && rm -rf /var/lib/apt/lists/*WORKDIR /appCOPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packagesCOPY --from=builder /usr/local/bin /usr/local/binCOPY main.py .RUN mkdir -p /opt/ml/modelEXPOSE 8080ENTRYPOINT [\"uvicorn\", \"main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8080\"]\n```\n\n**5. Packaging and Pushing**\n\n**Crucial Step:** When creating the model.tar.gz, ensure the files are at the root of the archive so SageMaker finds them at /opt/ml/model/.\n\n```\n# Package artifacts!tar -cvzf model.tar.gz Nanonets-OCR-s-Q4_K_M.gguf mmproj-F16.gguf# Upload to S3!aws s3 cp model.tar.gz s3://{S3_BUCKET_NAME}/nanonets-ocr/model.tar.gz# Build and Push Container!aws ecr get-login-password --region {AWS_REGION} | docker login --username AWS --password-stdin {ECR_URI}!docker build -t \"$ECR_REPOSITORY_NAME\" .!docker tag \"$ECR_REPOSITORY_NAME\" \"$ECR_REPOSITORY_URI\":\"$IMAGE_TAG\"!docker push \"$ECR_REPOSITORY_URI\":\"$IMAGE_TAG\"Upload to S3 bucket\n```\n\n**Building and pushing image to ECR**\n\n```\n!docker build -t \"$ECR_REPOSITORY_URI\":\"$IMAGE_TAG\" .!docker push \"$ECR_REPOSITORY_URI\":\"$IMAGE_TAG\"\n```\n\n**6. Deployment to SageMaker Endpoint**\n\nWe deploy to an ml.g4dn.xlarge instance. Even though GGUF is efficient on CPUs, using the **NVIDIA T4 GPU** ensures low-latency OCR performance.\n\n``` python\nimport sagemakerfrom sagemaker.model import Modelrole = sagemaker.get_execution_role()container_uri = f\"{ECR_REPOSITORY_URI}:{IMAGE_TAG}\"model_s3_url = f\"s3://{S3_BUCKET_NAME}/nanonets-ocr/model.tar.gz\"ocr_model = Model(    image_uri=container_uri,    model_data=model_s3_url,    role=role,    name=\"nanonets-ocr-gguf-model\")predictor = ocr_model.deploy(    initial_instance_count=1,    instance_type=\"ml.g4dn.xlarge\",    endpoint_name=\"vision-ocr-endpoint\",    container_startup_health_check_timeout=600)\n```\n\n**7. Conclusion**\n\nBy combining **llama.cpp** and **SageMaker BYOC**, you’ve built a production-grade OCR pipeline that is both high-performance and cost-aware. You are no longer restricted by pre-built cloud containers and can leverage the latest quantized multimodal models as soon as they drop on Hugging Face.\n\nDon’t forget to clean up:\n\n```\nsagemaker_c = boto3.client('sagemaker')sagemaker_c.delete_endpoint(EndpointName=endpoint_name)sagemaker_c.get_waiter('endpoint_deleted').wait(EndpointName=endpoint_name)print(\"Endpoint + model deleted!\")\n```\n\n[Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp](https://pub.towardsai.net/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp-fb6948e79e10) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp", "canonical_source": "https://pub.towardsai.net/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp-fb6948e79e10?source=rss----98111c9905da---4", "published_at": "2026-10-22 04:53:35+00:00", "updated_at": "2026-10-22 05:10:19.364533+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "computer-vision", "large-language-models", "ai-products"], "entities": ["Amazon SageMaker", "llama.cpp", "GGUF", "FastAPI", "Amazon Web Services", "Nanonets-OCR-s", "unsloth", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp", "markdown": "https://wpnews.pro/news/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp.md", "text": "https://wpnews.pro/news/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp.txt", "jsonld": "https://wpnews.pro/news/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp.jsonld"}}