# Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp

> Source: <https://pub.towardsai.net/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp-fb6948e79e10?source=rss----98111c9905da---4>
> Published: 2026-10-22 04:53:35+00:00

The “Local LLM” revolution has given us incredible tools like llama.cpp and the GGUF format, allowing us to run high-performance Vision models on modest hardware. But there is a massive leap between running an OCR model in a local terminal and serving it as a production-grade, auto-scaling API on the cloud.

When you need to move beyond “it works on my machine,” **Amazon SageMaker** is the gold standard. However, SageMaker’s pre-built environments don’t always support the cutting-edge efficiency of quantized GGUF vision models.

To bridge this gap, we must go **BYOC (Bring Your Own Container)**.

By wrapping the raw speed of llama.cpp inside the modern, asynchronous architecture of **FastAPI**, we can create a custom inference engine that SageMaker treats like any other managed endpoint. This approach gives you the best of both worlds: the extreme memory efficiency of 4-bit quantization and the industrial-strength reliability of AWS.

In this end-to-end guide, we will build a custom OCR solution that handles complex visual reasoning without the “GPU tax” of unquantized models. We’ll cover:

First, verify Docker is running and install the Hugging Face Hub library to pull our model weights.

```
!docker version!pip install huggingface-hub
```

We’ll dynamically pull your AWS account details to ensure this script is portable across regions.

***Note:*** *Ensure you replace* *"your-bucket-name"with an actual S3 bucket in your account.*

``` python
import subprocessS3_BUCKET_NAME = "bucket_name" # Please change the Bucketname when you running this cellECR_REPOSITORY_NAME = "ecr_repo_name"AWS_ACCOUNT_ID = subprocess.check_output("aws sts get-caller-identity --query Account --output text", shell=True).decode().strip()AWS_REGION = subprocess.check_output("aws configure get region", shell=True).decode().strip()ECR_URI=f"{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com"ECR_REPOSITORY_URI=f"{ECR_URI}/{ECR_REPOSITORY_NAME}"IMAGE_TAG="latest"MODEL_NAME="Nanonets-OCR-s-Q4_K_M.gguf"
```

**2. Downloading GGUF Model Files**

We need two files: the **quantized weights** and the **multimodal projector** (the “eyes” of the model). We’ll use the Nanonets-OCR-s-GGUF, which is highly efficient for document text extraction.

``` python
from huggingface_hub import hf_hub_downloadimport osGGUF_NAME = "Nanonets-OCR-s-Q4_K_M.gguf"MM_PROJ_NAME = "mmproj-F16.gguf"MODEL_PATH = hf_hub_download(    repo_id="unsloth/Nanonets-OCR-s-GGUF",    filename=GGUF_NAME,    local_dir="~/SageMaker/workspace/",    local_dir_use_symlinks=False)MMPROJ_PATH = hf_hub_download(    repo_id="unsloth/Nanonets-OCR-s-GGUF",    filename=MMPROJ_NAME,    local_dir=r"/home/ec2-user/SageMaker/workspace",    local_dir_use_symlinks=False,)print("✅ Model:", MODEL_PATH)print("📦 Size:", round(os.path.getsize(MODEL_PATH)/1024/1024, 2), "MB")
```

**3. Implementing the FastAPI Inference Server**

Our main.py handles the SageMaker lifecycle. We use **Lazy Loading** for the model to ensure the container starts quickly, though the very first request will bear the "Cold Start" load time.

``` python
%%writefile main.pyimport os, base64, io, loggingfrom fastapi import FastAPI, Request, Responsefrom PIL import Image, ImageOps, ImageEnhancefrom llama_cpp import Llamafrom llama_cpp.llama_chat_format import MiniCPMv26ChatHandler# Setup logging to see output in SageMaker/Docker logslogging.basicConfig(level=logging.INFO)logger = logging.getLogger(__name__)app = FastAPI()# --- DIAGNOSTIC LOGGING BLOCK ---MODEL_DIR = "/opt/ml/model"logger.info(f"Checking directory: {MODEL_DIR}")if os.path.exists(MODEL_DIR):    # List all files and subdirectories found in the model path    contents = []    for root, dirs, files in os.walk(MODEL_DIR):        for name in files:            contents.append(os.path.join(root, name))        if contents:        logger.info(f"✅ FOUND {len(contents)} files in {MODEL_DIR}:")        for f in contents:            # Print file size to ensure it's not a 0-byte corrupted download            size_mb = os.path.getsize(f) / (1024 * 1024)            logger.info(f" - {f} ({size_mb:.2f} MB)")    else:        logger.error(f"❌ {MODEL_DIR} exists but is EMPTY.")else:    logger.error(f"❌ {MODEL_DIR} directory DOES NOT EXIST.")# SageMaker unpacks S3 model.tar.gz into /opt/ml/modelMODEL_PATH = "/opt/ml/model/Nanonets-OCR-s-Q4_K_M.gguf"MMPROJ_PATH = "/opt/ml/model/mmproj-F16.gguf"# Global model variable for lazy loadingllm = Nonedef load_model():    global llm    if llm is None:        chat_handler = MiniCPMv26ChatHandler(clip_model_path=MMPROJ_PATH, verbose=False)        llm = Llama(            model_path=MODEL_PATH,            chat_handler=chat_handler,            n_ctx=2096, #max for t4 is 128000 (128k) , we have limited to 2096 (2k context)            n_gpu_layers=-1,             verbose=False        )@app.get("/ping")async def ping():    return Response(status_code=200)@app.post("/invocations")async def invocations(request: Request):    load_model()         payload = await request.json()        # 1. Standardize input to a list of tasks    # This supports {"image": "..."} OR {"images": ["...", "..."]}    raw_images = payload.get("images", [])    if not raw_images and "image" in payload:        raw_images = [payload["image"]]            prompt = payload.get("prompt", "Extract all text from this image")    max_tokens = int(payload.get("max_tokens", 512))    temperature = float(payload.get("temperature", 0.0))        if not raw_images:        return Response(content="Missing 'image' or 'images' in payload", status_code=400)    results = []    # 2. Process the batch sequentially    for i, image_b64 in enumerate(raw_images):        try:            out = llm.create_chat_completion(                messages=[{                    "role": "user",                    "content": [                        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},                        {"type": "text", "text": prompt},                    ],                }],                max_tokens=max_tokens,                temperature=temperature,            )                        results.append({                "index": i,                "text": out["choices"][0]["message"]["content"].strip(),                "finish_reason": out["choices"][0].get("finish_reason"),                "usage": out.get("usage")            })        except Exception as e:            results.append({"index": i, "error": str(e)})        # 3. Return the batch result    return {"results": results}
```

**4. Building the Docker Image (BYOC)**

We use a multi-stage build to keep the runtime image lean. We pull a pre-compiled llama-cpp-python wheel for **CUDA 12.4** to avoid lengthy compilation times during the build.

```
%%writefile DockerfileFROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS builderENV DEBIAN_FRONTEND=noninteractiveRUN apt-get update && apt-get install -y python3-pip python3-devRUN pip install --no-cache-dir --upgrade pipRUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124RUN pip install --no-cache-dir fastapi uvicorn pillowFROM nvidia/cuda:12.4.1-runtime-ubuntu22.04RUN apt-get update && apt-get install -y python3 python3-pip && rm -rf /var/lib/apt/lists/*WORKDIR /appCOPY --from=builder /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packagesCOPY --from=builder /usr/local/bin /usr/local/binCOPY main.py .RUN mkdir -p /opt/ml/modelEXPOSE 8080ENTRYPOINT ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
```

**5. Packaging and Pushing**

**Crucial Step:** When creating the model.tar.gz, ensure the files are at the root of the archive so SageMaker finds them at /opt/ml/model/.

```
# Package artifacts!tar -cvzf model.tar.gz Nanonets-OCR-s-Q4_K_M.gguf mmproj-F16.gguf# Upload to S3!aws s3 cp model.tar.gz s3://{S3_BUCKET_NAME}/nanonets-ocr/model.tar.gz# Build and Push Container!aws ecr get-login-password --region {AWS_REGION} | docker login --username AWS --password-stdin {ECR_URI}!docker build -t "$ECR_REPOSITORY_NAME" .!docker tag "$ECR_REPOSITORY_NAME" "$ECR_REPOSITORY_URI":"$IMAGE_TAG"!docker push "$ECR_REPOSITORY_URI":"$IMAGE_TAG"Upload to S3 bucket
```

**Building and pushing image to ECR**

```
!docker build -t "$ECR_REPOSITORY_URI":"$IMAGE_TAG" .!docker push "$ECR_REPOSITORY_URI":"$IMAGE_TAG"
```

**6. Deployment to SageMaker Endpoint**

We deploy to an ml.g4dn.xlarge instance. Even though GGUF is efficient on CPUs, using the **NVIDIA T4 GPU** ensures low-latency OCR performance.

``` python
import sagemakerfrom sagemaker.model import Modelrole = sagemaker.get_execution_role()container_uri = f"{ECR_REPOSITORY_URI}:{IMAGE_TAG}"model_s3_url = f"s3://{S3_BUCKET_NAME}/nanonets-ocr/model.tar.gz"ocr_model = Model(    image_uri=container_uri,    model_data=model_s3_url,    role=role,    name="nanonets-ocr-gguf-model")predictor = ocr_model.deploy(    initial_instance_count=1,    instance_type="ml.g4dn.xlarge",    endpoint_name="vision-ocr-endpoint",    container_startup_health_check_timeout=600)
```

**7. Conclusion**

By combining **llama.cpp** and **SageMaker BYOC**, you’ve built a production-grade OCR pipeline that is both high-performance and cost-aware. You are no longer restricted by pre-built cloud containers and can leverage the latest quantized multimodal models as soon as they drop on Hugging Face.

Don’t forget to clean up:

```
sagemaker_c = boto3.client('sagemaker')sagemaker_c.delete_endpoint(EndpointName=endpoint_name)sagemaker_c.get_waiter('endpoint_deleted').wait(EndpointName=endpoint_name)print("Endpoint + model deleted!")
```

[Deploying Vision GGUF Models to Amazon SageMaker with llama.cpp](https://pub.towardsai.net/deploying-vision-gguf-models-to-amazon-sagemaker-with-llama-cpp-fb6948e79e10) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.