Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive

Ok, so, maybe you're a software developer or data scientist who just heard about the new, massive 600GB Inkling-NVFP4 AI model, and you want to try running it yourself without breaking the bank. You also might have learned through some research that renting a "Spot A3" instance on Google Kubernetes Engine (GKE) is a brilliant way to do it. An A3 instance is essentially a massive supercomputer packed with 8 top-tier NVIDIA H100 GPUs, and "Spot" means we are renting it at a huge discount.

You set up your cluster, point it to your AI engine, hit deploy, and... it crashes..

But what exactly is causing that? Why does our shiny new AI supercomputer crash before we even send it a single prompt?

Deploying cutting-edge, massive models introduces a perfect storm of software conflicts and physical memory limits. At a high level, this is how you navigate these hurdles to get your AI engine running smoothly without writing hacky installation scripts or building your own custom Docker images.

The first major hurdle most people hit is a software conflict (specifically, an ABI mismatch). Deploying a model like Inkling requires gigabytes of bleeding-edge AI software (like vllm 0.25+ and transformers).

If you try to install these new AI packages on top of the default Ray cluster image (rayproject/ray), they will automatically upgrade underlying math libraries like numpy to version 2.0. Because the cluster's base system was compiled against numpy version 1.0, this causes a severe communication breakdown.

Imagine two high-speed construction foremen coordinating across a busy site. One foreman is speaking a legacy 1.0 dialect, while the other suddenly switches mid-shift to a brand-new 2.0 slang. When they try to send messages back and forth across the cluster nodes, they literally lose the ability to speak the same language and crash instantly.

Furthermore, if you try to install these heavy packages on the fly when the server starts up, you will almost certainly hit Ray's built-in 10-minute download timeout limit. It is a tricky one to debug when you are first starting out, and a quite noticeable performance hit to developer velocity.

To cleanly solve this without building a custom Docker container, you should simply use the official vLLM image as your cluster's base image.

The official vllm/vllm-openai:v0.26.0 image already has ray, vllm, transformers, and the correct version of numpy beautifully packaged together. By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts.

Once the software environment is running smoothly, we hit the hard physical limits of the hardware itself. An A3 instance gives us 8 NVIDIA H100 GPUs, providing a massive 640GB of total High Bandwidth Memory (VRAM). However, the Inkling-NVFP4 model we want to run is roughly 600GB. This means the model weights alone consume almost all the available space on the cards.

When the AI engine boots up, it tries to pre-allocate space for something called the KV Cache. The KV Cache is basically the AI's short-term memory that it uses while generating a sentence. By default, Inkling wants to remember up to 1 million words at a time. Trying to reserve enough short-term memory for 1 million words requires about 7GB of space per GPU, which immediately causes an Out-Of-Memory (OOM) crash because our graphics cards are already 95% full!

To visualize this memory squeeze, imagine trying to pack a massive grand piano into a delivery van. The piano takes up 95% of the cargo space. If you then try to cram ten giant moving trunks (a 1-million-token KV cache) into the tiny remaining sliver of space, the doors won't shut and the axle snaps.

To squeeze into the remaining memory safely, we tweak two critical engine arguments to swap those giant moving trunks for a compact briefcase:

max_model_len=4096`` gpu_memory_utilization=0.96 Because we used the official vLLM image for our cluster, our deployment code doesn't need any messy installation hacks or shell scripts. Here is the beautifully clean Python script using Ray Serve to configure the memory limits and serve the model across all 8 GPUs:

import ray
from ray import serve
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

ray.init(address="auto")
app = FastAPI()

@serve.deployment(
    num_replicas=1, 
    ray_actor_options={"num_gpus": 8}
)
@serve.ingress(app)
class InklingDeployment:
    def __init__(self):
        from vllm.engine.arg_utils import AsyncEngineArgs
        from vllm.engine.async_llm_engine import AsyncLLMEngine

        engine_args = AsyncEngineArgs(
            model="thinkingmachines/Inkling-NVFP4",
            tensor_parallel_size=8, # Split the model weights across all 8 GPUs
            gpu_memory_utilization=0.96, # Leave 4% VRAM headroom for the OS and CUDA
            max_model_len=4096, # Cap short-term memory to 4k tokens to fit VRAM
            trust_remote_code=True,
            enforce_eager=True 
        )
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    @app.get("/health")
    async def health(self):
        return {"status": "Inkling-NVFP4 is ready!"}

    @app.post("/completions")
    async def generate(self, request: Request):
        from vllm import SamplingParams
        from vllm.utils import random_uuid

        request_dict = await request.json()
        prompt = request_dict.pop("prompt")
        sampling_params = SamplingParams(**request_dict)

        results_generator = self.engine.generate(prompt, sampling_params, random_uuid())
        final_output = None
        async for request_output in results_generator:
            final_output = request_output

        return JSONResponse({"text": final_output.outputs[0].text})

if __name__ == "__main__":
    serve.start(detached=True)
    serve.run(InklingDeployment.bind(), route_prefix="/inkling")

Note here that we specify @serve.ingress(app) right at the top of the class definition. It is a good habit to be getting into to ensure the Ray cluster properly routes your HTTP endpoints (like /health and /generate) to external traffic.

So, now you've got your serving infrastructure configured, your 600GB Inkling-NVFP4 model slotting into VRAM alongside the KV cache on your GKE Spot A3 node pool, and you can successfully chat with your massive AI model without breaking the bank.

If you want to dive deeper into the model weights, engine options, or cluster orchestration used in this deep dive, check out these official documentation pages and resources:

Let us know how your GKE Spot A3 deployment went, and happy serving!