{"slug": "deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive", "title": "Deploying the 600GB Inkling-NVFP4 Model on Spot A3: A GKE and vLLM Deep Dive", "summary": "A developer documented how to run the 600GB Inkling-NVFP4 model on a Google Kubernetes Engine Spot A3 instance with eight NVIDIA H100 GPUs, tracing the initial crash to an ABI mismatch between Ray's default image and newer vLLM/transformers dependencies. The fix was to use the official vllm/vllm-openai:v0.26.0 image as the cluster base, and to avoid out-of-memory crashes by setting max_model_len=4096 and gpu_memory_utilization=0.96 so the KV cache fits alongside the model weights.", "body_md": "Ok, so, maybe you're a software developer or data scientist who just heard about the new, massive 600GB Inkling-NVFP4 AI model, and you want to try running it yourself without breaking the bank. You also might have learned through some research that renting a \"Spot A3\" instance on Google Kubernetes Engine (GKE) is a brilliant way to do it. An A3 instance is essentially a massive supercomputer packed with 8 top-tier NVIDIA H100 GPUs, and \"Spot\" means we are renting it at a huge discount.\n\nYou set up your cluster, point it to your AI engine, hit deploy, and... it crashes..\n\nBut what exactly is causing that? Why does our shiny new AI supercomputer crash before we even send it a single prompt?\n\nDeploying cutting-edge, massive models introduces a perfect storm of software conflicts and physical memory limits. At a high level, this is how you navigate these hurdles to get your AI engine running smoothly without writing hacky installation scripts or building your own custom Docker images.\n\nThe first major hurdle most people hit is a software conflict (specifically, an ABI mismatch). Deploying a model like Inkling requires gigabytes of bleeding-edge AI software (like `vllm` 0.25+ and `transformers`).\n\nIf you try to install these new AI packages on top of the default Ray cluster image (rayproject/ray), they will automatically upgrade underlying math libraries like numpy to version 2.0. Because the cluster's base system was compiled against `numpy` version 1.0, this causes a severe communication breakdown.\n\nImagine two high-speed construction foremen coordinating across a busy site. One foreman is speaking a legacy 1.0 dialect, while the other suddenly switches mid-shift to a brand-new 2.0 slang. When they try to send messages back and forth across the cluster nodes, they literally lose the ability to speak the same language and crash instantly.\n\nFurthermore, if you try to install these heavy packages on the fly when the server starts up, you will almost certainly hit Ray's built-in 10-minute download timeout limit. It is a tricky one to debug when you are first starting out, and a quite noticeable performance hit to developer velocity.\n\nTo cleanly solve this without building a custom Docker container, **you should simply use the official vLLM image as your cluster's base image.**\n\nThe official `vllm/vllm-openai:v0.26.0` image already has ray, vllm, transformers, and the correct version of numpy beautifully packaged together. By swapping out your Kubernetes YAML manifest to use this image for both the Head and Worker nodes, the servers boot up instantly with no API conflicts, no ABI mismatches, and zero download timeouts.\n\nOnce the software environment is running smoothly, we hit the hard physical limits of the hardware itself. An A3 instance gives us 8 NVIDIA H100 GPUs, providing a massive 640GB of total High Bandwidth Memory (VRAM). However, the Inkling-NVFP4 model we want to run is roughly 600GB. This means the model weights alone consume almost all the available space on the cards.\n\nWhen the AI engine boots up, it tries to pre-allocate space for something called the KV Cache. The KV Cache is basically the AI's short-term memory that it uses while generating a sentence. By default, Inkling wants to remember up to 1 million words at a time. Trying to reserve enough short-term memory for 1 million words requires about 7GB of space per GPU, which immediately causes an Out-Of-Memory (OOM) crash because our graphics cards are already 95% full!\n\nTo visualize this memory squeeze, imagine trying to pack a massive grand piano into a delivery van. The piano takes up 95% of the cargo space. If you then try to cram ten giant moving trunks (a 1-million-token KV cache) into the tiny remaining sliver of space, the doors won't shut and the axle snaps.\n\nTo squeeze into the remaining memory safely, we tweak two critical engine arguments to swap those giant moving trunks for a compact briefcase:\n\n`max_model_len=4096`` gpu_memory_utilization=0.96`\nBecause we used the official vLLM image for our cluster, our deployment code doesn't need any messy installation hacks or shell scripts. Here is the beautifully clean Python script using Ray Serve to configure the memory limits and serve the model across all 8 GPUs:\n\n``` python\nimport ray\nfrom ray import serve\nfrom fastapi import FastAPI, Request\nfrom fastapi.responses import JSONResponse\n\n# Connect to the Ray cluster\nray.init(address=\"auto\")\napp = FastAPI()\n\n# Tell Ray Serve we need 8 GPUs to run this deployment actor\n@serve.deployment(\n    num_replicas=1, \n    ray_actor_options={\"num_gpus\": 8}\n)\n@serve.ingress(app)\nclass InklingDeployment:\n    def __init__(self):\n        from vllm.engine.arg_utils import AsyncEngineArgs\n        from vllm.engine.async_llm_engine import AsyncLLMEngine\n\n        # Configure the vLLM inference engine\n        engine_args = AsyncEngineArgs(\n            model=\"thinkingmachines/Inkling-NVFP4\",\n            tensor_parallel_size=8, # Split the model weights across all 8 GPUs\n            gpu_memory_utilization=0.96, # Leave 4% VRAM headroom for the OS and CUDA\n            max_model_len=4096, # Cap short-term memory to 4k tokens to fit VRAM\n            trust_remote_code=True,\n            enforce_eager=True \n        )\n        self.engine = AsyncLLMEngine.from_engine_args(engine_args)\n\n    # Health check endpoint to confirm the model is online\n    @app.get(\"/health\")\n    async def health(self):\n        return {\"status\": \"Inkling-NVFP4 is ready!\"}\n\n    # Generation endpoint to handle incoming prompts\n    @app.post(\"/completions\")\n    async def generate(self, request: Request):\n        from vllm import SamplingParams\n        from vllm.utils import random_uuid\n\n        request_dict = await request.json()\n        prompt = request_dict.pop(\"prompt\")\n        sampling_params = SamplingParams(**request_dict)\n\n        results_generator = self.engine.generate(prompt, sampling_params, random_uuid())\n        final_output = None\n        async for request_output in results_generator:\n            final_output = request_output\n\n        return JSONResponse({\"text\": final_output.outputs[0].text})\n\nif __name__ == \"__main__\":\n    serve.start(detached=True)\n    serve.run(InklingDeployment.bind(), route_prefix=\"/inkling\")\n```\n\nNote here that we specify `@serve.ingress(app)` right at the top of the class definition. It is a good habit to be getting into to ensure the Ray cluster properly routes your HTTP endpoints (like /health and /generate) to external traffic.\n\nSo, now you've got your serving infrastructure configured, your 600GB Inkling-NVFP4 model slotting into VRAM alongside the KV cache on your GKE Spot A3 node pool, and you can successfully chat with your massive AI model without breaking the bank.\n\nIf you want to dive deeper into the model weights, engine options, or cluster orchestration used in this deep dive, check out these official documentation pages and resources:\n\nLet us know how your GKE Spot A3 deployment went, and happy serving!", "url": "https://wpnews.pro/news/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive", "canonical_source": "https://dev.to/alchemicduncan/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive-34hc", "published_at": "2026-09-18 00:35:50+00:00", "updated_at": "2026-09-18 00:52:47.000185+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools", "ai-chips"], "entities": ["Inkling-NVFP4", "Google Kubernetes Engine", "NVIDIA H100", "vLLM", "Ray", "numpy", "A3"], "alternates": {"html": "https://wpnews.pro/news/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive", "markdown": "https://wpnews.pro/news/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive.md", "text": "https://wpnews.pro/news/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive.txt", "jsonld": "https://wpnews.pro/news/deploying-the-600gb-inkling-nvfp4-model-on-spot-a3-a-gke-and-vllm-deep-dive.jsonld"}}