{"slug": "one-llm-model-multiple-adapters-deploying-on-aws-sagemaker", "title": "One LLM Model, Multiple Adapters: Deploying on AWS SageMaker", "summary": "A walkthrough published on AWS's documentation describes serving multiple fine-tuned tasks from a single vision-language model on Amazon SageMaker AI using vLLM and LoRA adapters, rather than deploying a separate endpoint for each version. The pattern uses one SageMaker endpoint with a base inference component holding the shared VLM and separate adapter components that point to adapter archives in Amazon S3 via the BaseInferenceComponentName setting, with each request selecting Adapter 1 or Adapter 2 through InferenceComponentName. The article cautions that shared adapters share serving capacity — a traffic burst on Adapter 1 can affect Adapter 2 — and that vLLM's support for multimodal tower and connector LoRA is experimental, so compatibility with the exact model and pinned runtime must be checked before deployment.", "body_md": "*How to serve different fine-tuned tasks from one shared model using vLLM and LoRA.*\n\nImagine you have a model that can look at an image and answer questions about it.\n\nYou fine-tune it for one task: reading a small table from an image. Later, you fine-tune it for another task: describing an image in a particular format.\n\nNow you have two useful versions of the same model. How should you deploy them?\n\nOne approach is to create a separate endpoint for each version. That is easy to understand, but each deployment may load another copy of the same large base model.\n\nAnother approach is to keep the base model loaded and attach small sets of fine-tuned weights called **LoRA adapters**. Each request chooses the adapter it needs.\n\nIn this article, we will walk through that approach using **Amazon SageMaker AI**, **vLLM**, and a **vision-language model**, or VLM. We will call our two adapters **Adapter 1** and **Adapter 2**.\n\nA vision-language model accepts images and text together. For example:\n\n*Look at this image and return the values in the table as JSON.*\n\nA typical image request follows this path:\n\n```\nImage + question       ↓Image processor       ↓Vision encoder       ↓Multimodal connector       ↓Language model       ↓Answer\n```\n\nThe image processor prepares the image. The vision encoder turns it into features. The connector makes those features usable by the language model, which generates the answer.\n\nA LoRA adapter contains a relatively small set of learned changes to selected layers. It works together with the base model it was trained for.\n\nThink of the base model as a shared engine and each adapter as a different setting for that engine. Adapter 1 might learn to return table values in a fixed JSON format. Adapter 2 might learn to produce short descriptions using a particular structure.\n\nThese are illustrative tasks. In your deployment, use adapters trained for your own requirements.\n\nThe important requirement is compatibility: both adapters must match the base model and the serving runtime.\n\nSageMaker uses **inference components** to represent the base and its adapters:\n\n```\nOne SageMaker endpoint          |          +-- Base inference component                  |                  +-- Shared VLM running in vLLM                  |                  +-- Adapter 1                  |                  +-- Adapter 2\n```\n\nThe base component has the compute allocation. Each adapter component points to that base and to an adapter archive in Amazon S3. This relationship is defined by SageMaker’s BaseInferenceComponentName setting. [AWS inference component reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InferenceComponentSpecification.html).\n\nA request selects Adapter 1 or Adapter 2 through InferenceComponentName.\n\nThis does not combine both adapters into one answer. Each request in our example selects one specialization.\n\nSharing a base can reduce duplicated model weights. It also means the adapters share serving capacity. If Adapter 1 receives a burst of traffic, it can affect the resources available to Adapter 2. Measure the combined workload before choosing instance size and scaling settings.\n\nFor a VLM, checking that the container supports “LoRA” is not enough.\n\nYou need support for the particular combination of:\n\nAn adapter can modify only the language layers while still answering image questions through the base model’s vision encoder. Alternatively, training may modify vision or connector layers too. Those cases have different serving requirements.\n\nvLLM documents experimental support for multimodal tower and connector LoRA. Check support for your exact model and pinned runtime before deployment. [vLLM LoRA documentation](https://docs.vllm.ai/en/stable/features/lora/#lora-support-for-tower-and-connector-of-multi-modal-model).\n\nThe following walkthrough assumes that you have already checked this compatibility. It provides a deployment pattern, rather than claiming that any VLM and adapter pair will work together. The snippets have not been run against an AWS account for this article.\n\nWe need three archives in S3:\n\n```\ns3://YOUR_BUCKET/models/base-model.tar.gzs3://YOUR_BUCKET/adapters/adapter-1.tar.gzs3://YOUR_BUCKET/adapters/adapter-2.tar.gz\n```\n\nThe base archive must contain the complete VLM checkpoint. Include its model configuration, weight files, tokenizer, image processor configuration, and chat template where supplied by the model repository.\n\nA common mistake is to package only the language-model weights. A VLM also needs the files that explain how to process images.\n\nFor a conventional PEFT LoRA adapter, the archive commonly contains these files at its root:\n\n```\nadapter-1.tar.gz├── adapter_config.json└── adapter_model.safetensors\n```\n\nAvoid an extra folder level inside the archive. The container must find the adapter files in the directory SageMaker provides. Some adapters require additional files; check your model and runtime before using this minimal layout. [Hugging Face SageMaker multi-adapter guide](https://huggingface.co/docs/sagemaker/examples/sagemaker-sdk-multi-lora-adapters-vllm).\n\nHere is a simple packaging example for an adapter you have already downloaded:\n\n``` python\nimport tarfilefrom pathlib import Path\nadapter_folder = Path(\"adapter-1\")\nwith tarfile.open(\"adapter-1.tar.gz\", \"w:gz\") as archive:    for filename in [\"adapter_config.json\", \"adapter_model.safetensors\"]:        path = adapter_folder / filename        if not path.is_file():            raise FileNotFoundError(path)        archive.add(path, arcname=filename, recursive=False)\n```\n\nKeep a record of the exact base and adapter revisions. When you release a new adapter, upload it under a new S3 key so that you can identify and restore the previous version.\n\nUse your AWS environment’s authentication and a SageMaker execution role. Keep credentials out of notebook cells and published examples.\n\nThe role needs access to the model artifacts, container image, logs, and any encryption keys used by the deployment. Your deployment identity also needs the necessary SageMaker permissions and permission to pass the role.\n\n``` python\nimport jsonimport osimport timeimport uuid\npython\nimport boto3from botocore.config import Config\nREGION = os.environ.get(\"AWS_REGION\", \"us-east-1\")ROLE_ARN = os.environ[\"SAGEMAKER_EXECUTION_ROLE_ARN\"]IMAGE_URI = os.environ[\"VALIDATED_VLLM_IMAGE_URI\"]BASE_MODEL_S3 = os.environ[\"BASE_MODEL_S3_URI\"]ADAPTER_1_S3 = os.environ[\"ADAPTER_1_S3_URI\"]ADAPTER_2_S3 = os.environ[\"ADAPTER_2_S3_URI\"]INSTANCE_TYPE = os.environ[\"SAGEMAKER_INSTANCE_TYPE\"]\nsession = boto3.Session(region_name=REGION)sm = session.client(\"sagemaker\")runtime = session.client(    \"sagemaker-runtime\",    config=Config(read_timeout=70, connect_timeout=10),)\nsuffix = uuid.uuid4().hex[:8]MODEL_NAME = f\"vlm-model-{suffix}\"ENDPOINT_NAME = f\"vlm-endpoint-{suffix}\"ENDPOINT_CONFIG_NAME = f\"vlm-config-{suffix}\"BASE_COMPONENT = f\"vlm-base-{suffix}\"ADAPTER_1_COMPONENT = f\"adapter-1-{suffix}\"ADAPTER_2_COMPONENT = f\"adapter-2-{suffix}\"\n```\n\nUse S3 artifacts in the deployment region. Select the instance based on measured memory needs, and check its regional availability and your SageMaker quota.\n\nVALIDATED_VLLM_IMAGE_URI should identify a container you have tested with your VLM. Pin its image digest so a later image change does not silently change your deployment.\n\nThe following environment variables use the Hugging Face vLLM container’s SM_VLLM_ interface. Confirm that your selected image supports that interface.\n\n```\nenvironment = {    \"SM_VLLM_MODEL\": \"/opt/ml/model\",    \"SM_VLLM_SERVED_MODEL_NAME\": \"shared-vlm\",    \"SM_VLLM_HOST\": \"0.0.0.0\",    \"SM_VLLM_MAX_MODEL_LEN\": \"4096\",    \"SM_VLLM_GPU_MEMORY_UTILIZATION\": \"0.80\",    \"SM_VLLM_ENABLE_LORA\": \"true\",    \"SM_VLLM_MAX_LORA_RANK\": \"32\",    \"SM_VLLM_MAX_LORAS\": \"2\",    \"SM_VLLM_MAX_CPU_LORAS\": \"2\",    \"SM_VLLM_LIMIT_MM_PER_PROMPT\": json.dumps({        \"image\": 1,        \"audio\": 0,        \"video\": 0,    }),    \"VLLM_ALLOW_RUNTIME_LORA_UPDATING\": \"true\",}\n```\n\nHere is what the main settings mean:\n\nThe runtime registration setting supports the adapter lifecycle used by SageMaker’s integration. [Hugging Face container configuration example](https://huggingface.co/docs/sagemaker/examples/sagemaker-sdk-multi-lora-adapters-vllm).\n\nThe rank limit must cover both adapters. Check r and any per-layer rank overrides in their configurations.\n\nAlso, MAX_LORAS=2 does not mean “only two requests at a time.” It controls the number of different adapters that can participate in a batch.\n\nFor a compatible prequantized BitsAndBytes checkpoint, the serving configuration may also need:\n\n```\n# Add only for a validated BitsAndBytes deployment.environment.update({    \"SM_VLLM_QUANTIZATION\": \"bitsandbytes\",    \"SM_VLLM_LOAD_FORMAT\": \"bitsandbytes\",    \"SM_VLLM_DTYPE\": \"bfloat16\",})\n```\n\nThese settings tell the runtime how to load and compute with the checkpoint. They do not convert an arbitrary checkpoint into a compatible 4-bit model.\n\nThe weights can be stored in 4-bit form while computation uses BF16. This reduces weight memory, but images, activations, adapters, and the KV cache still need memory.\n\nFirst, register the model and create the endpoint:\n\n```\nsm.create_model(    ModelName=MODEL_NAME,    ExecutionRoleArn=ROLE_ARN,    PrimaryContainer={        \"Image\": IMAGE_URI,        \"ModelDataUrl\": BASE_MODEL_S3,        \"Environment\": environment,    },)\nsm.create_endpoint_config(    EndpointConfigName=ENDPOINT_CONFIG_NAME,    ExecutionRoleArn=ROLE_ARN,    ProductionVariants=[{        \"VariantName\": \"AllTraffic\",        \"InstanceType\": INSTANCE_TYPE,        \"InitialInstanceCount\": 1,        \"RoutingConfig\": {            \"RoutingStrategy\": \"LEAST_OUTSTANDING_REQUESTS\",        },    }],)\nsm.create_endpoint(    EndpointName=ENDPOINT_NAME,    EndpointConfigName=ENDPOINT_CONFIG_NAME,)\nsm.get_waiter(\"endpoint_in_service\").wait(    EndpointName=ENDPOINT_NAME,    WaiterConfig={\"Delay\": 30, \"MaxAttempts\": 120},)\n```\n\nThis step creates a billable endpoint. Some containers also require a particular inference AMI or driver combination; set a compatible InferenceAmiVersion in the production variant when your selected image requires it.\n\nNext, add the base inference component:\n\n``` python\ndef wait_for_component(name, timeout=3600):    deadline = time.monotonic() + timeout    while time.monotonic() < deadline:        info = sm.describe_inference_component(            InferenceComponentName=name,        )        status = info[\"InferenceComponentStatus\"]        if status == \"InService\":            return        if status in {\"Failed\", \"UpdateFailed\", \"Deleting\"}:            raise RuntimeError(                f\"{name}: {status}: {info.get('FailureReason', '')}\"            )        time.sleep(15)    raise TimeoutError(f\"Timed out waiting for {name}\")\nsm.create_inference_component(    InferenceComponentName=BASE_COMPONENT,    EndpointName=ENDPOINT_NAME,    VariantName=\"AllTraffic\",    Specification={        \"ModelName\": MODEL_NAME,        \"ComputeResourceRequirements\": {            \"NumberOfAcceleratorDevicesRequired\": 1,            \"MinMemoryRequiredInMb\": 8192,        },        \"StartupParameters\": {            \"ModelDataDownloadTimeoutInSeconds\": 1800,            \"ContainerStartupHealthCheckTimeoutInSeconds\": 1800,        },    },    RuntimeConfig={\"CopyCount\": 1},)wait_for_component(BASE_COMPONENT)\n```\n\nThe 8,192 MB value is an example host-memory reservation, not GPU memory. Adjust it to your container and model-loading requirements.\n\nThis example creates one serving copy of the base. If you scale to multiple copies, each copy needs its own model weights.\n\nOnce the base is ready, attach both adapters:\n\n```\nadapters = {    ADAPTER_1_COMPONENT: ADAPTER_1_S3,    ADAPTER_2_COMPONENT: ADAPTER_2_S3,}\nfor component_name, artifact_uri in adapters.items():    sm.create_inference_component(        InferenceComponentName=component_name,        EndpointName=ENDPOINT_NAME,        Specification={            \"BaseInferenceComponentName\": BASE_COMPONENT,            \"Container\": {\"ArtifactUrl\": artifact_uri},        },    )    wait_for_component(component_name)\n```\n\nNotice that we do not request another GPU for each adapter. We link each child component to the existing base. [AWS adapter deployment example](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-adapt.html).\n\nSave these resource names and artifact versions in a deployment manifest. You will need them for updates, troubleshooting, and cleanup.\n\nNow for the part that makes this a VLM example: sending an actual image.\n\nThe message contains a text instruction and an image encoded as a data URL. This follows vLLM’s OpenAI-compatible multimodal request format. [vLLM image input examples](https://docs.vllm.ai/en/stable/features/multimodal_inputs/).\n\n``` python\nimport base64from pathlib import Pathfrom PIL import Image\npython\ndef image_message(image_path, instruction):    path = Path(image_path)    with Image.open(path) as image:        if image.format not in {\"JPEG\", \"PNG\", \"WEBP\"}:            raise ValueError(\"Use a JPEG, PNG, or WEBP image\")        mime_type = Image.MIME[image.format]\nencoded = base64.b64encode(path.read_bytes()).decode(\"ascii\")    return [{        \"role\": \"user\",        \"content\": [            {\"type\": \"text\", \"text\": instruction},            {\"type\": \"image_url\", \"image_url\": {                \"url\": f\"data:{mime_type};base64,{encoded}\",            }},        ],    }]\npython\ndef invoke(component_name, messages):    body = json.dumps({        \"model\": \"shared-vlm\",        \"messages\": messages,        \"max_tokens\": 256,        \"temperature\": 0,    }).encode(\"utf-8\")\n# A conservative application budget; check the API limits you use.    if len(body) > 5_500_000:        raise ValueError(\"Resize or compress the image before sending\")\nresponse = runtime.invoke_endpoint(        EndpointName=ENDPOINT_NAME,        InferenceComponentName=component_name,        ContentType=\"application/json\",        CustomAttributes=\"route=/v1/chat/completions\",        Body=body,    )    return json.loads(response[\"Body\"].read())\nmessages = image_message(    \"sample-table.png\",    \"Read this table and return the values as JSON.\",)result = invoke(ADAPTER_1_COMPONENT, messages)print(result[\"choices\"][0][\"message\"][\"content\"])\n```\n\nThe container must support the chat route and the model’s chat template. Use the same prompt style and output format that the adapter learned during training.\n\nFor example, imagine a synthetic image containing:\n\n```\nItem       QuantityApples     3Oranges    5\n```\n\nAn illustrative response might be:\n\n```\n{\"items\": [{\"name\": \"Apples\", \"quantity\": 3}, {\"name\": \"Oranges\", \"quantity\": 5}]}\n```\n\nThat is an example of the intended behavior, not a measured result. Asking for JSON does not guarantee correct or valid JSON. Validate the response and compare its values with the image.\n\nTo use Adapter 2, change the component name and give it an instruction appropriate to its training:\n\n```\nmessages = image_message(    \"sample-image.png\",    \"Describe this image using the format you were trained to follow.\",)result = invoke(ADAPTER_2_COMPONENT, messages)\n```\n\nThe endpoint stays the same. The selected adapter changes.\n\nYou can also invoke BASE_COMPONENT to compare the original model with either adapter. This is a useful way to check whether fine-tuning actually improved the task.\n\nA small JPEG is not necessarily a cheap model input. Image resolution and the model’s preprocessing strategy affect visual tokens, processing time, and memory.\n\nFor a table-reading task, crop away irrelevant background while keeping the text readable. Reducing resolution too aggressively can turn a clear number into an ambiguous one.\n\nBase64 also increases the transmitted image size by roughly one third. Check both the serialized request size and the model’s context budget.\n\nTest in stages:\n\nIf you receive ModelNotReadyException while an adapter loads, use bounded retries with backoff. Other errors need diagnosis rather than unlimited retries.\n\nSuppose an adapter contains weights for the vision encoder, but the serving runtime cannot apply them.\n\nDeleting those weights might allow the remaining adapter to load. It can also remove part of what the model learned.\n\nCheck the training configuration and the runtime’s supported targets. If trained modules are unsupported, use a compatible runtime or retrain with supported targets. Any conversion needs quality evaluation.\n\nSome model and container combinations need workarounds involving quantization metadata, eager execution, KV-cache handling, or cache-connector startup settings.\n\nThese fixes solve different problems. For example, disabling a KV-transfer connector does not disable the vision encoder, and disabling a multimodal processor cache does not remove the language model’s KV cache.\n\nRecord each fix with the exact image version and the error it addresses. Retest when upgrading. Copying a collection of old environment flags into a new image can make troubleshooting harder.\n\nTreat adapters as independently versioned artifacts.\n\nFor example, keep Adapter 1’s current component while creating a new component for its next version. Test the new one, then update the application’s component-name mapping.\n\nThis allows you to return to the previous version if evaluation finds a problem. During the overlap, allow for the additional adapter cache and memory demand.\n\nDeleting the existing component and recreating it under the same name is another approach, but requests to that adapter can fail during the gap.\n\nDelete the adapter components first, wait for their deletion, and then delete the base component. After that, delete the endpoint and wait for it to disappear before removing its endpoint configuration and model object.\n\nS3 artifacts, custom container images, and retained logs need their own cleanup or retention policy.\n\nFor a first experiment, start with one compatible base, one adapter, and one small image. Once that works, attach Adapter 2 and test both together. This makes it much easier to tell whether a failure comes from image processing, adapter compatibility, or shared serving capacity.\n\n[One LLM Model, Multiple Adapters: Deploying on AWS SageMaker](https://pub.towardsai.net/one-vision-language-model-multiple-adapters-deploying-on-amazon-sagemaker-26b7439d5d45) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/one-llm-model-multiple-adapters-deploying-on-aws-sagemaker", "canonical_source": "https://pub.towardsai.net/one-vision-language-model-multiple-adapters-deploying-on-amazon-sagemaker-26b7439d5d45?source=rss----98111c9905da---4", "published_at": "2026-10-24 06:00:47+00:00", "updated_at": "2026-10-24 06:12:24.473609+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "computer-vision", "mlops", "ai-tools"], "entities": ["Amazon SageMaker AI", "vLLM", "LoRA", "Amazon Web Services", "Amazon S3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/one-llm-model-multiple-adapters-deploying-on-aws-sagemaker", "markdown": "https://wpnews.pro/news/one-llm-model-multiple-adapters-deploying-on-aws-sagemaker.md", "text": "https://wpnews.pro/news/one-llm-model-multiple-adapters-deploying-on-aws-sagemaker.txt", "jsonld": "https://wpnews.pro/news/one-llm-model-multiple-adapters-deploying-on-aws-sagemaker.jsonld"}}