# One LLM Model, Multiple Adapters: Deploying on AWS SageMaker

> Source: <https://pub.towardsai.net/one-vision-language-model-multiple-adapters-deploying-on-amazon-sagemaker-26b7439d5d45?source=rss----98111c9905da---4>
> Published: 2026-10-24 06:00:47+00:00

*How to serve different fine-tuned tasks from one shared model using vLLM and LoRA.*

Imagine you have a model that can look at an image and answer questions about it.

You fine-tune it for one task: reading a small table from an image. Later, you fine-tune it for another task: describing an image in a particular format.

Now you have two useful versions of the same model. How should you deploy them?

One approach is to create a separate endpoint for each version. That is easy to understand, but each deployment may load another copy of the same large base model.

Another approach is to keep the base model loaded and attach small sets of fine-tuned weights called **LoRA adapters**. Each request chooses the adapter it needs.

In this article, we will walk through that approach using **Amazon SageMaker AI**, **vLLM**, and a **vision-language model**, or VLM. We will call our two adapters **Adapter 1** and **Adapter 2**.

A vision-language model accepts images and text together. For example:

*Look at this image and return the values in the table as JSON.*

A typical image request follows this path:

```
Image + question       ↓Image processor       ↓Vision encoder       ↓Multimodal connector       ↓Language model       ↓Answer
```

The image processor prepares the image. The vision encoder turns it into features. The connector makes those features usable by the language model, which generates the answer.

A LoRA adapter contains a relatively small set of learned changes to selected layers. It works together with the base model it was trained for.

Think of the base model as a shared engine and each adapter as a different setting for that engine. Adapter 1 might learn to return table values in a fixed JSON format. Adapter 2 might learn to produce short descriptions using a particular structure.

These are illustrative tasks. In your deployment, use adapters trained for your own requirements.

The important requirement is compatibility: both adapters must match the base model and the serving runtime.

SageMaker uses **inference components** to represent the base and its adapters:

```
One SageMaker endpoint          |          +-- Base inference component                  |                  +-- Shared VLM running in vLLM                  |                  +-- Adapter 1                  |                  +-- Adapter 2
```

The base component has the compute allocation. Each adapter component points to that base and to an adapter archive in Amazon S3. This relationship is defined by SageMaker’s BaseInferenceComponentName setting. [AWS inference component reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_InferenceComponentSpecification.html).

A request selects Adapter 1 or Adapter 2 through InferenceComponentName.

This does not combine both adapters into one answer. Each request in our example selects one specialization.

Sharing a base can reduce duplicated model weights. It also means the adapters share serving capacity. If Adapter 1 receives a burst of traffic, it can affect the resources available to Adapter 2. Measure the combined workload before choosing instance size and scaling settings.

For a VLM, checking that the container supports “LoRA” is not enough.

You need support for the particular combination of:

An adapter can modify only the language layers while still answering image questions through the base model’s vision encoder. Alternatively, training may modify vision or connector layers too. Those cases have different serving requirements.

vLLM documents experimental support for multimodal tower and connector LoRA. Check support for your exact model and pinned runtime before deployment. [vLLM LoRA documentation](https://docs.vllm.ai/en/stable/features/lora/#lora-support-for-tower-and-connector-of-multi-modal-model).

The following walkthrough assumes that you have already checked this compatibility. It provides a deployment pattern, rather than claiming that any VLM and adapter pair will work together. The snippets have not been run against an AWS account for this article.

We need three archives in S3:

```
s3://YOUR_BUCKET/models/base-model.tar.gzs3://YOUR_BUCKET/adapters/adapter-1.tar.gzs3://YOUR_BUCKET/adapters/adapter-2.tar.gz
```

The base archive must contain the complete VLM checkpoint. Include its model configuration, weight files, tokenizer, image processor configuration, and chat template where supplied by the model repository.

A common mistake is to package only the language-model weights. A VLM also needs the files that explain how to process images.

For a conventional PEFT LoRA adapter, the archive commonly contains these files at its root:

```
adapter-1.tar.gz├── adapter_config.json└── adapter_model.safetensors
```

Avoid an extra folder level inside the archive. The container must find the adapter files in the directory SageMaker provides. Some adapters require additional files; check your model and runtime before using this minimal layout. [Hugging Face SageMaker multi-adapter guide](https://huggingface.co/docs/sagemaker/examples/sagemaker-sdk-multi-lora-adapters-vllm).

Here is a simple packaging example for an adapter you have already downloaded:

``` python
import tarfilefrom pathlib import Path
adapter_folder = Path("adapter-1")
with tarfile.open("adapter-1.tar.gz", "w:gz") as archive:    for filename in ["adapter_config.json", "adapter_model.safetensors"]:        path = adapter_folder / filename        if not path.is_file():            raise FileNotFoundError(path)        archive.add(path, arcname=filename, recursive=False)
```

Keep a record of the exact base and adapter revisions. When you release a new adapter, upload it under a new S3 key so that you can identify and restore the previous version.

Use your AWS environment’s authentication and a SageMaker execution role. Keep credentials out of notebook cells and published examples.

The role needs access to the model artifacts, container image, logs, and any encryption keys used by the deployment. Your deployment identity also needs the necessary SageMaker permissions and permission to pass the role.

``` python
import jsonimport osimport timeimport uuid
python
import boto3from botocore.config import Config
REGION = os.environ.get("AWS_REGION", "us-east-1")ROLE_ARN = os.environ["SAGEMAKER_EXECUTION_ROLE_ARN"]IMAGE_URI = os.environ["VALIDATED_VLLM_IMAGE_URI"]BASE_MODEL_S3 = os.environ["BASE_MODEL_S3_URI"]ADAPTER_1_S3 = os.environ["ADAPTER_1_S3_URI"]ADAPTER_2_S3 = os.environ["ADAPTER_2_S3_URI"]INSTANCE_TYPE = os.environ["SAGEMAKER_INSTANCE_TYPE"]
session = boto3.Session(region_name=REGION)sm = session.client("sagemaker")runtime = session.client(    "sagemaker-runtime",    config=Config(read_timeout=70, connect_timeout=10),)
suffix = uuid.uuid4().hex[:8]MODEL_NAME = f"vlm-model-{suffix}"ENDPOINT_NAME = f"vlm-endpoint-{suffix}"ENDPOINT_CONFIG_NAME = f"vlm-config-{suffix}"BASE_COMPONENT = f"vlm-base-{suffix}"ADAPTER_1_COMPONENT = f"adapter-1-{suffix}"ADAPTER_2_COMPONENT = f"adapter-2-{suffix}"
```

Use S3 artifacts in the deployment region. Select the instance based on measured memory needs, and check its regional availability and your SageMaker quota.

VALIDATED_VLLM_IMAGE_URI should identify a container you have tested with your VLM. Pin its image digest so a later image change does not silently change your deployment.

The following environment variables use the Hugging Face vLLM container’s SM_VLLM_ interface. Confirm that your selected image supports that interface.

```
environment = {    "SM_VLLM_MODEL": "/opt/ml/model",    "SM_VLLM_SERVED_MODEL_NAME": "shared-vlm",    "SM_VLLM_HOST": "0.0.0.0",    "SM_VLLM_MAX_MODEL_LEN": "4096",    "SM_VLLM_GPU_MEMORY_UTILIZATION": "0.80",    "SM_VLLM_ENABLE_LORA": "true",    "SM_VLLM_MAX_LORA_RANK": "32",    "SM_VLLM_MAX_LORAS": "2",    "SM_VLLM_MAX_CPU_LORAS": "2",    "SM_VLLM_LIMIT_MM_PER_PROMPT": json.dumps({        "image": 1,        "audio": 0,        "video": 0,    }),    "VLLM_ALLOW_RUNTIME_LORA_UPDATING": "true",}
```

Here is what the main settings mean:

The runtime registration setting supports the adapter lifecycle used by SageMaker’s integration. [Hugging Face container configuration example](https://huggingface.co/docs/sagemaker/examples/sagemaker-sdk-multi-lora-adapters-vllm).

The rank limit must cover both adapters. Check r and any per-layer rank overrides in their configurations.

Also, MAX_LORAS=2 does not mean “only two requests at a time.” It controls the number of different adapters that can participate in a batch.

For a compatible prequantized BitsAndBytes checkpoint, the serving configuration may also need:

```
# Add only for a validated BitsAndBytes deployment.environment.update({    "SM_VLLM_QUANTIZATION": "bitsandbytes",    "SM_VLLM_LOAD_FORMAT": "bitsandbytes",    "SM_VLLM_DTYPE": "bfloat16",})
```

These settings tell the runtime how to load and compute with the checkpoint. They do not convert an arbitrary checkpoint into a compatible 4-bit model.

The weights can be stored in 4-bit form while computation uses BF16. This reduces weight memory, but images, activations, adapters, and the KV cache still need memory.

First, register the model and create the endpoint:

```
sm.create_model(    ModelName=MODEL_NAME,    ExecutionRoleArn=ROLE_ARN,    PrimaryContainer={        "Image": IMAGE_URI,        "ModelDataUrl": BASE_MODEL_S3,        "Environment": environment,    },)
sm.create_endpoint_config(    EndpointConfigName=ENDPOINT_CONFIG_NAME,    ExecutionRoleArn=ROLE_ARN,    ProductionVariants=[{        "VariantName": "AllTraffic",        "InstanceType": INSTANCE_TYPE,        "InitialInstanceCount": 1,        "RoutingConfig": {            "RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS",        },    }],)
sm.create_endpoint(    EndpointName=ENDPOINT_NAME,    EndpointConfigName=ENDPOINT_CONFIG_NAME,)
sm.get_waiter("endpoint_in_service").wait(    EndpointName=ENDPOINT_NAME,    WaiterConfig={"Delay": 30, "MaxAttempts": 120},)
```

This step creates a billable endpoint. Some containers also require a particular inference AMI or driver combination; set a compatible InferenceAmiVersion in the production variant when your selected image requires it.

Next, add the base inference component:

``` python
def wait_for_component(name, timeout=3600):    deadline = time.monotonic() + timeout    while time.monotonic() < deadline:        info = sm.describe_inference_component(            InferenceComponentName=name,        )        status = info["InferenceComponentStatus"]        if status == "InService":            return        if status in {"Failed", "UpdateFailed", "Deleting"}:            raise RuntimeError(                f"{name}: {status}: {info.get('FailureReason', '')}"            )        time.sleep(15)    raise TimeoutError(f"Timed out waiting for {name}")
sm.create_inference_component(    InferenceComponentName=BASE_COMPONENT,    EndpointName=ENDPOINT_NAME,    VariantName="AllTraffic",    Specification={        "ModelName": MODEL_NAME,        "ComputeResourceRequirements": {            "NumberOfAcceleratorDevicesRequired": 1,            "MinMemoryRequiredInMb": 8192,        },        "StartupParameters": {            "ModelDataDownloadTimeoutInSeconds": 1800,            "ContainerStartupHealthCheckTimeoutInSeconds": 1800,        },    },    RuntimeConfig={"CopyCount": 1},)wait_for_component(BASE_COMPONENT)
```

The 8,192 MB value is an example host-memory reservation, not GPU memory. Adjust it to your container and model-loading requirements.

This example creates one serving copy of the base. If you scale to multiple copies, each copy needs its own model weights.

Once the base is ready, attach both adapters:

```
adapters = {    ADAPTER_1_COMPONENT: ADAPTER_1_S3,    ADAPTER_2_COMPONENT: ADAPTER_2_S3,}
for component_name, artifact_uri in adapters.items():    sm.create_inference_component(        InferenceComponentName=component_name,        EndpointName=ENDPOINT_NAME,        Specification={            "BaseInferenceComponentName": BASE_COMPONENT,            "Container": {"ArtifactUrl": artifact_uri},        },    )    wait_for_component(component_name)
```

Notice that we do not request another GPU for each adapter. We link each child component to the existing base. [AWS adapter deployment example](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-adapt.html).

Save these resource names and artifact versions in a deployment manifest. You will need them for updates, troubleshooting, and cleanup.

Now for the part that makes this a VLM example: sending an actual image.

The message contains a text instruction and an image encoded as a data URL. This follows vLLM’s OpenAI-compatible multimodal request format. [vLLM image input examples](https://docs.vllm.ai/en/stable/features/multimodal_inputs/).

``` python
import base64from pathlib import Pathfrom PIL import Image
python
def image_message(image_path, instruction):    path = Path(image_path)    with Image.open(path) as image:        if image.format not in {"JPEG", "PNG", "WEBP"}:            raise ValueError("Use a JPEG, PNG, or WEBP image")        mime_type = Image.MIME[image.format]
encoded = base64.b64encode(path.read_bytes()).decode("ascii")    return [{        "role": "user",        "content": [            {"type": "text", "text": instruction},            {"type": "image_url", "image_url": {                "url": f"data:{mime_type};base64,{encoded}",            }},        ],    }]
python
def invoke(component_name, messages):    body = json.dumps({        "model": "shared-vlm",        "messages": messages,        "max_tokens": 256,        "temperature": 0,    }).encode("utf-8")
# A conservative application budget; check the API limits you use.    if len(body) > 5_500_000:        raise ValueError("Resize or compress the image before sending")
response = runtime.invoke_endpoint(        EndpointName=ENDPOINT_NAME,        InferenceComponentName=component_name,        ContentType="application/json",        CustomAttributes="route=/v1/chat/completions",        Body=body,    )    return json.loads(response["Body"].read())
messages = image_message(    "sample-table.png",    "Read this table and return the values as JSON.",)result = invoke(ADAPTER_1_COMPONENT, messages)print(result["choices"][0]["message"]["content"])
```

The container must support the chat route and the model’s chat template. Use the same prompt style and output format that the adapter learned during training.

For example, imagine a synthetic image containing:

```
Item       QuantityApples     3Oranges    5
```

An illustrative response might be:

```
{"items": [{"name": "Apples", "quantity": 3}, {"name": "Oranges", "quantity": 5}]}
```

That is an example of the intended behavior, not a measured result. Asking for JSON does not guarantee correct or valid JSON. Validate the response and compare its values with the image.

To use Adapter 2, change the component name and give it an instruction appropriate to its training:

```
messages = image_message(    "sample-image.png",    "Describe this image using the format you were trained to follow.",)result = invoke(ADAPTER_2_COMPONENT, messages)
```

The endpoint stays the same. The selected adapter changes.

You can also invoke BASE_COMPONENT to compare the original model with either adapter. This is a useful way to check whether fine-tuning actually improved the task.

A small JPEG is not necessarily a cheap model input. Image resolution and the model’s preprocessing strategy affect visual tokens, processing time, and memory.

For a table-reading task, crop away irrelevant background while keeping the text readable. Reducing resolution too aggressively can turn a clear number into an ambiguous one.

Base64 also increases the transmitted image size by roughly one third. Check both the serialized request size and the model’s context budget.

Test in stages:

If you receive ModelNotReadyException while an adapter loads, use bounded retries with backoff. Other errors need diagnosis rather than unlimited retries.

Suppose an adapter contains weights for the vision encoder, but the serving runtime cannot apply them.

Deleting those weights might allow the remaining adapter to load. It can also remove part of what the model learned.

Check the training configuration and the runtime’s supported targets. If trained modules are unsupported, use a compatible runtime or retrain with supported targets. Any conversion needs quality evaluation.

Some model and container combinations need workarounds involving quantization metadata, eager execution, KV-cache handling, or cache-connector startup settings.

These fixes solve different problems. For example, disabling a KV-transfer connector does not disable the vision encoder, and disabling a multimodal processor cache does not remove the language model’s KV cache.

Record each fix with the exact image version and the error it addresses. Retest when upgrading. Copying a collection of old environment flags into a new image can make troubleshooting harder.

Treat adapters as independently versioned artifacts.

For example, keep Adapter 1’s current component while creating a new component for its next version. Test the new one, then update the application’s component-name mapping.

This allows you to return to the previous version if evaluation finds a problem. During the overlap, allow for the additional adapter cache and memory demand.

Deleting the existing component and recreating it under the same name is another approach, but requests to that adapter can fail during the gap.

Delete the adapter components first, wait for their deletion, and then delete the base component. After that, delete the endpoint and wait for it to disappear before removing its endpoint configuration and model object.

S3 artifacts, custom container images, and retained logs need their own cleanup or retention policy.

For a first experiment, start with one compatible base, one adapter, and one small image. Once that works, attach Adapter 2 and test both together. This makes it much easier to tell whether a failure comes from image processing, adapter compatibility, or shared serving capacity.

[One LLM Model, Multiple Adapters: Deploying on AWS SageMaker](https://pub.towardsai.net/one-vision-language-model-multiple-adapters-deploying-on-amazon-sagemaker-26b7439d5d45) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.