{"slug": "async-inference-in-practice-a-video-indexing-service-on-ray-serve", "title": "Async inference in practice: a video-indexing service on Ray Serve", "summary": "Ray Serve's async inference feature was used to build a video-indexing service that returns a task ID immediately and processes videos in the background, with the IndexingIngress deployment enqueuing tasks, VideoIndexConsumer autoscaling on queue depth, and VideoEncoder holding the SigLIP model on GPU. The service handles requests that can take tens of minutes, addressing issues of long-running model calls and burst traffic.", "body_md": "# Async inference in practice: a video-indexing service on Ray Serve\n\n[Harshit Agarwal](/blog?author=harshit-agarwal)| August 18, 2026\n\n*In an *__earlier post__*, we introduced asynchronous inference in Ray Serve: a way to run long-running model calls off the request path, backed by a message queue, with automatic retries and queue-depth autoscaling. This post is a practical follow-up. We build a real service on top of that feature, a video-indexing pipeline, run it under a heavy load, and compare it against a common managed alternative.*\n\n## LinkA quick recap: what async inference is for\n\nIn serving ML models, some applications make a single model call that runs for several seconds to minutes, e.g. - transcribing an hour of audio, indexing a video, or generating an image or a video. On a normal synchronous HTTP endpoint the request stays open for the whole computation. At millisecond latencies that coupling is invisible, but for calls this long it creates two problems.\n\nFirst, a burst of traffic piles onto a fleet that takes seconds or minutes to finish each request, so the workers fall behind and clients, or any intermediary hop, start timing out. Also, a connection held open for minutes has much more exposure to a dropped packet or a failed hop. As a result, the service and overall system gets less reliable at exactly the moments you can't predict.\n\nAsync inference solves these problems by decoupling the request scheduling from the actual computation. The client submits a job and gets an id back right away, the work goes onto a message queue, a pool of workers, autoscaled to the queue depth, processes it in the background, and the client polls for the result.\n\nWe covered the API and the design in the [ earlier post](https://www.anyscale.com/blog/ray-serve-autoscaling-async-inference-custom-routing), so here we focus on building a service on top of it, and benchmark it under real-world traffic. The pieces Ray Serve provides, in brief:\n\n`@task_consumer / @task_handler`\n\n- these decorators turn a deployment method into a background worker that pulls from a queue.`TaskProcessorConfig`\n\n- configuration that points your code at a message broker you run (e.g. Redis or RabbitMQ are supported) and adds an*at-least-once*delivery mechanism, retries, androuting.__dead-letter__`enqueue_task_sync(...)`\n\n- function that submits work from the producer side and returns a task id which client can poll for the result.`AsyncInferenceAutoscalingPolicy`\n\n- it is the autoscaling policy that scales the worker pool based on queue depth.\n\n## LinkThe application: a video-indexing service\n\nVideo indexing is a natural fit for async inference. As each request demands a whole video to download, decode, and embed,it can take seconds to minutes of work, and the traffic tends to arrive in bursts such as a batch upload or a nightly backfill. This video indexing service will take an S3 URI and return a task id immediately, and in the background it downloads the video, splits it into frames with ffmpeg, embeds the frames with SigLIP on a GPU, and writes the vectors back to S3, a pipeline that can take tens of minutes per request.\n\nThe application, comprises of three deployments, each scaling on its own signal:\n\n**IndexingIngress**(CPU) - this deployment accepts`POST /index`\n\n, enqueues the task sent by the user, and returns the task id.**VideoIndexConsumer**(CPU) - the`@task_consumer`\n\n, it consumes tasks from the queue, and downloads and chunks the video with ffmpeg, then calls the encoder. This is the deployment that autoscales on queue depth.**VideoEncoder**(GPU) - a standard deployment holding the SigLIP neural network. The video index consumer deployment calls it through a handle, so frames move between the two over Ray's RPC rather than the network, and it scales on its own GPU load.\n\nThe producer (the deployment which is accepting the user’s request) is a very simple ingress deployment. It enqueues the task and returns an id, so the caller never waits on the work:\n\n``` python\n@serve.ingress(fastapi_app)\nclass IndexingIngress:\n    def __init__(self, consumer):\n        self.adapter = instantiate_adapter_from_config(PROCESSOR_CONFIG)\n\n    @fastapi_app.post(\"/index\")\n    async def index(self, req: IndexRequest):\n        result = self.adapter.enqueue_task_sync(\n            task_name=TASK_INDEX_VIDEO,\n            kwargs={\"video_uri\": req.video_uri, \"video_id\": req.video_id},\n        )\n        return {\"task_id\": result.id, \"status\": result.status}\n```\n\nThe video index consumer is a simple synchronous Python code. It is idempotent on the video id, chunks the video on CPU, calls the GPU encoder through a handle, and stores the result:\n\n```\n@serve.deployment(ray_actor_options={\"num_cpus\": FFMPEG_THREADS}, ...)\n@task_consumer(task_processor_config=PROCESSOR_CONFIG)\nclass VideoIndexConsumer:\n    def __init__(self, encoder):\n        self.encoder = encoder               # GPU DeploymentHandle\n\n    @task_handler(name=TASK_INDEX_VIDEO)\n    def index_video(self, video_uri, video_id=None):\n        if is_done(video_id):                  # idempotent on redelivery\n            return {\"status\": \"skipped_already_indexed\"}\n        chunks = chunk_video(download(video_uri))         # CPU: ffmpeg\n        refs = [self.encoder.remote(c.frames) for c in chunks]\n        vectors = [r.result()[\"frame_embeddings\"] for r in refs]\n        write_video_embeddings(video_id, vectors)         # store to S3\n        mark_done(video_id)\n        return {\"status\": \"indexed\", \"num_chunks\": len(chunks)}\n```\n\n** Note** - For the benchmarks below, we fused the\n\n**VideoIndexConsumer**(CPU) and\n\n**VideoEncoder**(GPU) deployments into a single GPU worker, while keeping the same lightweight ingress in front. We did this to make the comparison as fair as possible and to better match the deployment model supported by the alternative, Amazon SageMaker.\n\nSageMaker does not support deploying these two components as separate CPU and GPU services behind a single endpoint. The only alternative would be to expose them as completely separate endpoints; that route would put SageMaker at a further disadvantage: the intermediate tensors would have to travel between the two endpoints over the network, passed by value, whereas Ray Serve passes them by reference through its shared object store. Therefore, all benchmark results presented below were collected using a two-deployment architecture (the ingress and one fused worker), rather than the three-deployment architecture described above.\n\n## LinkBehavior under load\n\nOur goal was to stress-test the service and evaluate both its reliability and performance under sustained overload. Ideally, the service should scale with incoming traffic, process requests as quickly as possible, and do so without dropping user requests.\n\nWe evaluate reliability by measuring the number of failed requests during the load test, and performance by observing how the system responds to increasing load — particularly its ability to scale up and down in response to traffic while maintaining throughput and stability.\n\nTo exercise both aspects, we ran a 20-minute flood test at a sustained **50 RPS** with periodic spikes to **100 RPS**. This workload is approximately **5–10×** higher than what a 4-GPU fleet can process, resulting in roughly **67,000 video requests** over the duration of the test. Since the incoming request rate significantly exceeds the fleet's processing capacity, a request backlog is inevitable, making this a good test of the system's scaling behavior and resilience under sustained pressure.\n\nAs a result, we observed\n\n**Replicas scaled with demand**- As the backlog grew, the application automatically increased the number of GPU replicas from** 1**to** 4**, which was the configured maximum. Once the backlog was cleared, it scaled back down to** 1**. There was no fixed fleet and no manual tuning per burst.** Zero failed requests**- All**~67,000 video requests** were successfully accepted, queued, and eventually processed. The dead-letter queue remained empty throughout the test, indicating that no requests were dropped or failed during the sustained overload.\n\n## LinkComparing against a managed alternative: Amazon SageMaker\n\nTo put these numbers in context, we ran the same workload on [Amazon SageMaker Async Inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html), a common managed solution for this type of workload. Like our setup, SageMaker queues incoming requests, scales based on the backlog, and reads from and writes to S3. Both setups used the same hardware (\n\n**4× NVIDIA T4 GPUs**), the same\n\n**1-frame SigLIP** workload, and the same\n\n**20-minute flood test**, with both starting from a single cold instance. Since a SageMaker async endpoint runs a single model container, we used the same single, fused deployment on the Ray Serve side, as described above.\n\nThis means the deployment architecture, hardware, and workload were the same on both sides, making the orchestration engine the primary difference between the two systems.\n\n|\n|\n|\n| ~155 s | ~589 s |\n| ~5 s | ~104 s |\n| 1 policy block | 2 step policies + 2 alarms (explained below)* |\n| 0 | 0 |\n| 12.8 ms | 11.8 ms |\n\n* We autoscaled the SageMaker endpoint with two Application Auto Scaling step policies, each tied to a CloudWatch alarm. A **backlog-fast** alarm fires when __ApproximateBacklogSizePerInstance__ hits 5 or more (on a single 60-second datapoint), triggering the **fast-out** policy to jump straight to the 4-instance cap, and **backlog-empty** alarm fires when the total __ApproximateBacklogSize__ drops below 1, triggering the **fast-in** policy to drop back to a single instance.\n\nBelow are the few observations from the above runs:\n\n**Scale-up is close.** Both reach a full 4-GPU fleet in the same ballpark, and most of that time went to node provisioning, which both pay. The difference is in the decision to scale: Ray Serve polls the queue depth directly and reacts in seconds, while SageMaker's decision is bounded by the resolution of its CloudWatch metric.**Scale-down differs more.** Once the queue was empty, Ray Serve scaled back down within a few seconds. The fastest scale-down we saw with SageMaker was about**104 seconds**.** Ray Serve was easier to set up.**The Ray Serve side is one autoscaling policy block. To get similar behavior in SageMaker, we had to configure multiple autoscaling policies and several CloudWatch alarms.**Neither dropped anything.** Both processed the full flood with an empty dead-letter queue.\n\nA note on the SageMaker setup: autoscaling is configured to be as responsive as the platform allows. It uses a single 60-second CloudWatch alarm on the **ApproximateBacklogSize** and **ApproximateBacklogSizePerInstance** metric to trigger both scale-out and scale-in actions. Faster scaling isn't possible because these metrics are standard-resolution metric, so CloudWatch alarms cannot evaluate it more frequently than once every 60 seconds.\n\n### LinkEnd-to-end latency\n\nScaling speed is only part of the story. Another is - **how long a user has to wait** for their request to finish end to end. To measure this, we tracked the time from when a video was submitted until its embeddings were written to S3 for every request in the flood test, and below are the results:.\n\nEach request had a **base processing time of about 1–2 seconds**, which included downloading the video, running FFmpeg, generating embeddings, and writing the results to S3. This part was nearly the same on both platforms. The difference came from **waiting in the queue**. Since there were only a limited number of GPUs available, requests had to wait for a free GPU before they could be processed. The longer a request waits in the queue, the higher its overall latency.\n\nRay Serve responded to the growing backlog more quickly and started new replicas sooner than SageMaker. As a result, requests spent less time waiting in the queue, so users received their results earlier.\n\n## LinkWhen async inference is the right fit\n\nAsync inference is useful for workloads that take longer to finish and where keeping an HTTP connection open is not practical. It is also a good fit for workloads with sudden traffic spikes, since requests can wait in a queue until resources are available. With Ray Serve, you get request queuing, reliable request processing, and automatic scaling based on queue size with just a small amount of configuration. Since it is built into Ray Serve, the same application can run on any cloud, on-premises, or even on a laptop without code changes.\n\nThe feature is available in Ray Serve today; You can find more details in the [ Ray Serve documentation](https://docs.ray.io/en/latest/serve/asynchronous-inference.html).\n\n## LinkReferences\n\n- the post that introduced the feature this one builds on.__Ray Serve: autoscaling, async inference, and custom routing____SageMaker inference launches faster auto scaling for generative AI models__\n\n[ SageMaker supported-features matrix](https://docs.aws.amazon.com/sagemaker/latest/dg/model-deploy-feature-matrix.html) - async inference is single-container, which is why the comparison used a single collapsed deployment.", "url": "https://wpnews.pro/news/async-inference-in-practice-a-video-indexing-service-on-ray-serve", "canonical_source": "https://anyscale.com/blog/ray-serve-async-inf-in-practice", "published_at": "2026-08-18 16:00:00+00:00", "updated_at": "2026-08-19 00:11:13.109973+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-tools"], "entities": ["Ray Serve", "SigLIP", "ffmpeg", "S3", "Redis", "RabbitMQ", "Harshit Agarwal"], "alternates": {"html": "https://wpnews.pro/news/async-inference-in-practice-a-video-indexing-service-on-ray-serve", "markdown": "https://wpnews.pro/news/async-inference-in-practice-a-video-indexing-service-on-ray-serve.md", "text": "https://wpnews.pro/news/async-inference-in-practice-a-video-indexing-service-on-ray-serve.txt", "jsonld": "https://wpnews.pro/news/async-inference-in-practice-a-video-indexing-service-on-ray-serve.jsonld"}}