Zero-Copy, Multi-Protocol Storage: The Architecture Pattern Data Pipelines Are Missing

A developer built a proof of concept on Amazon FSx for NetApp ONTAP (FSxN) demonstrating a zero-copy, multi-protocol storage pattern that eliminates the need for sync jobs in data pipelines. The pattern allows the same data to be accessed via SMB, NFS, and S3 API without replication, reducing latency and operational complexity. The developer applied it to an insurance claims intake pipeline, enabling near-real-time processing instead of batch sweeps.

Every cloud architecture diagram you've ever drawn for a "modernize the legacy system" project probably has the same box in it: a sync job. Something that watches an old file share and copies what it finds into S3, so the new, shiny, event-driven, AI-powered part of the stack has something to read. That box is a lie we tell ourselves. It says "the data is now in the cloud." What it actually means is: there are now two copies of the data, and something has to keep them the same. This post is about a pattern that gets rid of that box entirely — zero-copy, multi-protocol storage — and a proof of concept I built on Amazon FSx for NetApp ONTAP FSxN that uses it to turn an insurance claims intake pipeline into something that reacts in seconds instead of on the next scheduled sweep. "Multi-protocol" storage means the same underlying data can be accessed through more than one protocol — SMB, NFS, and the S3 API, in this case — without translation, replication, or a gateway service in between. "Zero-copy" means that when a second consumer reads the data through a different protocol, it isn't reading a copy. It's reading the same bytes , through a different door. That distinction sounds academic until you draw the two architectures next to each other. The bridge pattern what most teams build : php flowchart LR A Legacy system<br/ writes over SMB/NFS -- B File share B -- C Sync job / gateway C -- D S3 bucket D -- E Cloud-native processing The zero-copy pattern: php flowchart LR A Legacy system<br/ writes over SMB/NFS -- B Same volume B -- "S3 Access Point<br/ no copy " -- E Cloud-native processing In the bridge pattern, every file exists twice, on two different consistency models, and something — a Lambda, a cron job, a Kafka Connect worker — is responsible for making sure copy two matches copy one. That something adds latency the sync interval , cost storage twice over, plus the compute to move it , and a new failure mode: drift. When copy two is wrong, is it a bug in the sync job, a race condition, a partial failure that never retried? Now you're debugging a distributed consistency problem to answer a question that shouldn't have existed. Amazon FSx for NetApp ONTAP removes the sync job because ONTAP volumes are natively multi-protocol. The same volume can be exported as an NFS share, a CIFS/SMB share, and fronted by an S3 Access Point that serves GetObject calls straight off the underlying blocks. No object gets written to S3 — the Access Point is a read path into the existing volume. One copy of the data. Two or three ways to reach it, chosen by whatever's most convenient for the consumer: a legacy Windows app keeps mapping a network drive; a Lambda function calls boto3.client "s3" .get object . The obvious win is cost and reduced operational surface — no second storage tier, no reconciliation logic, one less system that pages someone at 2am. But the bigger win is architectural: it changes what "real-time" means for systems that were never designed to be real-time. Legacy file-based systems don't emit events. They write files. If your AI/analytics layer only exists downstream of a sync job, your event is "the sync job ran," not "the document arrived." You've inherited the legacy system's batch cadence even though you built a completely event-driven pipeline on top of it. Zero-copy access solves the read side of that problem — cloud compute can see the file the instant it exists, with no propagation delay. But you still need something to solve the notification side: how does anything know to look? That's where the second piece of this pattern comes in, and it's the part that took empirical testing, not documentation reading, to get right. Insurance claims intake is a good stress test for this pattern because it has all three of the constraints that make the bridge pattern painful: I built a proof of concept — fsx-ontap-fnol-poc https://github.com/varunrai/fsx-ontap-fnol-poc — around exactly this: a simulated claims system writes a photo and a claim form over SMB to an FSxN volume, and from that single write, a serverless pipeline does real-time AI extraction and creates tamper-evident audit evidence, with zero code or workflow changes on the writer's side. flowchart TD subgraph client "Claims system simulated " EC2 "EC2 demo client<br/ SMB write" end subgraph fsxn "Amazon FSx for NetApp ONTAP" VOL "Claim intake volume<br/ NTFS security, AD-joined SVM " SNAP "Snapshots / SnapLock WORM" S3AP "S3 Access Point<br/ zero-copy read " end subgraph events "Event-driven automation" FPOL "FPolicy engine" ECS "ECS Fargate<br/ FPolicy TCP server" SQS "SQS queue" end subgraph compute "Serverless processing" CP "claim processor Lambda<br/ Bedrock AI extraction" DDB "DynamoDB<br/ claims table" STREAM "DynamoDB Streams" ES "evidence stamper Lambda" end DASH "Streamlit dashboard" EC2 -- "1. write claim files" -- VOL VOL -- "2. file close event" -- FPOL FPOL -- "3. TCP notify" -- ECS ECS -- "4. normalize + enqueue" -- SQS SQS -- "5. trigger" -- CP CP -- "6. zero-copy read" -- S3AP S3AP -.- VOL CP -- "7. extracted JSON + fraud score" -- DDB DDB -- "8. change event" -- STREAM STREAM -- ES ES -- "9. snapshot / SnapLock" -- SNAP DDB -- "live claims + timing" -- DASH Four things do the load-bearing work here: file close event the instant a claim document finishes writing over SMB. An ECS-hosted TCP server receives that notification and drops a normalized message on SQS. There's no polling, no bucket-notification workaround — the pipeline starts within seconds of the document landing.Here's the part that separates "read the docs and build it" from "actually validate the thing you're relying on." The natural assumption going in was that any write to the volume — over SMB, NFS, or the S3 Access Point — would fire an FPolicy event, since it's the same underlying volume regardless of which door you walked through. That assumption is wrong, and it only cost nothing to find out because it was tested empirically against a live deployment before the architecture was locked in: writes through the S3 Access Point never generate an FPolicy notification , regardless of security style or which ONTAP file operations are enabled. FPolicy is wired to the NAS protocol stack SMB/NFS , not to the S3 gateway path. That's why this pipeline uses SMB as the sole ingestion trigger and keeps the S3 Access Point strictly for zero-copy reads on the compute side — the two protocols play different roles in the same architecture, and conflating them would have quietly broken the "process within seconds" promise for any client that happened to write through S3 instead. The lesson generalizes past this one POC: multi-protocol storage gives you multiple ways to read and write the same data — it does not mean every protocol has identical semantics. If your architecture depends on an event, a lock, or a consistency guarantee, verify it against the actual protocol you're using for that operation, not against "the storage layer" in the abstract. Numbers from the working POC: a live end-to-end run — SMB write, FPolicy notification, SQS, Bedrock extraction, DynamoDB write, ONTAP snapshot — consistently completes in a few seconds from SQS receipt to the DynamoDB write, with Bedrock inference dominating that latency. The S3 Access Point read is visible immediately after the SMB write completes, with no measurable consistency delay found across single or concurrent 20-way writes on either SMB or NFS. Compare that to the bridge-pattern alternative: a sync job on some interval 5 minutes if you're aggressive, an hour if you're not , a second S3 bucket to secure and pay for, and a reconciliation process to catch the writes that didn't sync. The zero-copy version isn't a faster version of that pipeline — it's a different pipeline that doesn't have that failure mode to begin with. FNOL intake is the use case in this POC, but the underlying shape — legacy protocol writers, cloud-native readers, one copy of the data, an audit trail that can't be edited after the fact — shows up anywhere a regulated or slow-moving system produces files that a modern pipeline needs to react to quickly: The pattern is the point, not the insurance angle. If you've got a bridge-pattern sync job in your architecture today, it's worth asking whether the storage underneath it is actually capable of serving both sides directly — because the box you're trying to eliminate might not need to exist at all. The full proof of concept — Terraform for the FSxN/ECS/Lambda/DynamoDB stack, the FPolicy TCP server, the Bedrock-backed claim processor, and the empirical protocol tests referenced above — is on GitHub at varunrai/fsx-ontap-fnol-poc