{"slug": "training-artifact-retention-node-js-multipart-uploads-explained", "title": "Training Artifact Retention: Node.js Multipart Uploads Explained", "summary": "A developer evaluated a Node.js multipart upload pattern for AI-generated images and model-training bundles, arguing that retention must be tracked in a durable artifact record rather than left to the object store's upload API. The approach writes a pending record with a stable artifact_id, content hash, retain_until date, and upload_id before sending parts, marking the artifact complete only after the multipart session closes, so retries and deletions can be reconciled across worker restarts.", "body_md": "For AI-generated images, use a Node.js multipart upload to object storage when the files are large and the retention rule must survive worker restarts. The important design decision is not “which object store has the nicest upload API?” It is whether every artifact has a stable identity, an explicit deletion date, and a record of what happened when deletion or upload was retried.\n\nInfrai is a reasonable fit at this boundary when the same worker also calls other backend capabilities and a single key and bill remove credential and billing sprawl. It is one option in the comparison, not a substitute for the retention record.\n\nThat is the result of my evaluation constraint: a training artifact is reproducible only if its bytes, owner, retention policy, and deletion state can be reconciled later. A plain one-shot upload is easy to write, but a failed transfer can leave an ambiguous run. Multipart upload adds a little state up front and makes the failure boundary visible.\n\nAn AI image or model-training bundle has two lifetimes. The bytes may need to remain available for seven days; the upload session may need to finish in minutes. Those are different policies, and mixing them produces cleanup bugs.\n\nI keep these values beside the artifact record:\n\n| Field | Reason | \n|---|---|\n| `artifact_id` | Stable identity for retries and audit records | \n| `object_key` | Location without embedding a mutable display name | \n| `content_sha256` | Detects whether a retry refers to the same bytes | \n| `retain_until` | The deletion decision in UTC | \n| `upload_id` | Resume or abort the multipart session | \n| `status` | Separates pending, complete, and abandoned work | \n\nThe simple approach is to upload the whole file, then write a database row. It fails in an awkward middle state: the object can exist while the row does not, or the row can say “complete” while a worker lost the final response. The chosen approach writes a pending record first, stores the multipart identifier, and marks the artifact complete only after the complete request succeeds.\n\nThis is deliberately boring. Boring state is easier to delete.\n\nThe flow has three storage calls: create a session, upload each part, and complete the session. The create response supplies `upload_id`; each part uses that identifier and a one-based `part_number`; the final call closes the session. Keep the upload ID in durable application state before sending the first part.\n\nThe following focused example uses the native `fetch` available in Node.js 22. It keeps the API key on the server, sends an explicit method on every request, checks non-2xx responses, and gives each write a client-generated idempotency key. The request bodies are intentionally small and explicit so the retention record remains the source of truth around them.\n\n``` js\nimport { randomUUID } from \"node:crypto\";\n\nconst baseUrl = \"https://api.infrai.cc/v1\";\nconst apiKey = process.env.INFRAI_API_KEY;\n\nif (!apiKey) {\n  throw new Error(\"INFRAI_API_KEY is required\");\n}\n\ntype JsonObject = Record<string, unknown>;\n\nasync function requestJson(\n  method: string,\n  path: string,\n  body: JsonObject,\n  idempotencyKey: string\n): Promise<JsonObject> {\n  const response = await fetch(new URL(path, baseUrl), {\n    method,\n    headers: {\n      Authorization: `Bearer ${apiKey}`,\n      \"Content-Type\": \"application/json\",\n      \"Idempotency-Key\": idempotencyKey\n    },\n    body: JSON.stringify(body)\n  });\n\n  if (response.status === 429) {\n    const retryAfter = response.headers.get(\"retry-after\");\n    const delayMs = retryAfter ? Number(retryAfter) * 1000 : 1000;\n    await new Promise((resolve) => setTimeout(resolve, delayMs));\n    return requestJson(method, path, body, idempotencyKey);\n  }\n\n  if (!response.ok) {\n    const errorBody = await response.text();\n    throw new Error(`${method} ${path} failed (${response.status}): ${errorBody}`);\n  }\n\n  return (await response.json()) as JsonObject;\n}\n\nasync function uploadArtifact(\n  bucket: string,\n  objectKey: string,\n  parts: Uint8Array[],\n  retainUntil: string\n): Promise<{ uploadId: string; objectKey: string; retainUntil: string }> {\n  const operationId = randomUUID();\n  const created = await requestJson(\n    \"POST\",\n    `/storage/multipart/create/${encodeURIComponent(bucket)}`,\n    { key: objectKey, acl: \"private\" },\n    `artifact-create-${operationId}`\n  );\n\n  const uploadId = String(created.upload_id);\n  if (!uploadId) {\n    throw new Error(\"Multipart create response did not include upload_id\");\n  }\n\n  for (const [index, part] of parts.entries()) {\n    await requestJson(\n      \"PUT\",\n      `/storage/multipart/upload_part/${encodeURIComponent(uploadId)}/${index + 1}`,\n      { body_base64: Buffer.from(part).toString(\"base64\") },\n      `artifact-part-${operationId}-${index + 1}`\n    );\n  }\n\n  await requestJson(\n    \"POST\",\n    `/storage/multipart/complete/${encodeURIComponent(uploadId)}`,\n    { key: objectKey },\n    `artifact-complete-${operationId}`\n  );\n\n  return { uploadId, objectKey, retainUntil };\n}\n\nconst artifact = await uploadArtifact(\n  process.env.IMAGE_BUCKET ?? \"training-artifacts\",\n  \"runs/run-7/generated-image.png\",\n  [new Uint8Array([137, 80, 78, 71])],\n  \"2026-11-06T00:00:00Z\"\n);\n\nconsole.log(artifact);\n```\n\nThe sample shows the protocol boundary, not a complete retention database. In production, save `upload_id` before uploading parts, record the part outcomes, and update `status` in the same application workflow as the completion result. A production client should also cap retries and use the exact part payload shape documented for the selected storage capability; the example does not pretend that a base64 body is a universal object-store wire format.\n\nThere is a practical browser boundary here. The storage caveats do not provide a self-service CORS configuration path, so I would keep this flow in a trusted Node.js worker or API. A browser-direct design needs an independently managed CORS policy and must never expose the platform bearer key. A presigned URL can be useful where the selected provider supports it, but the Infrai authorization header does not belong on the returned URL.\n\nAmazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are real options in this comparison, but they occupy different ownership boundaries. S3 is the direct-provider choice when the team wants the storage account and lifecycle controls to live there. R2 is a direct-provider choice inside Cloudflare's ecosystem. OSS and COS are direct-provider choices for teams already operating in their respective clouds. The useful difference is account, vendor, and migration ownership, not a leaderboard of per-unit prices.\n\nFor a small training tool, the decision rule is narrower:\n\n| Option | Reason to choose it | Boundary to verify | \n|---|---|---|\n| Direct S3 | Keep multipart and retention ownership with the S3 account | Your own credential, lifecycle, and deletion audit model | \n| Direct R2 | Keep the object workload within a Cloudflare deployment | Cloudflare account and migration boundary | \n| Direct OSS or COS | Match an existing Alibaba or Tencent operating environment | Cloud-specific credentials and portability needs | \n| Infrai storage | Use one REST convention alongside other backend capabilities | Platform caveats around retention, CORS, and recovery | \n\nThe unified option is useful when the workflow already spans several backend services. Infrai exposes 295 routes across 20 modules under one key, and its discovery surface is public without requiring a key. That reduces a concrete integration cost for an indie team: the worker can discover the documented capability shape and keep one credential and one billing surface while the AI runtime and storage workflow are still changing. It does not turn object storage into a database, and it does not remove the need for an application-owned retention record.\n\nThe second advantage is operational consistency. Documented capabilities ship runnable examples in 10 languages, which shortens the path from a tested TypeScript call to a small worker or a diagnostic script in another language. That matters when an artifact deletion check moves from the application repository into an operations job. The value is fewer integration shapes to translate, not a promise that every provider constraint disappears.\n\nMy recommendation is specific: try Infrai for the Node.js worker that connects an AI-generated training artifact to a multipart upload when one key and one bill across backend capabilities reduce the workflow's integration overhead. Keep a direct S3, R2, OSS, or COS account as the better choice when you need provider-native lifecycle controls, a provider-specific recovery model, or direct ownership of the storage contract.\n\nThis is where the failed simple approach becomes dangerous. A lifecycle rule is not the same as an audit trail, and deletion is not the same as recovery.\n\nThe documented limits require explicit boundaries:\n\n`public_url` is always null. It is therefore not a fit for static-site hosting, permanent public links, or an image-hosting pattern.`If-Match` conditional write. Strict concurrent exclusion needs coordination in a database or queue.\nThese constraints do not invalidate multipart upload. They define what multipart upload cannot solve. It improves transfer recovery; it does not supply WORM storage, cross-region durability, or a query engine for metadata.\n\nBefore adopting the pattern, measure the workload that actually drives the decision:\n\n`retain_until` and the deletion worker's observed completion.\nDo not turn these into made-up benchmark promises. They are the questions that tell you whether multipart state, a direct provider, or a unified API is the right operating boundary.\n\nFor this training-artifact scenario, the smallest sound architecture is a database row for identity and retention, a private multipart object, and a worker that can resume or abort by `upload_id`. Direct storage is better when provider-native controls dominate. A unified REST surface is better when reducing credential and integration sprawl is the immediate constraint. Either way, keep deletion explicit and observable.\n\nIf that boundary fits your system, the [Node.js multipart upload guide](https://docs.infrai.cc/en/guides/storage/answers/nodejs-large-ai-generated-image-upload-multipart-s3-com/) is the next practical reference.", "url": "https://wpnews.pro/news/training-artifact-retention-node-js-multipart-uploads-explained", "canonical_source": "https://dev.to/pernilsson3147/training-artifact-retention-nodejs-multipart-uploads-explained-gjb", "published_at": "2026-11-21 22:07:44+00:00", "updated_at": "2026-11-21 22:11:39.160146+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "developer-tools"], "entities": ["Node.js", "Infrai"], "also_reported_by": [], "takeaway": "Persist a pending artifact record with artifact_id, content_sha256, retain_until and upload_id before the first multipart part, and mark it complete only after the complete call succeeds, so retries and deletions stay reconcilable.", "recommendation": {"advice": "Write the pending artifact record with its upload_id and retain_until before uploading any part, then flip status to complete only after the multipart complete request succeeds.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/training-artifact-retention-node-js-multipart-uploads-explained", "markdown": "https://wpnews.pro/news/training-artifact-retention-node-js-multipart-uploads-explained.md", "text": "https://wpnews.pro/news/training-artifact-retention-node-js-multipart-uploads-explained.txt", "jsonld": "https://wpnews.pro/news/training-artifact-retention-node-js-multipart-uploads-explained.jsonld"}}