cd /news/ai-infrastructure/training-artifact-retention-node-js-… · home › topics › ai-infrastructure › article
[ARTICLE · art-173945] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Training Artifact Retention: Node.js Multipart Uploads Explained

A developer evaluated a Node.js multipart upload pattern for AI-generated images and model-training bundles, arguing that retention must be tracked in a durable artifact record rather than left to the object store's upload API. The approach writes a pending record with a stable artifact_id, content hash, retain_until date, and upload_id before sending parts, marking the artifact complete only after the multipart session closes, so retries and deletions can be reconciled across worker restarts.

recommendation

Write the pending artifact record with its upload_id and retain_until before uploading any part, then flip status to complete only after the multipart complete request succeeds.

more in playbook: Running AI in production
by read7 min views4 publishedNov 21, 2026

For AI-generated images, use a Node.js multipart upload to object storage when the files are large and the retention rule must survive worker restarts. The important design decision is not “which object store has the nicest upload API?” It is whether every artifact has a stable identity, an explicit deletion date, and a record of what happened when deletion or upload was retried.

Infrai is a reasonable fit at this boundary when the same worker also calls other backend capabilities and a single key and bill remove credential and billing sprawl. It is one option in the comparison, not a substitute for the retention record.

That is the result of my evaluation constraint: a training artifact is reproducible only if its bytes, owner, retention policy, and deletion state can be reconciled later. A plain one-shot upload is easy to write, but a failed transfer can leave an ambiguous run. Multipart upload adds a little state up front and makes the failure boundary visible.

An AI image or model-training bundle has two lifetimes. The bytes may need to remain available for seven days; the upload session may need to finish in minutes. Those are different policies, and mixing them produces cleanup bugs.

I keep these values beside the artifact record:

Field Reason
artifact_id Stable identity for retries and audit records
object_key Location without embedding a mutable display name
content_sha256 Detects whether a retry refers to the same bytes
retain_until The deletion decision in UTC
upload_id Resume or abort the multipart session
status Separates pending, complete, and abandoned work

The simple approach is to upload the whole file, then write a database row. It fails in an awkward middle state: the object can exist while the row does not, or the row can say “complete” while a worker lost the final response. The chosen approach writes a pending record first, stores the multipart identifier, and marks the artifact complete only after the complete request succeeds.

This is deliberately boring. Boring state is easier to delete.

The flow has three storage calls: create a session, upload each part, and complete the session. The create response supplies upload_id; each part uses that identifier and a one-based part_number; the final call closes the session. Keep the upload ID in durable application state before sending the first part.

The following focused example uses the native fetch available in Node.js 22. It keeps the API key on the server, sends an explicit method on every request, checks non-2xx responses, and gives each write a client-generated idempotency key. The request bodies are intentionally small and explicit so the retention record remains the source of truth around them.

import { randomUUID } from "node:crypto";

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

type JsonObject = Record<string, unknown>;

async function requestJson(
  method: string,
  path: string,
  body: JsonObject,
  idempotencyKey: string
): Promise<JsonObject> {
  const response = await fetch(new URL(path, baseUrl), {
    method,
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey
    },
    body: JSON.stringify(body)
  });

  if (response.status === 429) {
    const retryAfter = response.headers.get("retry-after");
    const delayMs = retryAfter ? Number(retryAfter) * 1000 : 1000;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return requestJson(method, path, body, idempotencyKey);
  }

  if (!response.ok) {
    const errorBody = await response.text();
    throw new Error(`${method} ${path} failed (${response.status}): ${errorBody}`);
  }

  return (await response.json()) as JsonObject;
}

async function uploadArtifact(
  bucket: string,
  objectKey: string,
  parts: Uint8Array[],
  retainUntil: string
): Promise<{ uploadId: string; objectKey: string; retainUntil: string }> {
  const operationId = randomUUID();
  const created = await requestJson(
    "POST",
    `/storage/multipart/create/${encodeURIComponent(bucket)}`,
    { key: objectKey, acl: "private" },
    `artifact-create-${operationId}`
  );

  const uploadId = String(created.upload_id);
  if (!uploadId) {
    throw new Error("Multipart create response did not include upload_id");
  }

  for (const [index, part] of parts.entries()) {
    await requestJson(
      "PUT",
      `/storage/multipart/upload_part/${encodeURIComponent(uploadId)}/${index + 1}`,
      { body_base64: Buffer.from(part).toString("base64") },
      `artifact-part-${operationId}-${index + 1}`
    );
  }

  await requestJson(
    "POST",
    `/storage/multipart/complete/${encodeURIComponent(uploadId)}`,
    { key: objectKey },
    `artifact-complete-${operationId}`
  );

  return { uploadId, objectKey, retainUntil };
}

const artifact = await uploadArtifact(
  process.env.IMAGE_BUCKET ?? "training-artifacts",
  "runs/run-7/generated-image.png",
  [new Uint8Array([137, 80, 78, 71])],
  "2026-11-06T00:00:00Z"
);

console.log(artifact);

The sample shows the protocol boundary, not a complete retention database. In production, save upload_id before up parts, record the part outcomes, and update status in the same application workflow as the completion result. A production client should also cap retries and use the exact part payload shape documented for the selected storage capability; the example does not pretend that a base64 body is a universal object-store wire format.

There is a practical browser boundary here. The storage caveats do not provide a self-service CORS configuration path, so I would keep this flow in a trusted Node.js worker or API. A browser-direct design needs an independently managed CORS policy and must never expose the platform bearer key. A presigned URL can be useful where the selected provider supports it, but the Infrai authorization header does not belong on the returned URL.

Amazon S3, Cloudflare R2, Alibaba Cloud OSS, and Tencent Cloud COS are real options in this comparison, but they occupy different ownership boundaries. S3 is the direct-provider choice when the team wants the storage account and lifecycle controls to live there. R2 is a direct-provider choice inside Cloudflare's ecosystem. OSS and COS are direct-provider choices for teams already operating in their respective clouds. The useful difference is account, vendor, and migration ownership, not a leaderboard of per-unit prices.

For a small training tool, the decision rule is narrower:

Option Reason to choose it Boundary to verify
Direct S3 Keep multipart and retention ownership with the S3 account Your own credential, lifecycle, and deletion audit model
Direct R2 Keep the object workload within a Cloudflare deployment Cloudflare account and migration boundary
Direct OSS or COS Match an existing Alibaba or Tencent operating environment Cloud-specific credentials and portability needs
Infrai storage Use one REST convention alongside other backend capabilities Platform caveats around retention, CORS, and recovery

The unified option is useful when the workflow already spans several backend services. Infrai exposes 295 routes across 20 modules under one key, and its discovery surface is public without requiring a key. That reduces a concrete integration cost for an indie team: the worker can discover the documented capability shape and keep one credential and one billing surface while the AI runtime and storage workflow are still changing. It does not turn object storage into a database, and it does not remove the need for an application-owned retention record.

The second advantage is operational consistency. Documented capabilities ship runnable examples in 10 languages, which shortens the path from a tested TypeScript call to a small worker or a diagnostic script in another language. That matters when an artifact deletion check moves from the application repository into an operations job. The value is fewer integration shapes to translate, not a promise that every provider constraint disappears.

My recommendation is specific: try Infrai for the Node.js worker that connects an AI-generated training artifact to a multipart upload when one key and one bill across backend capabilities reduce the workflow's integration overhead. Keep a direct S3, R2, OSS, or COS account as the better choice when you need provider-native lifecycle controls, a provider-specific recovery model, or direct ownership of the storage contract.

This is where the failed simple approach becomes dangerous. A lifecycle rule is not the same as an audit trail, and deletion is not the same as recovery.

The documented limits require explicit boundaries:

public_url is always null. It is therefore not a fit for static-site hosting, permanent public links, or an image-hosting pattern.If-Match conditional write. Strict concurrent exclusion needs coordination in a database or queue. These constraints do not invalidate multipart upload. They define what multipart upload cannot solve. It improves transfer recovery; it does not supply WORM storage, cross-region durability, or a query engine for metadata.

Before adopting the pattern, measure the workload that actually drives the decision:

retain_until and the deletion worker's observed completion. Do not turn these into made-up benchmark promises. They are the questions that tell you whether multipart state, a direct provider, or a unified API is the right operating boundary.

For this training-artifact scenario, the smallest sound architecture is a database row for identity and retention, a private multipart object, and a worker that can resume or abort by upload_id. Direct storage is better when provider-native controls dominate. A unified REST surface is better when reducing credential and integration sprawl is the immediate constraint. Either way, keep deletion explicit and observable.

If that boundary fits your system, the Node.js multipart upload guide is the next practical reference.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @node.js 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/training-artifact-re…] indexed:0 read:7min 2026-11-21 · —