8 min read

Image pipeline

The live server never resizes an image on request. Every width and format a page might ask for is generated once, right after an upload is confirmed, by a worker job — not by next/image at request time, and not by a CDN transform.

Why upload-time, not request-time

A request-time image transform (Next.js’s default optimizer, a Cloudflare Image Resizing rule) means the first visitor to ever request a given size pays for the resize, and the box doing it has to be sized for that spike. Upload-time processing means uploads are slightly slower (a few seconds in the background) and every page load is fast — a much better trade for a small deployment. It also means EXIF (see below) is stripped from everything stored, not only from what a resize happened to touch.

What happens, in order

1. Confirm. The browser finishes its presigned PUT; confirmUpload (packages/core/src/media/service.ts) does a server-side HEAD against the object — a browser can lie about Content-Length at presign time, so this is the real size check, enforced per type (images 10 MB, video/mp4 100 MB; see packages/storage/src/keys.ts’s maxUploadBytesFor). An image row goes to processing; a video (no pipeline exists for video) goes straight to ready.
2. Enqueue. The API router (packages/api/src/routers/media.ts) enqueues a {media}-process BullMQ job (packages/queue’s mediaProcessQueue), keyed by media id so a retried confirm call can’t double-enqueue. The Shopify importer is the other producer; see Shopify imports below.
3. Process (apps/worker/src/media-process.ts, calling packages/core/src/media/pipeline.ts’s processImage):
   • limitInputPixels (40 MP) rejects a decompression bomb before any full decode.
   • Auto-orient from EXIF, then strip all metadata — GPS included. This is not a targeted “remove GPS” step: sharp strips EXIF/ICC/XMP by default unless you call .withMetadata(), so simply never calling it is the strip. Nothing GPS-tagged is kept anywhere, including in the master.
   • Cap the long edge at 2560px (MAX_MASTER_LONG_EDGE) — this cleaned, capped copy is the master, and it replaces the original at the same storage key. The pre-pipeline original (which may have carried GPS) is never left reachable under any key once this step finishes.
   • Generate every width in VARIANT_WIDTHS (320/640/1280/1920; media made before 960 was dropped may still carry a 960 variant, and delivery uses whatever widths the row has) that doesn’t exceed the master’s width — never upscale — in both AVIF and WebP.
   • Hash a ThumbHash (thumbhash, MIT) from a small decode of the master.
4. Write back. markMediaProcessed sets status: 'ready', variants (a { "<width>": { avif, webp } } map of public URLs), thumbhash, and the real width/height. A job that exhausts BullMQ’s retries (5, exponential backoff) instead calls markMediaFailed, setting status: 'failed' with the error, so the admin media library can show it and let the merchant delete and re-upload.

Animated GIF

Converted to a single animated WebP master, not resized into multiple widths. Two reasons: WebP’s animated encoder is materially smaller than GIF at the same look, and generating five widths of an animated asset multiplies encode cost per frame — not a good trade on a small box for a feature (animated product media) used rarely. The WebP master still gets the same auto-orient + full metadata strip and a ThumbHash from its first frame. Because the extension changes (.gif → .webp), this is the one case where the master writes to a new key — the worker deletes the original key immediately after, so nothing GPS-tagged is left behind under it either.

Delivery: <Picture> and the loader

packages/core/src/media/loader.ts has the pure mapping logic (no sharp — that stays in pipeline.ts, worker/build-only, so it never ends up in an app bundle):

• pickVariantUrls(media, targetWidth) — nearest generated width at or below the target (never upscale), falling back to the original url when variants is null (still processing, failed, or legacy/imported media the pipeline hasn’t reached yet).
• mediaImageLoader — a next/image loader function. Its contract is one URL per call, so it can’t vary by Accept header; it always resolves to the WebP variant. Use it only where a real next/image component is unavoidable.
• thumbhashPlaceholder(base64) — decodes straight to a small PNG data: URL server-side (thumbhash’s thumbHashToDataURL), for the blur-up placeholder.

Everywhere that matters (packages/blocks/src/components/picture.tsx, wired into product-card.tsx), a real <picture> element is used instead: an AVIF <source>, a WebP <source>, both with a full srcSet across every generated width, and a plain <img src={original}> fallback for unprocessed media. The browser negotiates format and width itself — no loader indirection, no JS. images.formats: ['image/avif', 'image/webp'] is set in both apps’ next.config.ts for the remaining call sites that still go through the default optimizer (blog/CMS content images, anything not yet migrated to <Picture>).

<Picture> also sets the ThumbHash as the <img>’s own inline background-image (decoded with thumbhash’s thumbHashToDataURL directly — packages/blocks stays free of a @commerce/core dependency, so it doesn’t call thumbhashPlaceholder for this). No wrapper element or JS: the blur shows through until the real image paints over it, and a corrupt/missing hash just renders with no placeholder rather than throwing.

Storage usage and quota

storageUsageBytes(tx, storeId) sums media.size for status = 'ready' rows — no new column, computed on read. checkStorageQuota(tx, storeId, quotaBytes?) compares that against a quota (default: STORAGE_QUOTA_BYTES_DEFAULT env var, or a generous 20 GB constant) and reports { usedBytes, quotaBytes, ok }. Nothing calls this to block an upload yet — it’s a hook for a future “Settings > Files > usage” page or a billing gate, kept deliberately schema-free so it costs nothing today.

Shopify imports

The importer (packages/plugin-shopify-import) re-hosts every Shopify image in the store’s own storage (src/media.ts’s createMediaHost) and writes the row as processing, like a confirmed upload. Right after the row is written, host() calls its onMediaImported hook, which src/run.ts wires (queueImportedMedia) to ctx.processMedia, a function on the plugin job context (AppHookContext) that the worker backs with its {media}-process queue (apps/worker/src/app-events.ts, via DispatchDeps.mediaProcess). Both the API and CSV paths go through the same media host, so both are covered.

• Once per image. The hook runs only when a source URL is hosted for the first time. A URL already in the import mapping (the same image on two products, or any re-run) returns the existing media id without touching the queue. As a second guard the job id is the media id (addMediaProcessJob in packages/core/src/media/queue.ts, the same scheme the upload path uses), so BullMQ ignores a repeat.
• Concurrency. The enqueue happens inside host(), so it shares the importer’s existing limits (IMAGE_CONCURRENCY downloads per product, 3 products at a time on the API path); processing itself is bounded by the worker’s MEDIA_PROCESS_CONCURRENCY (default 3) and MEDIA_PROCESS_PER_STORE (default 3, never above the concurrency). Import jobs carry bulk: true, which the worker turns into the bulk profile: AVIF effort 2 (instead of 4) and the widths encoded one at a time, so three parallel jobs stay well inside the 1.5 GB worker limit. A job without the flag also gets it when its store has 25 or more images still processing (a big upload, or a backlog queued before the flag existed). A merchant’s own single upload keeps the interactive profile. At start the worker splits the cores across the parallel jobs (tuneImagePipeline: libvips threads = cores / concurrency) and caps libvips’ cache at 100 MB.
• Dry run. Writers return before hosting any media on a dry run, and run.ts does not wire the hook for a dry run at all: nothing is downloaded or queued.
• Queue down. addMediaProcessJob throws (unlike enqueueMediaProcess), and queueImportedMedia turns that into an import issue on the resource being imported (ref is the source URL). The row still imports and the image stays processing, which bun run media:reprocess picks up.
• Video is stored ready and never queued, the same as a direct upload.

Outside the worker (processMedia absent, for example a script calling executeRun directly), imported images stay processing until the backfill runs.

Keeping the shop fast while images process

Image processing shares the server’s CPU with the storefront. On the 2-vCPU production box a backlog of a few hundred imported images held the worker at 200% CPU and pushed store pages from about 1 s to 3 to 5 s. Two things keep that in check:

• A CPU cap on the worker container (the real fix). Docker gives every container an equal share of the CPU, and nice inside a container only reorders that container’s own threads, so the cap has to be set on the service. Values and where to set them: Deployment → CPU for the worker.
• Fewer jobs at once: MEDIA_PROCESS_CONCURRENCY=1 on a 2-vCPU box (each job then uses every core it is allowed through tuneImagePipeline, so a single upload is not slower).

The “store may be slower” notice

While more than 50 images are processing, or a real (not preview) Shopify import is running, the admin shows “Processing N images in the background: your store may be a little slower until it finishes (about X min)” on Home, Content → Files and the importer (components/media/background-work-notice.tsx, polled every 30 s). The numbers come from media.backgroundWork → backgroundWork(mediaBacklog(tx), importInProgress(tx)) in core: N is the store’s rows still processing (pending is an upload the browser never confirmed, not work), X is N divided by how many finished in the last ten minutes, or six seconds an image when none did. Copy: data/background-work.ts (Hindi in data/hi/).

Backfill

bun run media:reprocess [--store <slug>]

Runs the same processImage pipeline directly (no queue) over every row that still needs it: stuck processing (a worker restart mid-job, or a Shopify-import image whose enqueue failed and was listed as an import issue), failed (retry), and ready rows with variants IS NULL (legacy media from before this feature). Batched (LIMIT, not OFFSET — a fixed row already re-fetches out of the WHERE on its own) and resumable: interrupt it any time, re-run it, it continues.

Testing

packages/core/src/media/pipeline.test.ts and loader.test.ts cover the processing function and the delivery mapping with small in-memory fixtures generated by sharp itself (including a synthetic EXIF GPS tag + a rotation tag, asserted gone/applied after processing), the decompression-bomb guard (via an injected maxInputPixels override rather than a real 40 MP fixture), never-upscale, and the animated-GIF path. No running worker or Redis needed for any of it.

packages/core/src/media/queue.test.ts checks the job id scheme and that only addMediaProcessJob throws; src/plugins/events.test.ts checks ctx.processMedia is wired only when a queue is. The importer’s test/csv-import.test.ts and test/api-import.test.ts (“media processing”) run real imports against a fake queue: one add per hosted image, none on a dry run, none again on a re-run, a failing queue recorded as issues, and video left unqueued. Those need the test Postgres (bun run --cwd packages/plugin-shopify-import test).