{"slug": "foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning", "title": "Foveon – Bayer to Foveon X3, learned, Mac App using deep learning", "summary": "A developer created Foveon, a Mac app that uses a modified U-Net deep learning model to transform photos from any Bayer-array camera into the distinctive look of a Sigma DP2 Merrill Foveon X3 sensor, known for its stacked-pixel color capture and microdetail. The app injects a 1D encoding of the Foveon's three-layer pixel structure into the neural network's bottleneck, trained on matched scene pairs to replicate the sensor's color and tonality.", "body_md": "# Foveon\n\nA neural sensor translator. Takes a photo from any\nBayer-array camera and renders it as if it were shot on a Sigma DP2\nMerrill — the **Foveon X3** stacked-sensor look, with\nthe colour and microdetail Foveon is famous for, on hardware you already\nown.\n\nUnder the hood: a modified *U-Net* with an extra layer injected\nbetween the encoder bottleneck and the upsampling decoder. The injected\nchannel carries a one-dimensional encoding of *three-layer pixel-stack*\nstructure — the B·G·R photodiode column that a Foveon\nsensor captures and a Bayer sensor can’t. Trained end-to-end against\nmatched Bayer → Merrill scene pairs.\n\nwith 3-layer pixel\n\ninjection at bottleneck\n\nFoveon X3 stacked-\n\nsensor look out\n\nscene pairs against\n\nSigma DP2 Merrill\n\n## What it is\n\nMost digital cameras capture colour through a **Bayer colour filter\narray**: each photosite sees only one of R, G, or B, and the other\ntwo channels are interpolated from the neighbours (demosaiced). It’s\nefficient, but it costs you. The interpolation introduces colour fringing\non sharp edges, smears fine detail, and produces the “digital”\nmicro-contrast that even high-end Bayer cameras can’t fully shake.\n\nThe **Foveon X3 sensor** — Sigma’s now-rare design\nused in the DP1, DP2, and DP3 Merrill cameras — works the way colour\nfilm does. Three photodiode layers are stacked vertically at every single\npixel position. The top layer absorbs blue, the middle layer green, the\nbottom layer red. Every pixel captures the full colour. No interpolation,\nno demosaicing artefacts, no false detail. The result is the\n“Foveon look”: extraordinary microdetail and a particular colour\nrendition — warm, dimensional, almost slide-film — that people\nbuild entire camera systems around.\n\n**Foveon** (the app) is a neural network that learns the\nmapping between the two. Feed it a JPEG or RAW from a normal Bayer camera\n(phone, mirrorless, DSLR) and it predicts what the same scene would look\nlike shot on a Foveon X3 sensor. Geometry stays the same; colour,\ntonality, and micro-detail rendering shift toward the Merrill side of\nthe training distribution.\n\n## Bayer vs Foveon — the structural problem\n\n## The architecture\n\nThe core is a standard convolutional **U-Net**: an encoder\nthat downsamples the input image into a compact feature bottleneck,\npaired with a decoder that upsamples back to full resolution, with\n*skip connections* at every level so fine spatial detail survives\nthe trip through the bottleneck.\n\nThe modification is a single new layer dropped in between the encoder’s\nfinal downsampling block and the decoder’s first upsampling block: a\n**1D pixel-stack injection layer** that concatenates a\none-dimensional encoding of how colour absorbs through silicon depth on\na real Foveon sensor — blue first, then green, then red. The\ndecoder learns to use this prior to reconstruct the kind of inter-channel\ncoupling that real X3 captures exhibit — chroma that’s registered\nwith luminance instead of interpolated against it.\n\n*order*in which colour absorbs through silicon depth on a real Foveon sensor — blue at the top, green in the middle, red at the bottom. That ordering is a 1D signal per pixel column, not a 2D spatial structure. Keeping it 1D keeps the parameter count low and forces the network to learn the inter-channel coupling rather than memorise a per-region lookup.\n\n## Training\n\nThe model is trained end-to-end against matched scene pairs: the same subject captured simultaneously by a Bayer-sensor camera and a Sigma DP2 Merrill, the two images aligned and cropped to identical framing. The Bayer image is the network input; the Merrill image is the reconstruction target.\n\n### Capture matched pairs\n\nA Bayer-sensor camera and a Sigma DP2 Merrill are tripod-mounted side-by-side. Each scene is photographed simultaneously by both. We bias capture toward the categories where Foveon rendering differs most visibly from Bayer: outdoor light, foliage, skin tones, fabric, water, metallic surfaces.\n\n### Align & crop\n\nFor each pair we run a homography-based alignment (feature-matched on luminance) so the two images share an identical pixel grid despite the different sensor sizes, focal lengths, and viewpoints. The aligned tiles are cropped to common framing and resampled to a shared resolution. Misaligned pairs are filtered out.\n\n### Develop the Merrill target\n\nDP2 Merrill RAW (.X3F) files are developed in Sigma Photo Pro with neutral parameters — zero added sharpening, native colour profile, no creative tone curve. This is the “ground truth” the network is asked to predict: what the X3 sensor saw, minimally interpreted.\n\n### Tile & augment\n\nEach aligned pair is sliced into overlapping 256×256 tiles for training (the U-Net handles arbitrary input sizes at inference, but training on fixed tiles keeps batches efficient). Augmentations: random crop, horizontal flip, small rotation, exposure jitter on both halves of the pair in lock-step.\n\n### Loss composition\n\nThree components, summed with weights tuned to keep colour and detail in balance:\n\n`L1`\n\npixel loss anchors absolute colour fidelity;\n`VGG perceptual`\n\nloss carries microdetail and texture\n(the network learns the Merrill *look*, not just the\naverage pixel value); a small `TV`\n\nsmoothness penalty\nkeeps the output from inventing sharpening artefacts in flat\nregions.\n\n### Train & validate\n\nAdamW optimiser at `lr = 2e-4`\n\n, cosine\nschedule, mixed-precision on a single 24 GB GPU. Batch size\n16 tiles. We hold out roughly 5% of pairs as a validation set\nstratified by category (skin, foliage, fabric, etc.) so the model\nisn’t accidentally memorising a few scenes. Best checkpoint\nis the one that minimises validation perceptual loss, not L1\n— perceptual loss tracks the X3 look more reliably than\npixel error.\n\n## What it produces\n\nA schematic comparison: what the same uniform skin patch looks like coming out of a Bayer demosaic pipeline (left) versus what the network predicts the Merrill would have rendered (right). The Bayer side has slight chroma drift between cells; the Merrill side is uniform.\n\n## The app\n\nFoveon ships as a native macOS app with a clean three-pane workflow: pick a photo on the left, watch the preview update on the right. Three sliders shape the conversion:\n\n`0`\n\nthe output sits closer to the original Bayer\ncolours; at `1.0`\n\n(default) the full Merrill warmth\nand saturation come through.\n`0.18`\n\nmatches Sigma Photo Pro’s neutral\ntone; push it for a more cinematic look, drop it for a flatter\nstarting point you can grade externally.\nDrag either preview panel to pan; both panels move together so the\nsame crop is always in view on both sides. `Reset`\n\nsnaps\nthe view back to fit, `Save Foveon Photo…`\n\nwrites\nthe result.\n\n## Install & use\n\nFoveon runs natively on Apple Silicon. A 24-megapixel input converts in well under a second on an M2; the model uses MPS for inference and stays within ~3 GB of unified memory.", "url": "https://wpnews.pro/news/foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning", "canonical_source": "https://code.intellios.ai/photo/", "published_at": "2026-06-28 04:27:03+00:00", "updated_at": "2026-06-28 05:04:53.482064+00:00", "lang": "en", "topics": ["machine-learning", "computer-vision", "neural-networks", "ai-products", "ai-tools"], "entities": ["Foveon", "Sigma DP2 Merrill", "Foveon X3", "Bayer", "U-Net", "Mac"], "alternates": {"html": "https://wpnews.pro/news/foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning", "markdown": "https://wpnews.pro/news/foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning.md", "text": "https://wpnews.pro/news/foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning.txt", "jsonld": "https://wpnews.pro/news/foveon-bayer-to-foveon-x3-learned-mac-app-using-deep-learning.jsonld"}}