{"slug": "physically-controllable-relighting-of-photographs", "title": "Physically Controllable Relighting of Photographs", "summary": "Researchers Chris Careaga and Yağız Aksoy presented a self-supervised method for physically controllable relighting of in-the-wild photographs at SIGGRAPH 2025, with the work under patent pending and licensing handled through SFU Technology Licensing Office. The pipeline infers a colored mesh from monocular geometry and intrinsic-component estimates, lets users define illumination in 3D, renders it with a path-tracing engine, and passes that approximate render through a feed-forward neural renderer to produce the final photorealistic result. A differentiable rendering process reconstructs in-the-wild scene illumination, enabling self-supervised training of the neural renderer on raw image collections and bringing Blender-style explicit light control to real photographs.", "body_md": "# Physically Controllable Relighting of Photographs\n\nPatent pending\n\n**Abstract**\n\nWe present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing. We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering. Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components. This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine. We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result. We develop a differentiable rendering process to reconstruct in-the-wild scene illumination, enabling self-supervised training of our neural renderer on raw image collections. Our method represents a significant step in bringing the explicit physical control over lights available in typical 3D computer graphics tools, such as Blender, to in-the-wild relighting.\n\n**Video**\n\n**Paper**\n\n**Media Release**\n\n|   |  | \n\n**Poster**\n\n**BibTeX**\n\nauthor={Chris Careaga and Ya\\u{g}{\\i}z Aksoy},\n\ntitle={Physically Controllable Relighting of Photographs},\n\nbooktitle={Proc. SIGGRAPH},\n\nyear={2025},\n\n}\n\n**License**\n\nThe methodology presented in this work is safeguarded under intellectual property protection. For inquiries regarding licensing opportunities, kindly reach out to SFU Technology Licensing Office <tlo_dir *ατ* sfu *δøτ* ca> and Dr. Yağız Aksoy <yagiz *ατ* sfu *δøτ* ca>.\n\n**Related Publications**\n\n|   |  \t\tACM Transactions on Graphics (Proc. SIGGRAPH Asia), 2024 \t\t ***Best Paper Award Honorable Mention***  \t\tIntrinsic image decomposition aims to separate the surface reflectance and the effects from the illumination given a single photograph.  \t\tDue to the complexity of the problem, most prior works assume a single-color illumination and a Lambertian world, which limits their use in illumination-aware image editing applications.  \t\tIn this work, we separate an input image into its diffuse albedo, colorful diffuse shading, and specular residual components.  \t\tWe arrive at our result by gradually removing first the single-color illumination and then the Lambertian-world assumptions.  \t\tWe show that by dividing the problem into easier sub-problems, in-the-wild colorful diffuse shading estimation can be achieved despite the limited ground-truth datasets.  \t\tOur extended intrinsic model enables illumination-aware analysis of photographs and can be used for image editing applications such as specularity removal and per-pixel white balancing. \t    \t\t@ARTICLE{careagaColorful,author={Chris Careaga and Ya\\u{g}{\\i}z Aksoy}, title={Colorful Diffuse Intrinsic Image Decomposition in the Wild}, journal={ACM Trans. Graph.}, year={2024}, volume = {43}, number = {6}, articleno = {178}, numpages = {12}, } | \n\n|   |  \t\tACM Transactions on Graphics, 2023 \t  \t\tIntrinsic decomposition is a fundamental mid-level vision problem that plays a crucial role in various inverse rendering and computational photography pipelines.  \t\tGenerating highly accurate intrinsic decompositions is an inherently under-constrained task that requires precisely estimating continuous-valued shading and albedo.  \t\tIn this work, we achieve high-resolution intrinsic decomposition by breaking the problem into two parts.  \t\tFirst, we present a dense ordinal shading formulation using a shift- and scale-invariant loss in order to estimate ordinal shading cues without restricting the predictions to obey the intrinsic model.  \t\tWe then combine low- and high-resolution ordinal estimations using a second network to generate a shading estimate with both global coherency and local details.  \t\tWe encourage the model to learn an accurate decomposition by computing losses on the estimated shading as well as the albedo implied by the intrinsic model.  \t\tWe develop a straightforward method for generating dense pseudo ground truth using our models predictions and multi-illumination data, enabling generalization to in-the-wild imagery.  \t\tWe present exhaustive qualitative and quantitative analysis of our predicted intrinsic components against state-of-the-art methods.  \t\tFinally, we demonstrate the real-world applicability of our estimations by performing otherwise difficult editing tasks such as recoloring and relighting. \t    \t\t@ARTICLE{careagaIntrinsic,author={Chris Careaga and Ya\\u{g}{\\i}z Aksoy}, title={Intrinsic Image Decomposition via Ordinal Shading}, journal={ACM Trans. Graph.}, year={2023}, volume = {43}, number = {1}, articleno = {12}, numpages = {24}, } | \n\n|   |  \t\tSIGGRAPH Asia, 2023 \t  \t\tDespite significant advancements in network-based image harmonization techniques, there still exists a domain disparity between typical training pairs and real-world composites encountered during inference.  \t\tMost existing methods are trained to reverse global edits made on segmented image regions, which fail to accurately capture the lighting inconsistencies between the foreground and background found in composited images.  \t\tIn this work, we introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain.  \t\tFirst, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region.  \t\tA network then refines this inferred shading to generate a harmonious re-shading that aligns with the background scene.  \t\tIn order to match the color appearance of the foreground and background, we utilize ideas from prior harmonization approaches to perform parameterized image edits in the albedo domain.  \t\tTo validate the effectiveness of our approach, we present results from challenging real-world composites and conduct a user study to objectively measure the enhanced realism achieved compared to state-of-the-art harmonization methods.  \t    \t\t@INPROCEEDINGS{careagaCompositing,author={Chris Careaga and S. Mahdi H. Miangoleh and Ya\\u{g}{\\i}z Aksoy}, title={Intrinsic Harmonization for Illumination-Aware Compositing}, booktitle={Proc. SIGGRAPH Asia}, year={2023}, } |", "url": "https://wpnews.pro/news/physically-controllable-relighting-of-photographs", "canonical_source": "https://yaksoy.github.io/PhysicalRelighting/", "published_at": "2026-11-11 11:38:43+00:00", "updated_at": "2026-11-11 11:50:49.733326+00:00", "lang": "en", "topics": ["computer-vision", "generative-ai", "ai-research", "neural-networks", "artificial-intelligence"], "entities": ["Chris Careaga", "Yağız Aksoy", "SIGGRAPH 2025", "Simon Fraser University", "SFU Technology Licensing Office", "Blender"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/physically-controllable-relighting-of-photographs", "markdown": "https://wpnews.pro/news/physically-controllable-relighting-of-photographs.md", "text": "https://wpnews.pro/news/physically-controllable-relighting-of-photographs.txt", "jsonld": "https://wpnews.pro/news/physically-controllable-relighting-of-photographs.jsonld"}}