Last month, multiple reports surfaced about Google Photos adding a new photo editing option called Ultra HDR within the app. We now know that it will help control the brightness levels of Ultra HDR images at a pixel level. But, there was no explanation of how it helps with complex editing. Now, Google details how complex edits in Google Photos will retain the Ultra HDR quality on Pixel 8 and above.
Gain Map issues while editing an Ultra HDR image using ML-powered tools on Pixel devices
Previously complex edits beyond cropping or rotating created a new copy of the image in SDR quality. This apparently happened due to additional metadata related to Ultra HDR. The company details that the issue is specifically with the Gain Map. For those unaware, Gain Map is “a log-encoded image stored alongside the SDR image that indicates how much to brighten each SDR pixel to produce the target HDR rendition.”
Notably, editing Gain Map with tools like Magic Eraser, Magic Editor, and more in Google Photos is quite complex. That’s because Google developed these tools for SDR images and models expect SDR images as inputs. This results in SDR images as outputs. Google further provided an example of using the Magic Eraser feature to explain the issue.
Earlier, when you used the Magic Eraser feature in Google Photos to remove a bright element from an image, “it’s likely that the matching Gain Map area was bright as well.” Inpainting the missing portion of the image would also require inpainting the Gain Map to prevent the “ghosting effect.” This happened because the original Gain Map appears as an overlay on the edited SDR image. Therefore, editing Ultra HDR images in Google Photos using MIL-powered tools simply downgrades the edited image to SDR quality.
The new Gain Map reconstruction model fixes the previous issue to retain Ultra HDR quality
Google Photos now uses a new ML model to resolve this issue. The new model reconstructs the missing Gain Map of the SDR images “after HDR image editing.” It predicts a new Gain Map based on the original SDR image, edited SDR image, and the Gain Map of the original image.
Google says that the model then blends the predicted Gain Map with the original one using a mask where pixels of SDR images were earlier modified during editing. Finally, everything blends to form a uniform image while “optimizing rendition for HDR displays.”
Worth noting that Google’s Gain Map reconstruction model is “under 1MB and runs at interactive frame rates on mobile devices.” Additionally, Google has trained this model on thousands of HDR images. Moreover, these Ultra HDR edits in Google Photos are now available on Pixel 8 and newer devices.
