Computational RAW on smartphones: the best of both worlds

What is RAW anyway?

The English word “raw” means “raw”. And that’s exactly what RAW is: The format contains the raw data from the camera sensor, which has usually just passed through the analog-to-digital converter. A lot of work is still required before these become a JPEG photo – according to the comparison that is often used: the RAW image corresponds to the analogue negative, the JPEG image is the fully developed photo.

The first step in turning a RAW into a JPEG in the digital photo lab is demosaicing. The image processing algorithm creates RGB pixels from the only red, green, blue and possibly white pixels and pours them into a color space. The software then optimizes the image with noise reduction, resharpening, lens correction, white balance, etc. – and saves the result as a JPEG.

This conversion process from RAW to JPEG is destructive and irreversible, i.e. information is irrevocably lost. Whether the image result really corresponds to the visual impression of the photographer or the desired result is a matter of luck – and depends on the quality of the image processing algorithms.

With the RAW format, the photographer is in control.

Why doesn’t RAW matter on smartphones?

While the vast majority of photographers shoot RAW or at least RAW+JPEG with dedicated cameras, hardly any smartphone users use the RAW option typically buried in Pro mode. But why not?

In addition to the hidden positioning, the biggest reason is probably because it doesn’t make any sense in many cases. While current smartphones in AI-boosted JPEG mode like to condense a dozen photos into a single image, in RAW mode – mostly – the raw data of a single shot end up in memory. With the help of intelligent algorithms, smartphones nowadays get much more quality out of five, ten or fifteen photos welded together than a photographer out of a single RAW image.

So for a long time it was a lot of effort for little reward to deal with the RAW files from smartphones. This could change now.

RAW comeback 2022 as Computational RAW?

Slowly but surely, Computational RAW is catching on. And the potential is huge, at least in theory, because Computational RAW welds multiple photos together into a single RAW shot. In this way, the advantages of both worlds are combined in one format.

The idea is not new. The Google Pixel 3 from 2018 was already capable of computational RAW and packs the data from 15 individual images into a single RAW file. As an order of magnitude: In 15 individual images, the 1/2.55 inch image sensor from the Google Pixel 3 (approx. 23.5 mm²) as much light as an APS-C format sensor (approx. 350 mm²), as found in most mid-range DSLMs.

Apple got more attention in 2020 for the ProRAW format in the iPhone 12 Pro (Max), which also condenses an entire batch of images into a single shot. In November, Samsung followed with Expert RAW for the Galaxy S21 Ultra. Oppo announced the MariSilicon X image processor in December, which apparently also supports Computational RAW and will probably be in the Find X4 Pro. And the OnePlus 10 Pro recently released in China also comes with a corresponding mode with RAW Plus.

There are now many smartphone apps for image processing for the RAW format – both from third-party providers and from the manufacturers themselves. And the memory hunger of raw data photos should no longer be a major problem in 2022.

So what’s missing?

On the one hand, it is probably the courage on the part of the manufacturer to place the feature prominently in the camera app and to offer a simple workflow for editing. Once that is in place, users will follow once they experience the flexibility of RAW formats for themselves.

What’s Next for Computational RAW?

The next big challenge for computational photography—and by extension, computational RAW—is combining photos from different cameras. Because the sensors in smartphones can no longer be enlarged much further without making the housing significantly thicker and/or impeding the image processing algorithms through optical effects such as selective depth of field.

The only way to increase the photosensitive area is to split it across multiple sensors. The Nokia 9 PureView and the Light L16 have tried this in the past few years, but failed due to a lack of computing power in the end devices. The problem will resolve itself in the months and years to come, so beware, trypophobics.

And you?

How does it look in the NextPit community: Do you already take photos in RAW format – and with a “real” camera or even with a smartphone? And if not: Will you give the RAW format in smartphones a chance in the near future?