Why not use a camera for computational photography that greatly upgrades the iPhone’s camera capabilities?

At the launch of the iPhone 11 series, Apple’s senior vice president Philip W. Schiller introduced the concept of computational photography for the first time when introducing the imaging system of the iPhone 11 Pro series. This concept was also the first Known to the public.

In fact, the concept of computational photography is not new. It first appeared in a public paper in 1994, and it was determined that in-camera synthesis of HDR, panoramic photos, and simulated bokeh belong to the category of computational photography. But at that time, the mainstream carrier of photos was still film, and digital cameras were just starting, and there were no cameras on mobile phones.

Decades later, the carrier of image recording has changed from film to digital, mobile phones have cameras, and computational photography has gradually become a major trend.

However, this trend has little to do with cameras. Camera manufacturers are still gradually improving the pixel, continuous shooting speed and video capabilities. They seem to be blind to computational photography. The photos taken (straight out) are still very mediocre and are gradually being used by smartphones. "Beyond".

On the contrary, the computing power of smart phone chips is getting stronger and stronger, AI, algorithms, and machine learning are involved in a wider range. There are more and more methods of image interpretation, and finally a series of "algorithm" processed photos It's also getting better.

Nowadays, many people are more willing to use mobile phones to record and share, and cameras are becoming less common. This is also reflected in the market performance of the two. The smartphone market has grown rapidly, and the camera market has shrunk year after year. Even DC (card camera ) Gradually disappeared.

At this time, some people may ask, since the photos taken by smartphones have such a good look and feel, why don't traditional camera manufacturers follow the trend of computational photography and consider improving the straightforward look and feel of photos?

Is the camera not having enough computing power to count?

Let's start with the "core" of this issue.

The core of a mobile phone is SoC, which integrates CPU, GPU, ISP, NPU, and baseband. It allows you to make calls, take pictures, watch videos, play games, and surf the Internet. It also directly determines the performance of the mobile phone.

The core component of the camera is the image sensor (CMOS), which is similar to a mobile phone except for the component area, for imaging and light sensitivity. In addition, the central processing chip that controls the entire camera system is called an image processor.

Take Sony’s BIONZ X image processor as an example (α7 series royal). It includes SoC and ISP chips. It does not integrate ISP in SoC. The advantage is that Sony can increase the number of ISP chips according to the performance requirements of CMOS. (The BIONZ X of α7RIII is equipped with dual ISPs). The disadvantage is that the degree of integration is not as high as that of mobile phones.

The role of the SoC in BIONZ X is similar to that of a mobile phone. The performance requirements for controlling the control interface and camera functions are not high. Perform Bayer transformation, demosaicing, noise reduction, sharpening and other operations on the "data" collected by the image sensor, mostly relying on ISP, and finally convert the data collected by CMOS into the real-time view of the camera. In this process, the camera's ISP does not involve the calculation process, but treats the photos as products on the assembly line for unified processing.

With the continuous improvement of the number of pixels, continuous shooting speed and video performance of the current camera, the image processor of the camera has a high demand for the speed and throughput of image processing, and the single data volume is very large, without involving "calculation". , The processing power of the camera image processor far exceeds the processing power of the current smart phone ISP.

But when it comes to computational photography, or AI capabilities, there is something different. The imaging process of a smartphone is somewhat similar to that of a camera, but before the final image is presented, it also requires ISP and DSP calculations, real-time adjustment and optimization, especially after the multi-camera system becomes the mainstream, the calculation data volume of the mobile phone doubles.

After the iPhone 11 Pro series launched the multi-camera system, behind the smooth and seamless switching of the multi-camera system is the huge data processing capabilities of the two newly added machine learning accelerators in A13 Bionic, reaching one trillion times per second. High-frequency and efficient data processing capabilities can be regarded as eating up the huge amount of data generated by three cameras.

The image processor of the camera mostly preprocesses the original data, and there is almost no calculation process, while the mobile phone SoC includes data acquisition preprocessing and subsequent calculation processes, and the two focus on different directions.

Different groups, the result of market segmentation

Mobile computing photography has developed rapidly. The root cause is that the size of the image sensor (CMOS) of the mobile phone is too small. With the current technology, if you want to physically surpass or approach the camera, you can only optimize it through algorithms to make it look straightforward, for example, automatic HDR, super night scene, simulated large aperture, magic sky change and other functions.

But the interpretation of these algorithms is still difficult to achieve "personalized" intervention, such as how much filters are added, and how much HDR highlights and shadows are retained. However, for mobile phones facing the masses, as far as possible, let most people take good photos, which is more in line with the market positioning and crowd positioning of mobile phones.

Since the invention of the camera, the camera has an absolute "tool" attribute. In order to be efficient, the appearance, control, function, etc. will all compromise to efficiency. Facing a niche professional group, it will naturally be more in line with their needs. Cameras will record color depth, color, light and other information as much as possible, so that users can make a wider range of post-adjustments to determine whether it is good or not. Not in their needs.

For most people who don’t have a foundation in photography, getting a good-looking photo at hand is far more important than getting an informative photo. For professional camera manufacturers, increasing the color depth of RAW recording is more in line with the market positioning than improving the straight-out effect of JPG.

However, things are not so absolute, and cameras are also trying to change. Fuji has always been committed to the straight-out effect of the camera, introducing "film simulation", through different algorithms, to make the photos more tasteful and look better. However, this process does not go through scene calculations, but requires users to choose by themselves. This is similar to some film simulation apps on mobile phones and does not involve so-called "computational photography."

After AI, is the general direction of the camera?

In the field of photography, post-processing is an indispensable step. On the one hand, post-processing software can make full use of the rich information recorded in the RAW format, and on the other hand, it can also use the high performance and computing power of the PC to quickly process photos.

Unlike camera manufacturers, almost mainstream professional post-production software has begun to work on AI, emphasizing the processing capabilities of AI.

Adobe's Photoshop has added an automatic recognition function to the operations such as cutout, repair, and dermabrasion in recent versions of the update, making the operation more and more brainless and the effect more and more accurate. The Pixelmator Pro retouching software on the Mac platform began to use Apple's Core ML machine learning to recognize images as early as 2018, so as to perform color adjustment, matting, selection, and even compression output. ML machine learning was used. engine.

As mentioned in the previous article, due to the limitation of chip AI computing power and the problem of niche market, camera manufacturers have hardly exerted their efforts in computational photography. However, the explosion of later software in AI can also be regarded as making up for the shortcomings of cameras in computational photography.

Even if the AI ​​of the later software is included, the cameras still have not got rid of the traditional process. The cameras record and the software processes. This process is still cumbersome for the public. For professional photographers, the intervention of later software AI can indeed reduce the workload and make the original complicated cutout operations a lot easier, but it still cannot reverse the photo processing (creation) process of the traditional photography industry, which is completely different from the mobile phone. different.

According to CIPA's data, the camera market is gradually shrinking. On the contrary, the mobile phone market continues to grow. The trend of "computational photography" on smartphones will not change the direction of cameras becoming more professional, nor will it reverse the gradual shrinking of the camera market.

In the face of menacing mobile phones, cameras can only develop in a more professional direction and continue to subdivide the market upwards. In recent years, the full-frame 40 million and 60 million high pixels, medium-format over 100 million pixels, and micro-single Video capabilities continue to approach professional camcorders, which are all products of the camera segment.

The increasing specialization of cameras means the need for better-performing image sensors (CMOS), but "computational photography" relies on a separate machine learning module. As we all know, the high cost and high risk of chip development make it difficult for camera manufacturers. Take both into consideration. Computational photography and development of specialization are two different paths. At the same time, for features such as "computational photography" and "AI intervention" that are of little use to professional users, camera manufacturers are likely to be strategically abandoned temporarily due to balancing research and development costs.

At this stage or in the foreseeable future, it is even more difficult to want camera manufacturers to embrace "computational photography" with high risk, high investment, and slow results, not to mention that there are still many professional post-production software using AI to retouch photos. At the end.

#Welcome to follow Aifaner's official WeChat account: Aifaner (WeChat ID: ifanr), more exciting content will be provided to you as soon as possible

Ai Faner | Original link · View comments · Sina Weibo