The self-developed surging chip on the Mi MIX folding screen is the future of mobile photography?

"Processor chips are the commanding heights of technology in the mobile phone industry, and they are at the top of the pyramid. If you want to become a great company in this industry, you must have autonomy in core technologies so that the company can go far."

This was Lei Jun's speech at the press conference when Xiaomi announced its entry into the chip field and released the first mass-produced chip "Surge S1".

However, after the surging S1, the S2 chip was difficult to produce, and Xiaomi's self-developed surging chip gradually became low-key and disappeared from public view. It was not until August last year that Lei Jun revealed that the surging chip was still being developed. I believe that at this time, Xiaomi MIX FOLD has already completed trial production and started mass production, as does the built-in "Surge C1" chip.

Compared with the Surging S1 and the dystocia S2, the Surging C1 is just an ISP. Its functions are much simpler than SoC, and the chip design is not so complicated. It is more suitable for the “surging” of a fresh start.

At the same time, the self-developed Surging C1 is an independent ISP (image processor) inside MIX FOLD, and it can be regarded as the only product equipped with an independent ISP in the current flagship machine.

In the product line planning, MIX FOLD is not a mass-oriented product like the Mi 11 series. Therefore, the surging C1 chip, or the independent ISP in the mobile phone, has not yet moved to the masses. It is still a small-scale attempt.

This does not exclude the emergence of independent ISPs on the next generation (in the second half of the year?) Xiaomi flagships, or may trigger a "butterfly effect", becoming a standard feature of domestic flagships and becoming a trend in mobile imaging. Is this possible?

What role does ISP play in imaging?

In a nutshell, ISP is responsible for processing the images collected by the sensor.

Regardless of smartphones, mirrorless cameras, and DSLRs, ISPs play almost the same roles, and the processes are similar. They all perform Bayer transformation, de-mosaic, noise reduction, and sharpening operations on the "raw data" collected by the image sensor. But on the smartphone side, the ISP will also include the process of "computational photography", with different algorithms and different adjustments, and ultimately produce different photos.

▲ Sony BIONZ X image processor. Picture from: SONY

The configuration of SoC with independent ISP is actually similar to a professional mirrorless camera. Take Sony’s BIONZ X image processor as an example (α7 series royal). It packs SoC and ISP chips and is not packaged together. The advantage is that Sony can increase the number of ISP chips according to the performance requirements of CMOS (α7RIII BIONZ). X is equipped with dual ISP), the disadvantage is that the degree of integration is not high.

Before MIX FOLD, the ISPs of most smartphones were packaged in SoCs, and every year they would be upgraded with SoC upgrade iterations.

For example, now Qualcomm’s flagship chip, Snapdragon 888, has three built-in ISP chips, with a pixel processing capacity of 2.7 billion pixels per second, and can capture three 28-megapixel still photos or three 4K HDR videos at the same time. Compared with the previous generation of Snapdragon 865, there is a 35% improvement.

MediaTek in the other camp has also greatly improved the performance of the built-in ISP in the latest Dimensity 1200 SoC. From the 80 million pixels of the previous generation Dimensity 1000, it has been upgraded to the level of supporting 200 million pixels in one go, and it has also added a Series of new functions based on AI algorithms.

The processing power of ISP must not only match the performance of the sensor, but also be associated with the algorithm of "computational photography". It will directly determine the imaging tendency of a smartphone, whether the color is rich or not, how much highlights are retained, and the choice of dark parts. How much, and how much gradient information is retained in the HDR algorithm, etc.

It can be seen from this that when upstream chip manufacturers are designing next-generation SoCs, the improvement of ISP performance will be a major direction, which is closely related to the importance of mobile phone imaging.

From the perspective of the ability to control high-pixel sensors alone, the current mainstream SoC's built-in ISP performance is still rich, and it is enough to process the current highest 100 million pixel photos in a single time.

In addition, the ultra-high pixel sensor we are talking about now runs in the "four-in-one" and "nine-in-one" modes. When converted, a photo is actually 20 million pixels. Even if you include a different calculation process, SoC's ISP will still not drag down "computational photography".

Why carry an independent ISP?

Since the ISP in the SoC has mature technology, mature algorithms, and excellent performance, why should it be equipped with an independent ISP chip?

For Xiaomi, this "small chip" can be regarded as a new start. Since it is not enough to launch an independent SoC, it is better to put aside the burden, start from the "small", start from the ISP, and slowly accumulate experience. It is a good choice. Boldly say that many "small chips" like ISP may be used in subsequent car building, which is also considered an advance layout.

As mentioned above, ISP is not only for "computational photography", but the main function is to process the raw data collected by the sensor. This processing process is no different from that of a traditional camera. At the press conference, for the function of the surging C1 self-developed ISP, the advantages that Xiaomi mentioned mostly focused on 3A processing, that is, focusing (AF), white balance algorithm (AWB), and exposure strategy (AE). The handling is better.

▲ ISP processing flow. Picture from: CSDN@brandonyue

In fact, 3A processing is just the "tip of the iceberg" for the data collected by the sensors. Strictly speaking, ISP will process more than a dozen items of raw data, and 3A processing can also be divided into "broad sense" and "narrow sense".

In a narrow sense, it is what Xiaomi said at the press conference. In a broad sense, AF can be extended to auto focus, auto focus motor control, and auto focus point selection. AE involves more, auto ISO, auto ND Filters, automatic shutters, etc., AWB will also involve various hue shifts.

In general, the effect of the final imaging processing is basically caused by the different strategies and trade-offs of each step of the ISP.

Equipped with an independent ISP is basically to improve the final imaging effect. In recent years, Xiaomi has continued to exert efforts on images and has accumulated in the strategy of ISP processing images. Combined with the independent ISP Surging C1, I still want to further understand the ISP's process and strategy in processing raw data, and it can also be regarded as paving the way for subsequent Xiaomi mobile phones to continue to improve the image power.

Will independent ISPs become a trend of "moving images" in the future?

The Mi 11 Ultra uses an outsole that is close to one inch. If there is no "optical breakthrough" in the follow-up, this is almost a physical limit. And the current number of CMOS pixels is also in the state of "arms race", we should soon see smart phones enter the threshold of 200 million pixels.

The continuous enhancement of image power is already a major trend in the smart phone industry, and it has also brought about "homogenization." In order to create different image experiences, many manufacturers will invest resources to develop different technologies, and some even cooperate with traditional image manufacturers to improve their image style.

But for moving images, as long as the iron law of physical optics is still there, it is the traditional aspects of sensor quality, ISP processing capacity, and lens quality that affect the final image. The others are just icing on the cake.

Ever since, mobile phone manufacturers will customize CMOS (sensor) in order to obtain a better imaging foundation, to refine the algorithm to make the final imaging more natural, and work hard on lens collocation to adapt to more scenes.

As for independent ISPs, they want to become a major trend similar to the above-mentioned situation. Looking at it now, it is too early. To be precise, it should be a future "general direction."

On the one hand, the ISP built into the current SoC will not be a hindrance, nor is it the shortest wooden board on the moving image barrel. On the other hand, with an independent ISP, there is a not low threshold. At present, upstream chip manufacturers such as Qualcomm and MediaTek have not launched independent ISPs for smartphones. If they want to carry independent ISPs, they may have to rely on self-production. There are not many mobile phone manufacturers that can produce their own chips.

Furthermore, the current independent ISPs have not yet shown their due advantages. After experiencing MIX FOLD Zhijian's original words, MIX FOLD cameras are no different. In other words, whether it is a "liquid lens" or an independent ISP, it is more like laying out future products, accumulating small flows into a river.

#Welcome to follow Aifaner's official WeChat account: Aifaner (WeChat ID: ifanr), more exciting content will be provided to you as soon as possible.

Ai Faner | Original link · View comments · Sina Weibo