Hate mosaics? Google says you can “fix” it

The sci-fi movie "Blade Runner" released in 1982 envisioned a 2019 full of cyberpunk technology: the sky is full of flying cars, and humans can use a large-screen machine (Esper) to accomplish many incredible tasks.

When the protagonist Rick Deckard traced the whereabouts of the clones, he used Esper to unearth some clues that were ignored by the naked eye.

Rick stuffed a suspicious photo found at the scene into Esper, and magnified a corner of the photo non-destructively over and over again, and finally found the android Jura in the reflection of the mirror.

The imagination of sci-fi works is indeed limited by the technology of the time. PCs just became popular in 1982. "Blade Runner" did not imagine that the emergence of the Internet had changed the way of human life, nor could it imagine that simulation technology would have One day was replaced by digital technology.

However, its imagination of the lossless magnification image technology is very advanced, which is a difficult problem to be solved so far.

You may also have encountered this situation: after the party, friends took a group photo together, and then zoomed in to see the old classmate’s face or the drink brand that evening when I went back, but what I saw was blurred. Mosaic.

This is because when we enlarge the photo to a certain extent, the local resolution is already very low, and all we see are images composed of pixels.

Is it possible for us to extract additional image information from trivial "mosaics" like the fantasy 30 years ago?

A recent blog by Google’s AI team mentioned a new image algorithm, which is very close to the vision of "Blade Runner".

Incredible resolution increase

▲ 64 x 64 Pikachu

How big is a 64 X 64 pixel photo? Using the 12-megapixel photo taken by the iPhone as a template, it is only about one-third of its size. When displayed on a high-definition screen, you will only see a full "mosaic."

▲ Super high resolution photos

In the digital age, every image we see on the screen is composed of densely packed pixels. The more pixels per unit area make up the image, the higher the resolution and the clearer the image.

Google's AI researchers are thinking, is it possible to extract enough picture information from low resolution, use machine learning to restore the original picture as much as possible, increase the resolution of the picture, and get a clear picture?

▲ Picture from: Google

In its recently published blog, Google showed its latest research results, which is very shocking from the effect-through two different algorithms, 64 X 64 pixel photos can be restored to 1024 X 1024 pixel resolution, and details The effect is very realistic.

It should be pointed out that the photos restored by Google through machine learning algorithms are bound to have some deviations from the original photos, but when we cannot obtain the original scene (such as old photos in the past), a "restore" that is as close to the real as possible The photos are really valuable.

▲ Picture from: Google

According to Google, repairing a "mosaic" photo consists of two processes-"destroy" and "reorganization".

First, in order to dig out the graphic details of the "mosaic" pixel block as much as possible, Google researchers will first process the test samples with the Gaussian noise algorithm to obtain a "snowflake map" composed entirely of noise, which looks a bit like the previous analog TV. The picture of the signal.

▲ The third line is Google’s repair algorithm, and the fourth line is the original picture reference. Picture from: Google

Then, the researchers use neural network algorithms to reverse the destruction process of Gaussian noise, and synthesize new image data through the reverse restoration process, and reduce the noise as much as possible from the pure noise image to obtain a clear picture.

▲ Picture from: Google

The principle of image restoration is not complicated, but the algorithm involved is not simple. In order to restore the "one-to-one restoration" high-definition large image, Google researchers proposed the super-resolution algorithm SR3 and the cascaded diffusion model CDM. Improve the accuracy of restoration through large-scale picture comparison learning.

It is worth mentioning that although we have always used "mosaic" to refer to low-resolution large-pixel low-resolution pictures, this is essentially different from the real coded photos.

▲ Picture from: Google

The reason why Google's restoration algorithm can make low-definition pictures clear is essentially based on the correct image information contained in the picture itself, through comparison and matching of countless images in a huge database, and finally a simulated approximate pixel filling.

When the photo is smeared with mosaic, the image information contained in the photo will change.

In simple terms, the mosaic algorithm is to randomly select the color of the pixels in an area at a fixed interval, and then get the average value of all the pixels in the area, and fill it in the square with a new color.

After coding, the original pixel information is lost, and only the error information calculated randomly is obtained. At this time, let the machine learning restore it, just like asking it to make a correct answer to a question that is completely wrong. It is impossible to answer.

So if someone wants to use the Google algorithm to mine some of the erased private information, they can dispel this idea.

You have entered the future

▲ Picture from: Google

Google's HD repair algorithm is likely to be finally applied to Google image processing software such as Google Photos, Snapseed, etc. It will become one of our photo editing tools like HDR, viewing angle correction and other algorithms.

Going back to the "Blade Runner" movie, Esper is actually a very interesting machine, it is a bit of a fusion of analog technology and digital technology.

On the one hand, it is very advanced, people can control it with voice, and achieve lossless magnification; on the other hand, it is very old-fashioned, with a clear large screen but still CRT structure, the process of importing photos is to scan from physical photos.

According to the effect of the movie, Esper may be a certain coordinate of the fixed-point photo, and then magnify the photo through a precise lens structure (microscope). Looking at it now, the idea of ​​lossless amplification is very advanced, but the simulation technology is obviously not a realistic future.

For modern people, the mobile phones and computers in their hands are everyone's "Esper".

▲ Picture from: Adobe

Now that photos have already completed the evolution of a fully digital workflow, it is not difficult to enlarge photos with digital technology. In other words, you have actually entered the "future" described in "Blade Runner".

▲ Picture from: Adobe

Image super-resolution has always been a hot research topic in the field of computer vision. Companies such as Adobe are developing related image processing technologies, which have been applied in graphics processing software such as Photoshop and Lightroom.

Take Photoshop as an example. After importing the RAW format picture, you can select the "Super Resolution" function of the "Enhanced" function. The software will refer to similar content to enrich the texture of the picture and enlarge the resolution of the picture by 4 times. The whole process It takes about a minute.

▲ Picture from: Adobe

It can be seen from the photos before and after the contrast enhancement that the sharpness of the photos has been significantly improved after the resolution is increased, and some fuzzy and unrecognizable details have also become clear.

Adobe mentioned in a technical blog published in March this year that the super-resolution algorithm it uses has also undergone a lot of machine learning training and is constantly improving and improving.

▲ Picture from: Adobe

Does it make sense to explode the resolution of pictures? Maybe after taking a photo, you won't enlarge it to delve into every detail, but when you need to print this photo, the resolution of the photo's imaging directly determines the maximum size of the print.

This is especially important for photographers. Sometimes when shooting scenery with a wide-angle lens, an eagle flies over the sky. The feather details of the eagle can’t be captured with a wide-angle lens. It is possible to get the picture you want.

▲ First crop, and then zoom in with super pixels to get 10 million pixel photos. Image from: Adobe

Adobe used a 2.5-megapixel photo in its blog as an example, and used the super-resolution function to enlarge it to 10 million pixels so that it can be printed into a "decent" photo. Adobe describes this process as "digital zoom."

Comparing the algorithms of Adobe and Google, there are some differences between the two. Adobe requires RAW format photos that retain a large amount of original information for calculations, while Google's algorithm can restore photos based on some very rough information.

▲ Adobe's algorithm continues to progress through a lot of machine learning. Picture from: Adobe

At present, both algorithms are still not fully mature, and a lot of machine learning is needed to improve the accuracy of calculation and restoration.

But what is certain is that super-resolution technology will become one of the most popular imaging technologies in the near future, helping people get rid of the limitations of telephoto lenses and other equipment, and record every detail and moment of life. In order to see a clearer world, we have not stopped exploring.

Higher, higher.

#Welcome to follow Aifaner's official WeChat account: Aifaner (WeChat ID: ifanr), more exciting content will be provided to you as soon as possible.

Ai Faner | Original link · View comments · Sina Weibo