Meta launches Vincent 3D model “Blockbuster”, generating 3D materials in one second

Meta Company officially released a research paper, introducing a Vincent 3D model system called Meta 3D Gen, which can generate higher-quality 3D assets from text in less than a minute.

Technical Highlights:

  • High-speed generation: 3DGen can generate preliminary 3D assets in just 30 seconds, and the subsequent texture refinement stage only takes 20 seconds, significantly improving the efficiency of 3D asset generation.

▲ 3D models generated by different prompt words

  • Physically Based Rendering (PBR) support: 3DGen supports PBR, which is critical for reproducing the lighting effects of 3D assets in real-world applications.

  • High fidelity: Evaluated by professional 3D artists, the 3D models generated by 3DGen surpass existing commercial and non-commercial methods in terms of fidelity and visual quality under complex text prompts.

▲ Comparison of details of content generated by this technology and other technologies

Generate realistic 3D models in just two steps

The paper introduces that Meta 3D AssetGen adopts a two-stage design to generate 3D models compared to traditional 3D object generation methods:

in particular:

The first stage: text to image stage (blue part in the picture below): generate 3D mesh and texture based on text prompts, predict a 6-channel image that depicts 4 views of the object with shadow and albedo colors .

The second stage: the image to 3D stage consists of two steps.
First, the 3D reconstructor (called MetaILRM) outputs a three-dimensional SDF field, which is converted into a mesh with a textured PBR material (the orange part in the figure below).

These materials are then further enhanced with a texture refiner to restore detail that may have been lost from the input view (green in the image below), thereby improving the visual quality and detail of the model.

Multiple indices to quantify materials and light

In terms of technical implementation, Meta 3D AssetGen uses VolSDF formulas with different hyperparameters to render SDF values ​​and obtain the opacity of 3D points.

During training, the model is optimized by minimizing multi-view rendering losses, but since physically accurate rendering is very expensive, we bypass the complex rendering equation by directly using the original PBR fields to supervise the predicted counterparts.

▲ Alpacas with different materials and styles generated by Meta’s new technology

This kind of PBR refers to "Physically-Based Rendering", which is physically based rendering.

It simulates the physical behavior of light on the surface of an object, taking into account the impact of lighting, material properties and environmental factors on the appearance of the object. It can calculate the reflection, scattering and scattering of light based on different characteristics of the object surface, such as roughness, metallic feel, etc. absorb. To achieve more realistic and accurate rendering effects.

In Meta 3D AssetGen, PBR materials are used to enhance the realism of 3D models. Specifically, the PBR material includes the following key properties:

  • Albedo: refers to the color and brightness of the surface of an object, which determines the appearance of the object under different lighting conditions.
  • Metalness: Indicates the degree of metal on the surface of an object. Objects with a high metallic feel will have a more obvious metallic luster.
  • Roughness (Roughness): describes the smoothness of the surface of an object and affects the range of light scattering on the surface. The higher the roughness, the wider the light scattering and the softer the highlight.

▲ Model generated using the prompt word "A cat made of MATERIAL"

PBR materials actually integrate and represent a major advancement in AI-generated 3D content. It is considered to be possible to bridge the long-standing problem between AI-generated content and professional 3D workflows, seamlessly integrating AI-created materials into existing workflows, thus It is possible to accelerate the creation of virtual environments and digital twins across industries.

▲ Render the appearance textures of dragon eggs and bears through text prompts

The researchers also introduced a Meta 3D TextureGen technology consisting of a continuous network, which combines text generation models with 3D semantic conditions in 2D space to fuse them into a complete and high-resolution UV texture map in a short time. Generate high-quality textures for complex geometries.

▲ Comparison of 3D texture generation technologies: Meta’s new method on the far left shows more vivid colors and more complex details

In the experimental part, the researchers used a dataset of 140,000 meshes of diverse semantic categories created by 3D artists for training. Extensive user research was also conducted comparing Meta 3D AssetGen to other PBR-enabled text-to-3D methods in the industry, showing that Meta 3D AssetGen offers significant advantages in terms of visual quality and material control:

AssetGen achieves 17% improvement in chamfer distance, 40% improvement in LPIPS, and is highly user-friendly compared to best-in-class industry competitors at comparable speeds, including those with PBR support Out of 72%.

Chris McKay, founder and editor-in-chief of Maginative, commented:

The potential applications of this technology are vast. Game developers can use 3D Gen to quickly prototype environments and characters, significantly speeding up the development process. Architectural visualization companies can generate detailed 3D models of buildings and interiors from text descriptions, streamlining the design process. In the realm of virtual and augmented reality, 3D Gen enables the rapid creation of immersive environments and objects, potentially accelerating the development of Metaverse applications.

Obviously, Meta's new technology provides the possibility for realistic presentation of 3D models under different lighting environments. It has great potential in the fields of 3D graphics, animation, games and AR/VR, and will have great impact on games, film and television and even product development. help.

This may be the first step towards another level of world modeling.

# Welcome to follow the official WeChat public account of Aifaner: Aifaner (WeChat ID: ifanr). More exciting content will be provided to you as soon as possible.

Ai Faner | Original link · View comments · Sina Weibo