Неявные нейронные представления для 3D генерации и 3D реконструкции с нескольких точек обзора тема диссертации и автореферата по ВАК РФ 00.00.00, кандидат наук Игнатьев Савва Викторович

• Игнатьев Савва Викторович
• кандидат наук
• 2025

Оглавление диссертации кандидат наук Игнатьев Савва Викторович

Abbreviations

Introduction

1 Related work

1.1 Implicit Neural Representations

1.2 INRs for Neural Rendering

1.3 Generative Adversarial Networks

1.4 Diffusion models

2 Preliminaries

3 Adversarial Generation of Continuous Images

3.1 Introduction

3.2 Related Work

3.3 Method

3.3.1 Model overview

3.3.2 Factorized Multiplicative Modulation (FMM)

3.3.3 Multi-scale INRs

3.4 Experiments

3.4.1 Standard GAN training

3.4.2 Exploring the properties

3.5 Additional implementation details

3.6 Experiments details

3.6.1 Zooming out

3.6.2 Keypoints prediction

3.6.3 Additional samples

3.7 Positional encoding of coordinates

3.8 Geometric prior

3.9 FMM as a generalizaton of the common weight modulation schemes

3.10 Conclusions to chapter

4 Deforming Implicit Neural Representation Generative Adversarial Network

for Unsupervised Appearence Editing

4.1 Introduction

4.2 Related Work

4.3 Method

4.3.1 Warping-based generator

4.3.2 Implicit neural representations as a natural deformation prior

4.3.3 Reversible Laplacian Pyramid blurring

4.3.4 Guided learning via self-distillation

4.3.5 Deformation field regularization

4.4 Experiments

4.5 Conclusions to chapter

5 A3D: Does Diffusion Dream About 3D Alignment?

5.1 Introduction

5.2 Related work

5.2.1 Text-driven 3D asset generation

5.2.2 Text-driven 3D asset editing

5.2.3 Latent space regularization

5.3 Preliminaries

5.3.1 Neural radiance fields

5.3.2 Score distillation sampling

5.4 Method

5.4.1 Generation of multiple aligned 3D objects

5.4.2 Hybridization: combining the aligned 3D objects

5.4.3 Structure-preserving transformation of 3D models

5.5 Experiments

5.5.1 Generation of multiple aligned 3D objects

5.5.2 Hybridization: combining the aligned 3D objects

5.5.3 Structure-preserving transformation of 3D models

5.6 Ablation

5.7 Conclusions to chapter

6 NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion

6.1 Introduction

6.2 Related work

6.3 Method

6.3.1 NeuS

6.3.2 Score Distillation Sampling

6.3.3 SDS Surface Shape Guidance

6.3.4 Frozen SDS

6.3.5 Multi-view SDS

6.4 Experiments

6.4.1 Evaluation setup

6.4.2 Quantitative and Qualitative Results

6.4.3 Ablation study

6.5 Conclusions to chapter

7 Sphere-Guided Training of Neural Implicit Surfaces

7.1 Introduction

7.2 Related work

7.3 Approach

7.3.1 Rendering

7.3.2 Sphere-guided volume rendering

7.3.3 Optimization of sphere cloud

7.4 Experimental results

7.4.1 Base methods

7.4.2 Training

7.4.3 Realistic Synthetic 360 evaluation

7.4.4 DTU and BlendedMVS evaluation

7.4.5 Ablation

7.4.6 Limitations

7.5 Conclusions to chapter

References

List of Tables

List of Figures

Appendix A. Russian translation of the dissertation

Введение диссертации (часть автореферата) на тему «Неявные нейронные представления для 3D генерации и 3D реконструкции с нескольких точек обзора»

Introduction Background and motivation

In recent years, the problems of 3D generation and multi-view reconstruction have experienced rapid development due to the emergence of high-quality datasets [1, 2, 3] and the overall achievements in the field of deep neural networks [4, 5, 6]. These two tasks imply quite different problem statements. Generation is concerned with learning some distribution of the 3D objects or producing an object which is consistent with the given text description (or other condition). On the other hand, multi-view reconstruction is aimed at recovering accurate geometry from the set of posed 2D images.

Despite this difference in the formulation of these two problems, their solutions share two common aspects: in the context of deep learning, they are often optimized through 2D supervision employing the differentiable rendering framework [7, 8], and they require a powerful underlying 3D representation. While there are a number of established 3D representations in the field of computer graphics, not all of them are well-suited to the framework of differentiable rendering. The two most popular representations: meshes and point clouds, while computationally and memory effective, lack some important qualities. Since they represent only the boundaries of the object, they can be spatially optimized via gradient descent only locally, which makes the optimization prone to getting stuck in the local minima [9]. Also, these 3D representations cannot model the view-dependent appearance effects, which harms their ability to learn from the 2D images in the real-life scenarios. Thus, there emerges a need for more powerful and flexible representations which are well-suited for gradient optimization.

Implicit neural representation (INR) [10] is a universal framework for fitting arbitrary multidimensional signals, including images, videos, and audio. INR is a multilayer neural network that takes as input the embedding vector of the given coordinate and outputs a vector representing the signal value at that particular coordinate. Since INRs are general-purpose deep neural networks, they can be optimized through stochastic gradient descent to fit any given signal. Deep neural networks are designed to work with high-dimensional representations, which gives the INRs an advantage over established 3D representations. In application to 3D generation and multi-view reconstruction, INRs are mostly used to represent two kinds of neural fields - the radiance field and the signed distance field. Due to their volumetric nature, they can be effectively optimized with gradient descent [11] and, thanks to their representational power, they can model complex view-dependent radiance effects [7].

The relevance of research

Despite their overall performance and flexibility, there are multiple obstacles, which prevent INRs from being effectively applied to solving complex 3D generation and multi-view reconstruction problems. Here, we highlight the main problems and blind spots in the relevant research.

• Regular multilayer perceptron networks struggle to represent simultaneously a wide variety of objects because of their architectural restrictions.

• On the other hand, generating the set of objects separately results in severe misalignment of these objects in terms of structure and geometry, which significantly complicates the subsequent use in applications.

• Generative frameworks (GANs, diffusion models) when combined with INRs usually result in learning models with entangled appearance/shape representation, which restricts their effectiveness and the ability to manipulate samples.

• Rendering of the implicit neural field is a computationally expensive process due to ray marching, and significantly lags behind more efficient representations. This process is essential for learning-based geometry reconstruction and generation.

• Multi-view image reconstruction with an implicit neural surface is restricted to reconstructing the part of the surface that is visible from a large enough set of images.

Research objectives and scope

The aim of the current thesis is the development of the INR methods and architectures in application to 3D generation and multi-view reconstruction problems. This includes improving the representational power of the INRs in itself and improving rendering algorithms for the neural fields. However, applying INRs to generation tasks requires adapting them to different generative learning approaches, including generative adversarial learning [5] and diffusion models [12]. From a practical point of view, training the generative models with interpretable disentangled representations of the objects is desirable because it allows advanced manipulation of the results. Shape/appearance disentanglement (or, which is very similar, the generation of the aligned objects) is an important goal of the thesis, since it gives control over the pose, structure, and appearance of the generated objects. Another topic developed in the current thesis is at the junction of the multi-view reconstruction and 3D generation, covering the case where only part of the object is visible and could be reconstructed directly, while the other part requires the help of the 3D generation methods.

Importance of work

1. A method for INR-based image generation is suggested, which is based on generative adversarial networks and hypernetworks. It is demonstrated for the first time that a generator without convolutions can achieve results on par with the state-of-the-art methods.

2. A new image generative method for the unsupervised disentanglement of the shape from the appearance is proposed. The idea of the algorithm is to divide the generator network into two parts, with deformation part being represented by INR. Laplacian Pyramid smoothing

and self-distillation are introduced to improve optimization. Multiple applications of the method are demonstrated, including image editing and appearance swapping.

3. The new method for generation of the aligned 3D objects is suggested, based on embedding them separately, on par with their transitions into a single INR. The method can be applied for a variety of tasks, including 3D editing and combining the novel object from parts of the generated objects.

4. A method for a plausible reconstruction of the surface which is partially unobserved (covered) is introduced. The surface is represented by an INR, and the reconstruction of the unobserved part is guided by the image diffusion model.

5. A novel algorithm for multi-view reconstruction of the neural surface represented with an INR is introduced, which is based on joint optimization with the set of the guiding spheres. The algorithm improves the computational efficiency of neural surface rendering, which in turn improves the quality of the reconstructed surface.

Theoretical and practical significance

The work advances a unified view of implicit neural representations and score-based generative priors by casting SDS within a differentiable-rendering operator. It formalizes sphere-guided rendering as a geometric prior and analyzes conditions that promote optimization stability, multiview consistency, and shape/appearance disentanglement via hypernetwork modulation. These insights refine identifiability assumptions for INR-based reconstruction and outline general design principles for scalable 3D generative models.

Developed INR-based methods have multiple practical applications. For example, the method for the generation of the aligned 3D models can be used in multiple tasks, including 3D model design, industrial design, computer graphics, and animation. The sphere-based method for 3D reconstruction notably improves the rendering speed and reconstruction quality, making it appealing for industrial and robotics applications. On the other hand, diffusion-based method for the reconstruction of the inaccessible parts of the surface can aid in the preservation of cultural heritage by completing/restoring the 3D models of the damaged cultural artifacts and in medicine, for example, by reconstructing the organs after amputations. The hypernetwork-based generative framework has significant potential to be used in big generative AI systems, helping users in 2D/3D design.

Methodology and methods of research

Methodologically, the dissertation builds on differentiable rendering supervised by 2D projections and adopts implicit neural representations (INRs) as a universal 3D representation. Generative approaches—GANs and diffusion models with score distillation sampling (SDS)—are combined with hypernetworks and deformable INRs to achieve multi-view consistency and disentangle

shape from appearance. For reconstruction, NeRF/NeuS-style models are employed alongside a proposed sphere-guided rendering scheme. Empirical validation comprises ablations and comparisons to state-of-the-art methods on standard datasets (DTU, BlendedMVS, Realistic Synthetic-360, LSUN, FFHQ) using FID and geometric metrics (surface accuracy, cross-view consistency). Experiments are executed within a unified computing environment under fixed protocols with fully specified hyperparameters to ensure reproducibility.

Main works and contributions

The dissertation is based on the following works, done during its course.

1. Adversarial generation of continuous images. (see Publications: Conference 2) Summary This work explores the adversarial generation of continuous images represented in the form of implicit neural representations (INRs). Two principled architectural techniques are proposed in the work: factorized multiplicative modulation and multi-scale INRs, which allowed us to obtain solid state-of-the-art results in continuous image generation. Several attractive properties of INR-based decoders are explored and discussed. Contribution The author has proposed the key ideas of the work, including the main concept of the work, using the INR in a combination with the hypernetwork in the role of the generator. Also, the author has suggested the algorithm of the factorized multiplicative modulation and provided some initial experiments behind the work.

2. Deforming Implicit Neural Representation Generative Adversarial Network for Unsupervised Appearence Editing (see Publications: Journal 1)

Summary This work suggests a novel deformation-based image generation model, which separately produces the aligned appearance texture and the deformation displacement map. Such an approach is implemented within the Generative Adversarial framework and allows one to effectively disentangle the appearance of the generated objects in the images from their shape. Naive training of the deformation-based network faces optimization problems, thus a number of ways to improve convergence are suggested, including invertible blurring and self-distillation. The method is demonstrated to produce samples of decent quality with complex image datasets including churches and faces, while preserving the alignment of the generated texture maps. The disentanglement achieved by the method allows to effortlessly edit the whole distribution of the generated samples by applying the same edit to the aligned texture maps. It is demonstrated, that the approach can be used to obtain the segmentation masks for the generated image samples.

Contribution The author has suggested all the key ideas behind the method: the architecture for shape appearance disentanglement, invertible blurring and self-distillation. The author has written all the code behind the work and did most of the experiments.

3. A3D: Does Diffusion Dream about 3D Alignment? (see Publications: Conference 1) Summary A3D is presented, the first method designed to generate a collection of objects structurally aligned with each other. This is achieved by encouraging the transitions between the objects, jointly embedded into a shared latent space, to be smooth and meaningful, which is demonstrated to be an essential property for proper alignment. It is shown that, when applied to the generation of the structurally aligned objects, the method outperforms the editing-based competitors in terms of the asset quality and text-object alignment, while keeping the geometric structural alignment on the state-of-art level. When applied to the 3D editing task, the method provides results on par with recent methods specialized in this problem. The method allows to compose novel objects, seamlessly combining the parts from the different aligned objects in the generated collection.

Contribution The author has suggested the method algorithm and implemented all the code behind the method. The author has also provided the majority of the experiments, excluding the hybridization experiments.

4. NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion. (see Publications: Journal 2)

Summary A novel framework, called NeuSD is proposed. It mixes traditional NeuS-based 3D shape reconstruction with generative completion. NeuSD is especially useful for cases where part of a shape is observed by multiple images and another part of the shape is not observed. It creates a plausible completion of the shape that is semantically meaningful and consistent with the observed parts. The main components of the approach are: 1) surface diffusion guidance, 2) freezing noise, and 3) multi-view SDS for reconstruction-generation consistency. The results outperform the current state of the art in qualitative and quantitative metrics.

Contribution The author has suggested all the main ideas behind the method, including the GridSDS, FrozenSDS and NormalSDS. The author has also implemented all the code for the method and did a significant part of the experiments in the work.

5. Sphere-guided training of neural implicit surfaces. (see Publications: Conference 3) Summary A method is presented, for improving volume rendering of implicit surface representations by restricting the estimated volume to the region of interest, defined by the set of trainable spheres. It is shown that focusing the optimization process on the estimated surface region leads to an increased quality of reconstructions obtained by the algorithm. At the same time, the proposed sphere cloud optimization approach ensures that the guiding representation closely follows the estimated surface of the object and accurately represents it at each step of the training. An extensive evaluation of the method is conducted in the study, which includes combining it with four base systems for implicit function training, and it is demonstrated that the method improves their performance across multiple benchmark datasets. The method also shows clear improvements in the modeling of real-world

scenes when applied without any additional supervision and post-filtering of the obtained reconstructions, which shows its suitability for in the-wild applications. Contribution The author has suggested the idea of guided surface formation via jointly optimized point cloud. In addition, the author has implemented the initial prototype experiments and advised the project through its course.

Publications and approbation of the work

The main achievements of the dissertation are presented in the following published articles and

conference proceedings.

Conferences

1. Ignatyev S., Konovalova N., Selikhanovych D., Voynov O., Patakin N., Olkov I., Senushkin D., Artemov A., Konushin A., Filippov A. and Wonka P., Burnaev E. (2024). A3D: Does Diffusion Dream about 3D Alignment?. arXiv preprint arXiv:2406.15020. [Accepted to ICLR 2025, Conference paper, CORE, A*, first author]

2. Skorokhodov I., Ignatyev S., & Elhoseiny M. (2021). Adversarial generation of continuous images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10753-10764). [Conference paper, CORE, A*]

3. Dogaru A., Ardelean A. T., Ignatyev S., Zakharov E., & Burnaev E. (2023). Sphere-guided training of neural implicit surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20844-20853). [Conference paper, CORE, A*]

4. Fan, Y., Skorokhodov, I., Voynov, O., Ignatyev, S., Burnaev, E., Wonka, P., & Wang, Y. (2023). Factored-neus: Reconstructing surfaces, illumination, and materials of possibly glossy objects. arXiv preprint arXiv:2305.17929. [Accepted to CVPR 2025, Conference paper, CORE, A*]

The results of the work were presented in the following conferences:

• IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021, June 20 2021 to June 25 2021, Nashville, TN, USA

• IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023, Oct. 1 2023 to Oct.6 2023, Paris, France

• The Thirteenth International Conference on Learning Representations 2025, Apr.24 2025 to Apr.28 2025, Singapore

• IEEE/CVF Conference on Computer Vision and Pattern Recognition 2025, June 11 2025 to June 15 2025, Nashville, TN, USA

Seminars

1. Implicit neural representations for 3D generation and multi-view 3D reconstruction, AIRI Seminars, March 5, 2025, Moscow

Journals

1. Ignatyev S., Egiazarian V., Rakhimov R., & Burnaev E. (2024). Deforming Implicit Neural Representation Generative Adversarial Network for Unsupervised Appearance Editing. Doklady Mathematics, 2024, volume 505, p. 46-49 [Web of Science, Q2, first author]

2. Ignatyev S., Selikhanovych D., Voynov O., Wang Y., Wonka P., Lefkimmiatis S., & Burnaev E. (2023). NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion. arXiv preprint arXiv:2312.04654. [Accepted to IEEEAccess, Web of Science, Q1, first author]

Side works

During a Ph.D. period, a number of works not directly related to the dissertation topic were

published.

1. Egiazarian V.*, Ignatyev S.*, Artemov A., Voynov O., Kravchenko A., Zheng Y., ... & Burnaev E. (2019). Latent-space laplacian pyramids for adversarial representation learning with 3d point clouds. 15th International Conference on Computer Vision Theory and Applications. [Conference paper, CORE, B, equal contribution]

2. Ishimtsev V., Bokhovkin A., Artemov A., Ignatyev S., Niessner M., Zorin D., & Burnaev E. (2020). Cad-deform: Deformable fitting of cad models to 3d scans. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16 (pp. 599-628). Springer International Publishing. [Conference paper, CORE, A*]

3. Kovalchuk M.V., Chaivanov B.B.,Abalin S.S., Ignatyev S.V. (2021) Revisiting the Choice of Nuclear Power Source for Arctic. Physics of Atomic Nuclei, Dec. 2021, pp.31-43. [Web of Science, Q2]

Proposals Submitted for Defense

1. A novel GAN generator architecture based on the hypernetwork INR with sinusoidal em-beddings. A new computationally efficient method for low-rank hypernetwork weight modulation. Scalable INR architecture for high-resolution image generation.

2. New INR warping-based GAN generator architecture. An algorithm for warping-based GAN optimization relying on smoothing of the image spatial gradients and using self-distillation.

3. A novel method for generation of structurally aligned 3D objects, relying on learning smooth and plausible transitions between the objects, parametrized by INR.

4. A method for consistent multi-view reconstruction of the partially observable implicit neural surface. Multi-view consistent version of the SDS algorithm. Low noise variant of the SDS algorithm for improved convergence and quality.

5. A hybrid (sphere cloud/INR) representation for efficient rendering of the implicit surface. A new algorithm for joint optimization of the sphere cloud and neural implicit surface during multi-view surface reconstruction.

Structure and volume of the dissertation thesis

The dissertation thesis is structured into 7 sections and includes 125 pages , 45 figures, 10 tables

and 235 references.

Content of the dissertation thesis

The dissertation is structured into several sections, each focusing on a distinct research work:

• Chapter 3 - Adversarial Generation of Continuous Images (AGCI)

• Chapter 4 - Deforming Implicit Neural Representation Generative Adversarial Network for Unsupervised Appearence Editing (Deforming INR)

• Chapter 5 - A3D: Does Diffusion Dream About 3D Alignment? (A3D)

• Chapter 6 - NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion (NeuSD)

• Chapter 7 - Sphere-Guided Training of Neural Implicit Surfaces (SGTNIS)

In turn, every separate chapter includes:

1. Problem statement (Introduction)

2. Literary review of the related works and their description (Related work)

3. Description of the developed method or algorithm (Method)

4. Experimental comparison of the method with regard to the existing counterparts (Experiments)

5. Conclusions, discussion of the work, restrictions of the method and further research directions (Conclusion)

Along with the described above, the thesis includes the literary review of the most general topics related to the theme of the dissertation in general (Chapter 1). The necessary theoretical foundations of the INRs are set out in (Chapter 2). In conclusion of the the thesis (Chapter 7.5) there is the summary of all of the results achieved in the thesis.