pixelNeRF: Neural Radiance Fields from One or Few Images

• Authors: Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa

• Affiliations: UC Berkeley

• CVPR, 2021

• Links: project

Summary

Existing approaches invole optimizing the representation to every scene independently, requiring many calibrated view and significantly compute time. In this work, the authors proposed pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. Experiments on ShapeNet and DTU datasets show that pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction.

Key Ideas

Standard NeRF. In a standard NeRF \(f\), given a 3D point \(\mathbf{x} \in \mathbb{R}^3\) and viewing direction \(\mathbf{d} \in \mathbb{R}^3\), \(f\) returns a differential density \(\sigma\) and RGB color \(\mathbf{c}\): \(f(\mathbf{x}, \mathbf{c}) = (\sigma, \mathbf{c})\).

The volumetric radiance field can then be rendered into a 2D image via:

\[\hat{\mathbf{C}} = \int_{t_n}^{t_f} T(t)\sigma(t)\mathbf{c}(t)dt\]

where \(T(t) = \exp(-\int_{t_n}^t \sigma(s) ds)\) handles occlusion.

Image-conditioned NeRF. Given a input image \(I\) of a scene, we extract a feature volume \(W = E(I)\). For a point on a camera ray \(x\), we retrieve the image feature by projecting \(x\) onto thte image plane and obtain the feature vector \(W(\pi(x))\) with bilinear interpolation. Then the image features are passed into the NeRF network as

\[f(\gamma(x), d; W(\pi(x))) = (\sigma, c)\]

The image features are incorporated as a residual at each layer. The pipeline is depicted below.

Figure 1: Overview of pixelNeRF.

Technical Details

Incorporating multiple view. The model can be extended to allow for an arbitrary number of views at test time. Let the \(i\) input image be \(I^{(i)}\) and the associated camera transform from the world space to its view space be \(P^{(i)} = [R^{(i)}, t^{(i)}]\). Let the initial layers in the NeRF network be \(f_1\) and the final layers be \(f_2\). We obtain intermediate vectors from the initial layers and then aggregate with average pooling operator \(\phi\):

\[\begin{split}\begin{align*} V^{(i)} & = f_1 \left( \gamma(x^{(i)}), d^{(i)}; W^{(i)} (\pi(x^{(i)})) \right) \\ (\gamma, c) & = f_2 (\phi(V^{(i)}, \dots, V^{(n)})) \end{align*}\end{split}\]

Image encoder. To capture both local and global information, a feature pyramid is extracted. Then the image features are added as a residual at the beginning of each ResNet block.

Category-agnostic single-view reconstruction.

Figure 2: Quantitative results on category-agnostic single-view reconstruction.

Notes

References

[1] A. Yu, V. Ye, M. Tancik, A. Kanazawa. “pixelNeRF: Neural radiance fields from one or few images.”. In CVPR, 2021.