Neural Radiance Fields (NeRFs) have revolutionized the field of inverse rendering due to their ability to synthesize high-quality novel views and applicability in practical contexts. NeRFs leverage volume rendering, evaluating view-dependent color at each sample with an expensive network, where a high computational burden is placed on extracting an informative, view-independent latent code.
We propose a temporal coherence method to accelerate NeRF rendering by caching the latent codes of all samples in an initial viewpoint and reusing them in consecutive frames. By utilizing a sparse frustum volume grid for caching and performing lookups via backward reprojection, we enable temporal reuse of NeRF samples while maintaining the ability to re-evaluate view-dependent effects efficiently. To facilitate high-fidelity rendering from our cache with interactive framerates, we propose a novel cone encoding and explore a training scheme to induce local linearity into the latent information.
Extensive experimental evaluation demonstrates that these choices enable high-quality real-time rendering from our cache, even when reducing latent code size significantly. Our proposed method scales exceptionally well for large networks, and our highly optimized real-time implementation allows for cache initialization at runtime. For offline rendering of high-quality video sequences with expensive supersampled effects like motion blur or depth of field, our approach provides speed-ups of up to 2×.
We are able to perform fast lookups into our frustum-aligned volumetric cache via backward projection. By utilizing an occupancy grid, we can detect cache-misses and seamlessly combine cached samples with newly evaluated ones along view rays. For cache-hits, only the significantly smaller head network needs to be executed to evaluate view-dependent effects. For small viewpoint changes, cache renderings are almost indistinguishable from the baseline.
@article{steiner2024nerfcaching,
author = {Steiner, Michael and Köhler, Thomas and Radl, Lukas and Steinberger, Markus},
title = {{Frustum Volume Caching for Accelerated NeRF Rendering}},
journal = {Proc. ACM Comput. Graph. Interact. Tech.},
volume = {7},
number = {3},
articleno = {39},
numpages = {22},
year = {2024},
}