Capture to Rendering Pipeline for Generating Dynamically Relightable Virtual Objects with Handheld RGB-D Cameras

Chih Fan Chen, Evan Suma Rosenberg

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a complete end-to-end pipeline for generating dynamically relightable virtual objects captured using a single handheld consumer-grade RGB-D camera. The proposed system plausibly replicates the geometry, texture, illumination, and surface reflectance properties of non-Lambertian objects, making them suitable for integration within virtual reality scenes that contain arbitrary illumination. First, the geometry of the target object is reconstructed from depth images captured using a handheld camera. To get nearly drift-free texture maps of the virtual object, a set of selected images from the original color stream is used for camera pose optimization. Our approach further separates these images into diffuse (view-independent) and specular (view-dependent) components using low-rank decomposition. The lighting conditions during capture and reflectance properties of the virtual object are subsequently estimated from the computed specular maps. By combining these parameters with the diffuse texture, the reconstructed model can then be rendered in real-time virtual reality scenes that plausibly replicate real world illumination at the point of capture. Furthermore, these objects can interact with arbitrary virtual lights that vary in direction, intensity, and color.

Original languageEnglish (US)
Title of host publicationProceedings - VRST 2020
Subtitle of host publicationACM Symposium on Virtual Reality Software and Technology
EditorsStephen N. Spencer
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450376198
DOIs
StatePublished - Nov 1 2020
Externally publishedYes
Event26th ACM Symposium on Virtual Reality Software and Technology, VRST 2020 - Virtual, Online, Canada
Duration: Nov 1 2020Nov 4 2020

Publication series

NameProceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST

Conference

Conference26th ACM Symposium on Virtual Reality Software and Technology, VRST 2020
CountryCanada
CityVirtual, Online
Period11/1/2011/4/20

Keywords

  • content creation
  • reconstruction
  • scanning
  • virtual reality

Fingerprint Dive into the research topics of 'Capture to Rendering Pipeline for Generating Dynamically Relightable Virtual Objects with Handheld RGB-D Cameras'. Together they form a unique fingerprint.

Cite this