StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation, CVPR 2022 oral, 3% out of ~8k papers
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, JJ Park, Ira Kemelmacher-Shlizerman
Our method is trained on single-view RGB data only while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape.
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video, CVPR 2022 oral, 3% out of ~8k papers
Chung-Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose.
ClearBuds: Wireless Binaural Earbuds for Learning-based Speech Enhancement, Mobisys 2022 oral. Best demo award runner up.
Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher-Shlizerman, Shwetak Patel, Steven M. Seitz
ClearBuds is a state-of-the-art hardware and software system for real-time speech enhancement. Our neural network runs completely on an iphone, allowing to suppress unwanted noises while taking phone calls on the go. Results show that our wireless earbuds achieve a synchronization error less than 64 $\mu$s and our network has a runtime of 21.4 ms on an accompanying mobile phone.
A Light Stage on Every Desk
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz, ICCV 2021
While watching a YouTube video the monitor is projecting patterns on the face. Our algorithm shows how to leverage the patterns to relight and preserve privacy.
TryOnGAN: Body-Aware Try-On via Layered Interpolation
Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman, SIGGRAPH 2021
switch garments and change their size to adjust to new humans via stylegan latent space interpolation.
Real-Time High Resolution Background Matting
Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman, CVPR 2021 oral 3% of papers
Best student paper honorable mention; 0.08% of 7093 papers
We achieve HD matting at 60fps, by estimating alpha per frame in a sequence of CNNs from coarse to refined estimation.
Mixed Reality Spatial Computing in a Remote Learning Classroom
John Akers, Joelle Zimmermann, Laura Trutoiu, Brian Schowengerdt, Ira Kemelmacher-Shlizerman, ACM SUI 2020
Best Poster/Demo Honorable Mention
Reconstructing NBA players
Luyang Zhu, Konstantinos Rematas, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, ECCV 2020 (spotlight)
Key idea: Use synthetic NBA2k data to estimate pose and mesh from photos of real NBA players. project page
Kemelmacher-Shlizerman, Shechtman, Garg and Seitz, SIGGRAPH 2011
Moving Portraits, I. Kemelmacher-Shlizerman, E. Shechtman, R. Garg, S.M. Seitz, Communications of the ACM, Research Highlights, 2014.