PersonNeRF: Personalized Reconstruction from Photo Collections, CVPR 2023
Chung-Yi Weng, Pratul Srinivasan, Brian Curless, Ira Kemelmacher-Shlizerman
We present PersonNeRF, a method that takes a collection of photos of a subject (e.g., Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance.
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation, CVPR 2022 oral, 3% out of ~8k papers
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, JJ Park, Ira Kemelmacher-Shlizerman
Our method is trained on single-view RGB data only while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape.
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video, CVPR 2022 oral, 3% out of ~8k papers
Chung-Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose.
ClearBuds: Wireless Binaural Earbuds for Learning-based Speech Enhancement, Mobisys 2022 oral. Best demo award runner up.
Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher-Shlizerman, Shwetak Patel, Steven M. Seitz
ClearBuds is a state-of-the-art hardware and software system for real-time speech enhancement. Our neural network runs completely on an iphone, allowing to suppress unwanted noises while taking phone calls on the go. Results show that our wireless earbuds achieve a synchronization error less than 64 $\mu$s and our network has a runtime of 21.4 ms on an accompanying mobile phone.
A Light Stage on Every Desk
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz, ICCV 2021
While watching a YouTube video the monitor is projecting patterns on the face. Our algorithm shows how to leverage the patterns to relight and preserve privacy.
TryOnGAN: Body-Aware Try-On via Layered Interpolation
Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman, SIGGRAPH 2021
switch garments and change their size to adjust to new humans via stylegan latent space interpolation.
Real-Time High Resolution Background Matting
Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman, CVPR 2021 oral 3% of papers
Best student paper honorable mention; 0.08% of 7093 papers
We achieve HD matting at 60fps, by estimating alpha per frame in a sequence of CNNs from coarse to refined estimation.
Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild, 2021
Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman
Mixed Reality Spatial Computing in a Remote Learning Classroom
John Akers, Joelle Zimmermann, Laura Trutoiu, Brian Schowengerdt, Ira Kemelmacher-Shlizerman, ACM SUI 2020
Best Poster/Demo Honorable Mention
The Cone of Silence: Speech Separation by Localization
Teerapat Jenrungrot*, Vivek Jayaram*, Steve Seitz, Ira Kemelmacher-Shlizerman, NeurIPS, 2020 oral 2% of papers
Build a mic array that allows to use spatial info for speaker separation and denoising
project page | Science article
Reconstructing NBA players
Luyang Zhu, Konstantinos Rematas, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, ECCV 2020 (spotlight)
Key idea: Use synthetic NBA2k data to estimate pose and mesh from photos of real NBA players. project page
Lifespan Age Transformation Synthesis
Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman, ECCV, 2020
Key idea: GAN for unpaired aging while preserving the identity of the person.
project page | paper | dataset | code | colab
Background Matting: The World is Your Green Screen
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, CVPR 2020
project page | paper | code | Two Minute Papers video | Microsoft AI using our code | CEO of Microsoft Satya Nadella talks about our work.
Photo Wake-Up: 3D Character Animation from a Single Photo
Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman, CVPR 2019
project page | paper | MIT Tech Review | NVIDIA
Soccer on your Tabletop
Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, Steve Seitz
project page | paper | code | Forbes | NVIDIA
Audio to Body Dynamics
Eli Shlizerman, Lucio M. Dery, Hayden Schoen, and Ira Kemelmacher-Shlizerman, CVPR 2018
project page | paper | code | CNBC | Facebook Research blog
Video to Fully Automatic 3D Hair Model
Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda Shapiro, Ira Kemelmacher-Shlizerman, SIGGRAPH Asia 2018
project page | paper | data
Synthesizing Obama: Learning Lip Sync from Audio
Supasorn Suwajanakorn, Steve Seitz, and Ira Kemelmacher-Shlizerman, SIGGRAPH 2017,
project page | paper | training videos | code | Video @ Two Minute Papers
1Million + views on YouTube
Head Reconstruction from Internet Photos
Shu Liang, Linda Shapiro, Ira Kemelmacher-Shlizerman, ECCV 2016
project page | paper | dataset
The MegaFace Benchmark
The MegaFace Benchmark, Kemelmacher-Shlizerman, Seitz, Miller, Brossard. CVPR 2016
Level Playing Field for Million Scale Face Recognition, Nech and Kemelmacher-Shlizerman, CVPR 2017
project page | paper CVPR 2016 | paper CVPR 2017 | The Atlantic
What Makes Tom Hanks Look Like Tom Hanks
Suwajanakorn, Seitz, Kemelmacher-Shlizerman, ICCV 2015
project page | paper | video (450K views on YouTube)
The Meme Quiz: A Facial Expression Game Combining Human Agency and Machine Involvement
K. Tuite and I. Kemelmacher-Shlizerman, Foundations of Digital Games (FDG), 2015
3D Face Hallucination from a Single Depth Frame
S. Liang, I. Kemelmacher-Shlizerman, L.G. Shapiro, International Conf. on 3D Vision (3DV), Tokyo, Dec 2014
Total Moving Reconstruction
S. Suwajanakorn, I. Kemelmacher-Shlizerman, S.M. Seitz, European Conference on Computer Vision (ECCV), Zurich, Sep 2014
project page | paper | video
Illumination-aware Age Progression
Ira Kemelmacher-Shlizerman, S. Suwajanakorn, S.M. Seitz, CVPR 2014
Kemelmacher-Shlizerman, Shechtman, Garg and Seitz, SIGGRAPH 2011
Moving Portraits, I. Kemelmacher-Shlizerman, E. Shechtman, R. Garg, S.M. Seitz, Communications of the ACM, Research Highlights, 2014.
project page | paper SIGGRAPH | paper ACM Research Highlights | video | Face Movies of Google
Appeared on CACM cover, and SIGGRAPH cover.
Internet based morphable model
International Conf. on Comp. Vision (ICCV), Sydney, Dec 2013
project page | paper | video
3D Face Reconstruction from Single Two-Tone and Color Images
Ira Kemelmacher-Shlizerman, R. Basri and B. Nadler
Shape Perception in Human and Computer Vision, Springer London, 2013
Global Motion Estimation from Point Matches
Mica Arie-Nachimson, Shahar Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, Ronen Basri, 3DimPVT (3DV), 2012
Ira Kemelmacher-Shlizerman and Steven M. Seitz, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June, 2012
project page | paper | video
Face Reconstruction in the Wild
Ira Kemelmacher-Shlizerman and Steven M. Seitz, International Conference on Computer Vision (ICCV), Nov, 2011
project page | paper
Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg and Steven M. Seitz, ACM Transactions on Graphics (SIGGRAPH), Aug, 2011
SIGGRAPH cover and in the Technical papers trailer. Tech transfered to Google, as the Face Movies feature in Picasa.
project page | Face Movies
3D Face Reconstruction from a Single Image using a Single Reference Face Shape
Ira Kemelmacher-Shlizerman, Ronen Basri, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2010
A Theory of Locally Low Dimensional Light Transport
Dhruv Mahajan, Ira Kemelmacher-Shlizerman, Ravi Ramamoorthi and Peter Belhumeur, SIGGRAPH 2007
Photometric Stereo with General, Unknown Lighting
Ronen Basri, David W. Jacobs and Ira Kemelmacher, International Journal of Computer Vision (IJCV), 72(3):239-257, 2007
Molding Face Shapes by Example
Ira Kemelmacher and Ronen Basri, In European Conference on Computer Vision (ECCV), 2006
Indexing with Unknown Illumination and Pose
Ira Kemelmacher and Ronen Basri, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2005