Research
M&M VTO: Multi-Garment Virtual Try-On and Editing, CVPR 2024 (selected as Highlight, 11.9% of papers)
Luyang Zhu, Yingwei Li, Nan Liu, Hao Peng, Dawei Yang, Ira Kemelmacher-Shlizerman
Key idea: given three images (person, top garment, bottom garment) produce how would the input person look like in those two garments.
Webpage: coming up
Total Selfie: Generating Full Body Images from Selfies, CVPR 2024 (selected as Highlight, 11.9% of papers)
Bowei Chen, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman.
Key idea: Take an image of your face, legs, shoes, and body with your phone and we'll generate a full body selfie for you.
Generative Powers of Ten, CVPR 2024 (selected as Highlight, 11.9% of papers)
Xiaojuan Wang, Janne Kontkanen, Brian Curless, Steve Seitz, Ira Kemelmacher, Ben Mildenhall, Pratul Srinivasan, Dor Verbin, Aleksander Holynski
Key idea: Given a set of prompts describing the scene at various scales, our method creates a seamless zooming video.
Animating Street View, SIGGRAPH Asia 2023
Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz
Key idea: We make street scenes alive by inserting naturally behaving pedestrians and vehicles.
DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion, ICCV 2023
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, Ira Kemelmacher-Shlizerman
Key idea: DreamPose is a diffusion-based image-to-video synthesis model. Given an input image of a person and pose sequence, DreamPose synthesizes a photorealistic video of the input person following the pose sequence.
TryOnDiffusion: A Tale of Two UNets, CVPR 2023
Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, Ira Kemelmacher-Shlizerman
Key idea: Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person.
PersonNeRF: Personalized Reconstruction from Photo Collections, CVPR 2023
Chung-Yi Weng, Pratul Srinivasan, Brian Curless, Ira Kemelmacher-Shlizerman
Key idea:
We present PersonNeRF, a method that takes a collection of photos of a subject (e.g., Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance.
HRTF Estimation in the Wild, UIST 2023
Vivek Jayaram, Ira Kemelmacher-Shlizerman, Steve Seitz
Key idea:
Estimate personalized HRTF (for spatial audio) just from headphones. project page
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation, CVPR 2022 oral, 3% out of ~8k papers
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, JJ Park, Ira Kemelmacher-Shlizerman
Key idea:
Our method is trained on single-view RGB data only while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape.
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video, CVPR 2022 oral, 3% out of ~8k papers
Chung-Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
Key idea:
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose.
ClearBuds: Wireless Binaural Earbuds for Learning-based Speech Enhancement, Mobisys 2022 oral. Best demo award runner up.
Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher-Shlizerman, Shwetak Patel, Steven M. Seitz
Key idea:
ClearBuds is a state-of-the-art hardware and software system for real-time speech enhancement. Our neural network runs completely on an iphone, allowing to suppress unwanted noises while taking phone calls on the go. Results show that our wireless earbuds achieve a synchronization error less than 64 $\mu$s and our network has a runtime of 21.4 ms on an accompanying mobile phone.
A Light Stage on Every Desk
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz, ICCV 2021
Key idea:
While watching a YouTube video the monitor is projecting patterns on the face. Our algorithm shows how to leverage the patterns to relight and preserve privacy.
TryOnGAN: Body-Aware Try-On via Layered Interpolation
Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman, SIGGRAPH 2021
Key idea:
switch garments and change their size to adjust to new humans via stylegan latent space interpolation.
project page
Real-Time High Resolution Background Matting
Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman, CVPR 2021 oral 3% of papers
Best student paper honorable mention; 0.08% of 7093 papers
Key idea:
We achieve HD matting at 60fps, by estimating alpha per frame in a sequence of CNNs from coarse to refined estimation.
project page
Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild, 2021
Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman
The Cone of Silence: Speech Separation by Localization
Teerapat Jenrungrot*, Vivek Jayaram*, Steve Seitz, Ira Kemelmacher-Shlizerman, NeurIPS, 2020 oral 2% of papers
Key idea:
Build a mic array that allows to use spatial info for speaker separation and denoising
project page | Science article
Reconstructing NBA players
Luyang Zhu, Konstantinos Rematas, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, ECCV 2020 (spotlight)
Key idea: Use synthetic NBA2k data to estimate pose and mesh from photos of real NBA players. project page
Lifespan Age Transformation Synthesis
Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman, ECCV, 2020
Key idea: GAN for unpaired aging while preserving the identity of the person.
project page | paper | dataset | code | colab
Background Matting: The World is Your Green Screen
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, CVPR 2020
project page | paper | code | Two Minute Papers video | Microsoft AI using our code | CEO of Microsoft Satya Nadella talks about our work.
Mixed Reality Spatial Computing in a Remote Learning Classroom
John Akers, Joelle Zimmermann, Laura Trutoiu, Brian Schowengerdt, Ira Kemelmacher-Shlizerman, ACM SUI 2020
Best Poster/Demo Honorable Mention
Photo Wake-Up: 3D Character Animation from a Single Photo
Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman, CVPR 2019
project page | paper | MIT Tech Review | NVIDIA
Soccer on your Tabletop
Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, Steve Seitz
CVPR 2018
project page | paper | code | Forbes | NVIDIA
Audio to Body Dynamics
Eli Shlizerman, Lucio M. Dery, Hayden Schoen, and Ira Kemelmacher-Shlizerman, CVPR 2018
project page | paper | code | CNBC | Facebook Research blog
Video to Fully Automatic 3D Hair Model
Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda Shapiro, Ira Kemelmacher-Shlizerman, SIGGRAPH Asia 2018
project page | paper | data
Synthesizing Obama: Learning Lip Sync from Audio
Supasorn Suwajanakorn, Steve Seitz, and Ira Kemelmacher-Shlizerman, SIGGRAPH 2017,
project page | paper | training videos | code | Video @ Two Minute Papers
1Million + views on YouTube
Transfiguring Portraits
Ira Kemelmacher-Shlizerman, SIGGRAPH 2016
video| paper | The Daily Mail
Head Reconstruction from Internet Photos
Shu Liang, Linda Shapiro, Ira Kemelmacher-Shlizerman, ECCV 2016
project page | paper | dataset
The MegaFace Benchmark
The MegaFace Benchmark, Kemelmacher-Shlizerman, Seitz, Miller, Brossard. CVPR 2016
Level Playing Field for Million Scale Face Recognition, Nech and Kemelmacher-Shlizerman, CVPR 2017
project page | paper CVPR 2016 | paper CVPR 2017 | The Atlantic
What Makes Tom Hanks Look Like Tom Hanks
Suwajanakorn, Seitz, Kemelmacher-Shlizerman, ICCV 2015
project page | paper | video (450K views on YouTube)
The Meme Quiz: A Facial Expression Game Combining Human Agency and Machine Involvement
K. Tuite and I. Kemelmacher-Shlizerman, Foundations of Digital Games (FDG), 2015
project video
3D Face Hallucination from a Single Depth Frame
S. Liang, I. Kemelmacher-Shlizerman, L.G. Shapiro, International Conf. on 3D Vision (3DV), Tokyo, Dec 2014
project webpage
Total Moving Reconstruction
S. Suwajanakorn, I. Kemelmacher-Shlizerman, S.M. Seitz, European Conference on Computer Vision (ECCV), Zurich, Sep 2014
project page | paper | video
Illumination-aware Age Progression
Ira Kemelmacher-Shlizerman, S. Suwajanakorn, S.M. Seitz, CVPR 2014
project webpage
Exploring Photobios
Kemelmacher-Shlizerman, Shechtman, Garg and Seitz, SIGGRAPH 2011
Moving Portraits, I. Kemelmacher-Shlizerman, E. Shechtman, R. Garg, S.M. Seitz, Communications of the ACM, Research Highlights, 2014.
project page | paper SIGGRAPH | paper ACM Research Highlights | video | Face Movies of Google
Appeared on CACM cover, and SIGGRAPH cover.
Internet based morphable model
Ira Kemelmacher-Shlizerman
International Conf. on Comp. Vision (ICCV), Sydney, Dec 2013
project page | paper | video
3D Face Reconstruction from Single Two-Tone and Color Images
Ira Kemelmacher-Shlizerman, R. Basri and B. Nadler
Shape Perception in Human and Computer Vision, Springer London, 2013
book chapter
Global Motion Estimation from Point Matches
Mica Arie-Nachimson, Shahar Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, Ronen Basri, 3DimPVT (3DV), 2012
paper
Collection Flow
Ira Kemelmacher-Shlizerman and Steven M. Seitz, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June, 2012
project page | paper | video
Face Reconstruction in the Wild
Ira Kemelmacher-Shlizerman and Steven M. Seitz, International Conference on Computer Vision (ICCV), Nov, 2011
project page | paper
Exploring Photobios
Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg and Steven M. Seitz, ACM Transactions on Graphics (SIGGRAPH), Aug, 2011
SIGGRAPH cover and in the Technical papers trailer. Tech transfered to Google, as the Face Movies feature in Picasa.
project page | Face Movies
3D Face Reconstruction from a Single Image using a Single Reference Face Shape
Ira Kemelmacher-Shlizerman, Ronen Basri, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2010
project page
A Theory of Locally Low Dimensional Light Transport
Dhruv Mahajan, Ira Kemelmacher-Shlizerman, Ravi Ramamoorthi and Peter Belhumeur, SIGGRAPH 2007
project page
Photometric Stereo with General, Unknown Lighting
Ronen Basri, David W. Jacobs and Ira Kemelmacher, International Journal of Computer Vision (IJCV), 72(3):239-257, 2007
project page
Molding Face Shapes by Example
Ira Kemelmacher and Ronen Basri, In European Conference on Computer Vision (ECCV), 2006
project page
Indexing with Unknown Illumination and Pose
Ira Kemelmacher and Ronen Basri, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2005
project page
