Research

Fashion-VDM: Video Diffusion Model for Virtual Try-On, SIGGRAPH Asia, Japan, 2024

Johanna Karras, Yingwei Li, Nan Liu, Luyang Zhu, Innfarn Yoo, Andreas Lugmayr, Chris Lee, Ira Kemelmacher-Shlizerman

Key idea: We present Fashion-VDM, a video diffusion model (VDM) for generating virtual try-on videos. Given an input garment image and person video, our method aims to generate a high-quality try-on video of the person wearing the given garment, while preserving the person's identity and motion.

Coming soon 

Inverse Painting: Reconstructing The Painting Process SIGGRAPH Asia, Japan, 2024

Bowei Chen, Yifan Wang, Brian Curless, Steven M. Seitz, Ira Kemelmacher-Shlizerman

Key idea: Given an input painting, we reconstruct a time-lapse video of how it may be painted. 

Coming soon 

M&M VTO: Multi-Garment Virtual Try-On and Editing, CVPR 2024 (selected as Highlight, 11.9% of papers) 

Luyang Zhu, Yingwei Li, Nan Liu, Hao Peng, Dawei Yang, Ira Kemelmacher-Shlizerman

Key idea: given three images (person, top garment, bottom garment) produce how would the input person look like in those two garments. 

Webpage 

Total Selfie: Generating Full Body Images from Selfies, CVPR 2024 (selected as Highlight, 11.9% of papers) 

Bowei Chen, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman. 

Key idea: Take an image of your face, legs, shoes, and body with your phone and we'll generate a full body selfie for you. 

webpage

Generative Powers of Ten, CVPR 2024 (selected as Highlight, 11.9% of papers) 

Xiaojuan Wang, Janne Kontkanen, Brian Curless, Steve Seitz, Ira Kemelmacher, Ben Mildenhall, Pratul Srinivasan, Dor Verbin, Aleksander Holynski

Key idea: Given a set of prompts describing the scene at various scales, our method creates a seamless zooming video.

webpage

Animating Street View, SIGGRAPH Asia 2023

Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz

Key idea: We make street scenes alive by inserting naturally behaving pedestrians and vehicles.

webpage

DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion, ICCV 2023

Johanna Karras, Aleksander Holynski, Ting-Chun Wang, Ira Kemelmacher-Shlizerman

Key idea: DreamPose is a diffusion-based image-to-video synthesis model. Given an input image of a person and pose sequence, DreamPose synthesizes a photorealistic video of the input person following the pose sequence.

webpage

TryOnDiffusion: A Tale of Two UNets, CVPR 2023

Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, Ira Kemelmacher-Shlizerman

Key idea: Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person.

webpage

PersonNeRF: Personalized Reconstruction from Photo Collections, CVPR 2023 

Chung-Yi Weng, Pratul Srinivasan, Brian Curless, Ira Kemelmacher-Shlizerman

Key idea:
We present PersonNeRF, a method that takes a collection of photos of a subject (e.g., Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance.

project page

HRTF Estimation in the Wild, UIST 2023 

Vivek Jayaram, Ira Kemelmacher-Shlizerman, Steve Seitz 

Key idea:
Estimate personalized HRTF (for spatial audio) just from headphones. project page

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation, CVPR 2022 oral, 3% out of ~8k papers

Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, JJ Park, Ira Kemelmacher-Shlizerman


Key idea:
Our method is trained on single-view RGB data only while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape. 

project page

HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video, CVPR 2022 oral, 3% out of ~8k papers

Chung-Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman

Key idea:
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose.

project page

ClearBuds: Wireless Binaural Earbuds for Learning-based Speech Enhancement, Mobisys 2022 oral. Best demo award runner up. 

Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher-Shlizerman, Shwetak Patel, Steven M. Seitz

Key idea:
ClearBuds is a state-of-the-art hardware and software system for real-time speech enhancement. Our neural network runs completely on an iphone, allowing to suppress unwanted noises while taking phone calls on the go. Results show that our wireless earbuds achieve a synchronization error less than 64 $\mu$s and our network has a runtime of 21.4 ms on an accompanying mobile phone.

project page

A Light Stage on Every Desk

Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz, ICCV 2021


Key idea:
While watching a YouTube video the monitor is projecting patterns on the face. Our algorithm shows how to leverage the patterns to relight and preserve privacy.  

project page

TryOnGAN: Body-Aware Try-On via Layered Interpolation

Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman, SIGGRAPH 2021


Key idea: 
switch  garments and change their size to adjust to new humans via stylegan latent space interpolation.
project page

Real-Time High Resolution Background Matting

Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman, CVPR 2021 oral 3% of papers 

Best student paper honorable mention; 0.08% of 7093 papers

Key idea:
We achieve HD matting at 60fps, by estimating alpha per frame in a sequence of CNNs from coarse to refined estimation.   
project page

Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild, 2021

Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman

project page

The Cone of Silence: Speech Separation by Localization

Teerapat Jenrungrot*, Vivek Jayaram*, Steve Seitz, Ira Kemelmacher-Shlizerman, NeurIPS, 2020 oral 2% of papers

Key idea:
Build a mic array that allows to use spatial info for speaker separation and denoising
project page | Science article

Reconstructing NBA players

Luyang Zhu, Konstantinos Rematas, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, ECCV 2020 (spotlight)


Key idea: Use synthetic NBA2k data to estimate pose and mesh from photos of real NBA players. project page 

Lifespan Age Transformation Synthesis

Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman, ECCV, 2020

Key idea: GAN for unpaired aging while preserving the identity of the person.
project page | paper | dataset | code | colab

Background Matting: The World is Your Green Screen

Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, CVPR 2020
project page | paper | code | Two Minute Papers video | Microsoft AI using our code | CEO of Microsoft Satya Nadella talks about our work.

Mixed Reality Spatial Computing in a Remote Learning Classroom

John Akers, Joelle Zimmermann, Laura Trutoiu, Brian Schowengerdt, Ira Kemelmacher-Shlizerman, ACM SUI 2020 

Best Poster/Demo Honorable Mention

project page 

Photo Wake-Up: 3D Character Animation from a Single Photo

Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman, CVPR 2019
project page | paper | MIT Tech Review | NVIDIA 

Soccer on your Tabletop

Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, Steve Seitz
CVPR 2018
project page | paper | code | Forbes | NVIDIA

Audio to Body Dynamics

Eli Shlizerman, Lucio M. Dery, Hayden Schoen, and Ira Kemelmacher-Shlizerman, CVPR 2018
project page | paper | code | CNBC | Facebook Research blog

Video to Fully Automatic 3D Hair Model

Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda Shapiro, Ira Kemelmacher-Shlizerman, SIGGRAPH Asia 2018
project page | paper | data

Synthesizing Obama: Learning Lip Sync from Audio

Supasorn Suwajanakorn, Steve Seitz, and Ira Kemelmacher-Shlizerman, SIGGRAPH 2017,
project page | paper | training videos | code | Video @ Two Minute Papers
1Million + views on YouTube

Transfiguring Portraits

Ira Kemelmacher-Shlizerman, SIGGRAPH 2016
video| paper | The Daily Mail

Head Reconstruction from Internet Photos

Shu Liang, Linda Shapiro, Ira Kemelmacher-Shlizerman, ECCV 2016
project page | paper | dataset 

The MegaFace Benchmark

project page | paper CVPR 2016 | paper CVPR 2017 | The Atlantic

What Makes Tom Hanks Look Like Tom Hanks 

Suwajanakorn, Seitz, Kemelmacher-Shlizerman, ICCV 2015
project page | paper | video (450K views on YouTube)

The Meme Quiz: A Facial Expression Game Combining Human Agency and Machine Involvement

K. Tuite and I. Kemelmacher-Shlizerman, Foundations of Digital Games (FDG), 2015
project video

3D Face Hallucination from a Single Depth Frame

S. Liang, I. Kemelmacher-Shlizerman, L.G. Shapiro, International Conf. on 3D Vision (3DV), Tokyo, Dec 2014
project webpage

Total Moving Reconstruction

S. Suwajanakorn, I. Kemelmacher-Shlizerman, S.M. Seitz, European Conference on Computer Vision (ECCV), Zurich, Sep 2014
project page | paper | video 

Illumination-aware Age Progression

Ira Kemelmacher-Shlizerman, S. Suwajanakorn, S.M. Seitz, CVPR 2014
project webpage

Exploring Photobios 

project page | paper SIGGRAPH | paper ACM Research Highlights | video | Face Movies of Google
Appeared on CACM cover, and SIGGRAPH cover.

Internet based morphable model

Ira Kemelmacher-Shlizerman

International Conf. on Comp. Vision (ICCV), Sydney, Dec 2013
project page  |  paper | video 

3D Face Reconstruction from Single Two-Tone and Color Images
Ira Kemelmacher-Shlizerman, R. Basri and B. Nadler

Shape Perception in Human and Computer Vision, Springer London, 2013
book chapter

Global Motion Estimation from Point Matches

Mica Arie-Nachimson, Shahar Kovalsky, Ira Kemelmacher-Shlizerman, Amit Singer, Ronen Basri, 3DimPVT (3DV), 2012
paper

Collection Flow

Ira Kemelmacher-Shlizerman and Steven M. Seitz, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), June, 2012
project page | paper  | video  

Face Reconstruction in the Wild

Ira Kemelmacher-Shlizerman and Steven M. Seitz, International Conference on Computer Vision (ICCV), Nov, 2011
project page | paper 

Exploring Photobios

Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg and Steven M. Seitz, ACM Transactions on Graphics (SIGGRAPH), Aug, 2011

SIGGRAPH cover and in the Technical papers trailer. Tech transfered to Google, as the Face Movies feature in Picasa.
project page | Face Movies 

Being John Malkovich

Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman and Steven M. Seitz

In European Conference on Computer Vision (ECCV), 2010
paper | video 

3D Face Reconstruction from a Single Image using a Single Reference Face Shape

Ira Kemelmacher-Shlizerman, Ronen Basri, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2010
project page

3D Shape Reconstruction of Mooney Faces

Ira Kemelmacher-Shlizerman, Ronen Basri and Boaz Nadler, IEEE Conf. on Computer Vision Pattern Recognition (CVPR), 2008
paper

A Theory of Locally Low Dimensional Light Transport

Dhruv Mahajan, Ira Kemelmacher-Shlizerman, Ravi Ramamoorthi and Peter Belhumeur, SIGGRAPH 2007
project page

Photometric Stereo with General, Unknown Lighting

Ronen Basri, David W. Jacobs and Ira Kemelmacher, International Journal of Computer Vision (IJCV), 72(3):239-257, 2007
project page

Molding Face Shapes by Example

Ira Kemelmacher and Ronen Basri, In European Conference on Computer Vision (ECCV), 2006
project page

Indexing with Unknown Illumination and Pose

Ira Kemelmacher and Ronen Basri, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2005
project page

LinkTwitterLinkedIn