Animating Street View, SIGGRAPH Asia 2023
Mengyi Shan, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz
We present a system that automatically brings street view imagery to life by automatically populating it with naturally behaving, animated pedestrians and vehicles.
DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion, ICCV 2023
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, Ira Kemelmacher-Shlizerman
DreamPose is a diffusion-based image-to-video synthesis model. Given an input image of a person and pose sequence, DreamPose synthesizes a photorealistic video of the input person following the pose sequence.
HRTF Estimation in the Wild, UIST 2023
Vivek Jayaram, Ira Kemelmacher-Shlizerman, Steve Seitz
Head Related Transfer Functions (HRTFs) play a crucial role in creating immersive spatial audio experiences. However, HRTFs differ significantly from person to person, and traditional methods for estimating personalized HRTFs are expensive, time-consuming, and require specialized equipment. We imagine a world where your personalized HRTF can be determined by capturing data through earbuds in everyday environments.
TryOnDiffusion: A Tale of Two UNets, CVPR 2023
Luyang Zhu, Dawei Yang, Tyler Zhu, Fitsum Reda, William Chan, Chitwan Saharia, Mohammad Norouzi, Ira Kemelmacher-Shlizerman
Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects.
PersonNeRF: Personalized Reconstruction from Photo Collections, CVPR 2023
Chung-Yi Weng, Pratul Srinivasan, Brian Curless, Ira Kemelmacher-Shlizerman
We present PersonNeRF, a method that takes a collection of photos of a subject (e.g., Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance.
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation, CVPR 2022 oral, 3% out of ~8k papers
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, JJ Park, Ira Kemelmacher-Shlizerman
Our method is trained on single-view RGB data only while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape.
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video, CVPR 2022 oral, 3% out of ~8k papers
Chung-Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose.
ClearBuds: Wireless Binaural Earbuds for Learning-based Speech Enhancement, Mobisys 2022 oral. Best demo award runner up.
Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher-Shlizerman, Shwetak Patel, Steven M. Seitz
ClearBuds is a state-of-the-art hardware and software system for real-time speech enhancement. Our neural network runs completely on an iphone, allowing to suppress unwanted noises while taking phone calls on the go. Results show that our wireless earbuds achieve a synchronization error less than 64 $\mu$s and our network has a runtime of 21.4 ms on an accompanying mobile phone.
A Light Stage on Every Desk
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz, ICCV 2021
While watching a YouTube video the monitor is projecting patterns on the face. Our algorithm shows how to leverage the patterns to relight and preserve privacy.
TryOnGAN: Body-Aware Try-On via Layered Interpolation
Kathleen M Lewis, Srivatsan Varadharajan, Ira Kemelmacher-Shlizerman, SIGGRAPH 2021
switch garments and change their size to adjust to new humans via stylegan latent space interpolation.
Real-Time High Resolution Background Matting
Peter Lin*, Andrey Ryabtsev*, Soumyadip Sengupta, Brian Curless, Steve Seitz, Ira Kemelmacher-Shlizerman, CVPR 2021 oral 3% of papers
Best student paper honorable mention; 0.08% of 7093 papers
We achieve HD matting at 60fps, by estimating alpha per frame in a sequence of CNNs from coarse to refined estimation.
Mixed Reality Spatial Computing in a Remote Learning Classroom
John Akers, Joelle Zimmermann, Laura Trutoiu, Brian Schowengerdt, Ira Kemelmacher-Shlizerman, ACM SUI 2020
Best Poster/Demo Honorable Mention
Reconstructing NBA players
Luyang Zhu, Konstantinos Rematas, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman, ECCV 2020 (spotlight)
Key idea: Use synthetic NBA2k data to estimate pose and mesh from photos of real NBA players. project page
Kemelmacher-Shlizerman, Shechtman, Garg and Seitz, SIGGRAPH 2011
Moving Portraits, I. Kemelmacher-Shlizerman, E. Shechtman, R. Garg, S.M. Seitz, Communications of the ACM, Research Highlights, 2014.