Yunbo WANG

Associate Professor
Artificial Intelligence Institute, School of EIEE
Shanghai Jiao Tong University

Email: yunbow@sjtu.edu.cn
Mail: 5-514, Software Building, 800 Dongchuan RD, Shanghai 200240
Google Scholar


Our group's research lies in the intersection of machine learning and computer vision, especially in the areas of vision-based intuitive physics. We want to understand how the real world works through machine learning approaches (e.g., predictive learning, inverse graphics, and model-based reinforcement learning).

I received my Ph.D. and M.E. degrees from Tsinghua University, advised by Philip S. Yu, Mingsheng Long, and Yangdong Deng, and my B.S. degree from Xi'an Jiaotong University. I also spent time in Josh Tenenbaum's lab at MIT working with Jiajun Wu in probabilistic planning with uncertainty.

For prospective graduate students: Thank you for your interest in joining our group! We have 1 open position for Ph.D. students and 1 open position for Master students in Class 2025 in Department of Computer Science.

Preprints

  1. Vid2Act: Activate Offline Videos for Visual RL
    Minting Pan*, Yitao Zheng*, Wendong Zhang*, Yunbo Wang*, Xiaokang Yang
    [PDF]

  2. Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
    Qi Wang*, Junming Yang*, Yunbo Wang*, Xin Jin, Wenjun Zeng, Xiaokang Yang
    [PDF]

  3. Predictive Experience Replay for Continual Visual Control and Forecasting
    Wendong Zhang, Geng Chen, Xiangming Zhu, Siyu Gao, Yunbo Wang*, Xiaokang Yang
    [PDF] [PyTorch Code]

Publications

  1. Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video
    Xiangming Zhu*, Huayu Deng*, Haochen Yuan*, Yunbo Wang*, Xiaokang Yang
    ICLR 2024 [PDF] [Website] [PyTorch Code]

  2. DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization
    Yanpeng Zhao*, Siyu Gao*, Yunbo Wang*, Xiaokang Yang
    ICLR 2024 [PDF] [Website] [PyTorch Code]

  3. Model-Based Reinforcement Learning with Isolated Imaginations
    Minting Pan, Xiangming Zhu, Yitao Zheng, Yunbo Wang*, Xiaokang Yang
    TPAMI 2023 [PDF] [PyTorch Code]

  4. ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning
    Zhiyu Yao*, Yunbo Wang*, Haixu Wu, Jianmin Wang, Mingsheng Long*
    TPAMI 2023 [PDF] [PyTorch Code]

  5. Improving Masked Autoencoders by Learning Where to Mask
    Haijian Chen*, Wendong Zhang*, Yunbo Wang*, Xiaokang Yang*
    PRCV 2023 (Best Student Paper Award Nomination) [PDF] [PyTorch Code]

  6. Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid
    Wendong Zhang, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    Pattern Recognition 2023 [PDF] [PyTorch Code]

  7. StockFormer: Learning Hybrid Trading Machines with Predictive Coding
    Siyu Gao, Yunbo Wang*, Xiaokang Yang
    IJCAI 2023 [PDF] [PyTorch Code]

  8. Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
    Minting Pan*, Xiangming Zhu*, Yunbo Wang*, Xiaokang Yang
    NeurIPS 2022 Spotlight [PDF] [Website] [PyTorch Code]

  9. Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation
    Shanyan Guan, Jingwei Xu, Michelle Z. He, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    TPAMI 2022 [PDF] [Website] [PyTorch Code]

  10. NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields
    Shanyan Guan, Huayu Deng, Yunbo Wang*, Xiaokang Yang
    ICML 2022 Spotlight [PDF] [Slides] [Website] [PyTorch Code]

  11. Continual Predictive Learning from Videos
    Geng Chen*, Wendong Zhang*, Han Lu, Siyu Gao, Yunbo Wang*, Mingsheng Long, Xiaokang Yang
    CVPR 2022 Oral [PDF] [Website] [PyTorch Code]

  12. PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
    Yunbo Wang*, Haixu Wu*, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, Mingsheng Long*
    TPAMI 2022 (ESI - Top 1% highly cited) [PDF] [PyTorch Code]

  13. Learning Transferable Features for Point Cloud Detection via 3D Contrastive Co-training
    Yihan Zeng*, Chunwei Wang*, Yunbo Wang, Hang Xu, Chaoqiang Ye, Zhen Yang, Chao Ma
    NeurIPS 2021 [PDF] [Supplement]

  14. VideoDG: Generalizing Temporal Relations in Videos to Novel Domains
    Zhiyu Yao*, Yunbo Wang*, Jianmin Wang, Philip S. Yu, Mingsheng Long*
    TPAMI 2021 [PDF] [PyTorch Code]

  15. Context-Aware Image Inpainting with Learned Semantic Priors
    Wendong Zhang, Junwei Zhu, Ying Tai, Yunbo Wang*, Wenqing Chu, Bingbing Ni, Chengjie Wang, Xiaokang Yang
    IJCAI 2021 [PDF] [PyTorch Code]

  16. Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
    Shanyan Guan*, Jingwei Xu*, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    CVPR 2021 [PDF] [Website] [PyTorch Code]

  17. MetaSets: Meta-Learning on Point Sets for Generalizable Representations
    Chao Huang*, Zhangjie Cao*, Yunbo Wang*, Jianmin Wang, Mingsheng Long*
    CVPR 2021 [PDF] [PyTorch Code]

  18. Deep Predictive Learning: Problems and Methods
    PhD Dissertation (in Chinese) [PDF]
    Outstanding Doctoral Dissertation Award, China Computer Federation (CCF), 2020

  19. Unsupervised Transfer Learning for Spatiotemporal Predictive Networks
    Zhiyu Yao*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang
    ICML 2020 [PDF] [PyTorch Code]

  20. DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
    Yunbo Wang*, Bo Liu*, Jiajun Wu, Yuke Zhu, Simon S. Du, Li Fei-Fei, Joshua B. Tenenbaum
    IJCAI 2020 [PDF] [Website] [PyTorch Code]

  21. Probabilistic Video Prediction from Noisy Data with a Posterior Confidence
    Yunbo Wang, Jiajun Wu, Mingsheng Long, Joshua B. Tenenbaum
    CVPR 2020 [PDF] [Website]

  22. Progressive Adversarial Networks for Fine-Grained Domain Adaptation
    Sinan Wang, Xinyang Chen, Yunbo Wang, Mingsheng Long*, Jianmin Wang
    CVPR 2020 [PDF]

  23. A Multi-Player Minimax Game for Diversifying Multi-Discriminators of Generative Adversarial Networks
    Ying Jin, Yunbo Wang, Mingsheng Long*, Jianmin Wang, Philip S. Yu, Jiaguang Sun
    ICME 2020 [PDF]

  24. Multi-Task Learning of Generalizable Representations for Video Action Recognition
    Zhiyu Yao*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang, Philip S. Yu, Jiaguang Sun
    ICME 2020 [PDF]

  25. Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
    Yunbo Wang*, Jianjin Zhang*, Hongyu Zhu, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    CVPR 2019 [PDF] [TensorFlow Code]

  26. Eidetic 3D LSTM: A Model for Video Prediction and Beyond
    Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei
    ICLR 2019 [PDF] [Poster] [TensorFlow Code]

  27. PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
    Yunbo Wang, Zhifeng Gao, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    ICML 2018 [PDF] [TensorFlow Code]

  28. PredCNN: Predictive Learning with Cascade Convolutions
    Ziru Xu*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang
    IJCAI 2018 [PDF]

  29. PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs
    Yunbo Wang, Mingsheng Long*, Jianmin Wang, Zhifeng Gao, Philip S. Yu
    NeurIPS 2017 [PDF] [PyTorch Code]

  30. Spatiotemporal Pyramid Network for Video Action Recognition
    Yunbo Wang, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    CVPR 2017 [PDF]

  31. A Feasibility Study of Ray Tracing on Mobile GPUs
    Yunbo Wang, Chunfeng Liu, Yangdong Deng*
    SIGGRAPH Asia 2014 [PDF]

Teaching

  • CS0502: Introduction to Computer Science, 2022-
  • JCCX0021: Fundamentals of Artificial Intelligence - Reinforcement Learning, 2022- (Co-instructor)
  • AI3603: Artificial Intelligence: Theory and Applications - Reinforcement Learning, 2023- (Co-instructor)

Research Group

Research Grants

  • PI, Visual Intuitive Learning of Fluid Dynamics, NSFC Original Exploratory Program, 2023-2025
  • PI, Predictive Learning from Spatiotemporal Data, NSFC Young Scientists Fund, 2022-2024