Yunbo WANG

Associate Professor
Artificial Intelligence Institute, School of EIEE
Shanghai Jiao Tong University

Email: yunbow@sjtu.edu.cn
Mail: 5-514, Software Building, 800 Dongchuan RD, Shanghai 200240
Google Scholar


Our group's research lies in the intersection of machine learning and computer vision, especially in vision-based intuitive physics. We want to understand how the real world works through machine learning approaches (e.g., predictive learning, inverse graphics, and world models).

I received my Ph.D. and M.E. degrees from Tsinghua University, advised by Philip S. Yu, Mingsheng Long, and Yangdong Deng, and my B.S. degree from Xi'an Jiaotong University. I also spent time in Josh Tenenbaum's lab at MIT working with Jiajun Wu in probabilistic planning and model-based reinforcement learning.

For prospective graduate students: I appreciate your interest in joining our group! We have 1 open position for Ph.D. students and 1 open position for Master students in Class 2026 in Department of Computer Science.

Preprints

  1. DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning
    Yichen Song, Yunbo Wang*, Xiaokang Yang
    [PDF] [PyTorch Code]

  2. Open-World Reinforcement Learning over Long Short-Term Imagination
    Jiajian Li, Qi Wang, Yunbo Wang*, Xin Jin, Yang Li, Wenjun Zeng, Xiaokang Yang
    [PDF] [Website] [PyTorch Code]

  3. Discovering Message Passing Hierarchies for Mesh-Based Physics Simulation
    Huayu Deng, Xiangming Zhu, Yunbo Wang*, Xiaokang Yang
    [PDF]

  4. Learning Augmentation Policies from A Model Zoo for Time Series Forecasting
    Haochen Yuan, Xuelin Li, Yunbo Wang*, Xiaokang Yang
    [PDF]

  5. Dynamic Scene Understanding through Object-Centric Voxelization and Neural Rendering
    Yanpeng Zhao, Yiwei Hao, Siyu Gao, Yunbo Wang*, Xiaokang Yang
    [PDF] [Website] [PyTorch Code]

  6. GS-Phong: Meta-Learned 3D Gaussians for Relightable Novel View Synthesis
    Yumeng He, Yunbo Wang*, Xiaokang Yang
    [PDF] [PyTorch Code]

  7. Predictive Experience Replay for Continual Visual Control and Forecasting
    Wendong Zhang, Geng Chen, Xiangming Zhu, Siyu Gao, Yunbo Wang*, Xiaokang Yang
    [PDF] [PyTorch Code]

Publications

  1. Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
    Qi Wang*, Junming Yang*, Yunbo Wang*, Xin Jin, Wenjun Zeng, Xiaokang Yang
    NeurIPS 2024 [PDF] [Website] [PyTorch Code]

  2. Model-Based Reinforcement Learning with Multi-Task Offline Pretraining
    Minting Pan*, Yitao Zheng*, Yunbo Wang*, Xiaokang Yang
    ECML 2024 [PDF] [PyTorch Code]

  3. Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video
    Xiangming Zhu*, Huayu Deng*, Haochen Yuan*, Yunbo Wang*, Xiaokang Yang
    ICLR 2024 [PDF] [Website] [PyTorch Code]

  4. DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization
    Yanpeng Zhao*, Siyu Gao*, Yunbo Wang*, Xiaokang Yang
    ICLR 2024 [PDF] [Website] [PyTorch Code]

  5. Model-Based Reinforcement Learning with Isolated Imaginations
    Minting Pan, Xiangming Zhu, Yitao Zheng, Yunbo Wang*, Xiaokang Yang
    TPAMI 2023 [PDF] [PyTorch Code]

  6. ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning
    Zhiyu Yao*, Yunbo Wang*, Haixu Wu, Jianmin Wang, Mingsheng Long*
    TPAMI 2023 [PDF] [PyTorch Code]

  7. Improving Masked Autoencoders by Learning Where to Mask
    Haijian Chen*, Wendong Zhang*, Yunbo Wang*, Xiaokang Yang*
    PRCV 2023 (Best Student Paper Award Nomination) [PDF] [PyTorch Code]

  8. Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid
    Wendong Zhang, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    Pattern Recognition 2023 [PDF] [PyTorch Code]

  9. StockFormer: Learning Hybrid Trading Machines with Predictive Coding
    Siyu Gao, Yunbo Wang*, Xiaokang Yang
    IJCAI 2023 [PDF] [PyTorch Code]

  10. Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
    Minting Pan*, Xiangming Zhu*, Yunbo Wang*, Xiaokang Yang
    NeurIPS 2022 Spotlight [PDF] [Website] [PyTorch Code]

  11. Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation
    Shanyan Guan, Jingwei Xu, Michelle Z. He, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    TPAMI 2022 [PDF] [Website] [PyTorch Code]

  12. NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields
    Shanyan Guan, Huayu Deng, Yunbo Wang*, Xiaokang Yang
    ICML 2022 Spotlight [PDF] [Slides] [Website] [PyTorch Code]

  13. Continual Predictive Learning from Videos
    Geng Chen*, Wendong Zhang*, Han Lu, Siyu Gao, Yunbo Wang*, Mingsheng Long, Xiaokang Yang
    CVPR 2022 Oral [PDF] [Website] [PyTorch Code]

  14. PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
    Yunbo Wang*, Haixu Wu*, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, Mingsheng Long*
    TPAMI 2022 (ESI - Top 1% highly cited) [PDF] [PyTorch Code]

  15. Learning Transferable Features for Point Cloud Detection via 3D Contrastive Co-training
    Yihan Zeng*, Chunwei Wang*, Yunbo Wang, Hang Xu, Chaoqiang Ye, Zhen Yang, Chao Ma
    NeurIPS 2021 [PDF] [Supplement]

  16. VideoDG: Generalizing Temporal Relations in Videos to Novel Domains
    Zhiyu Yao*, Yunbo Wang*, Jianmin Wang, Philip S. Yu, Mingsheng Long*
    TPAMI 2021 [PDF] [PyTorch Code]

  17. Context-Aware Image Inpainting with Learned Semantic Priors
    Wendong Zhang, Junwei Zhu, Ying Tai, Yunbo Wang*, Wenqing Chu, Bingbing Ni, Chengjie Wang, Xiaokang Yang
    IJCAI 2021 [PDF] [PyTorch Code]

  18. Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
    Shanyan Guan*, Jingwei Xu*, Yunbo Wang*, Bingbing Ni, Xiaokang Yang
    CVPR 2021 [PDF] [Website] [PyTorch Code]

  19. MetaSets: Meta-Learning on Point Sets for Generalizable Representations
    Chao Huang*, Zhangjie Cao*, Yunbo Wang*, Jianmin Wang, Mingsheng Long*
    CVPR 2021 [PDF] [PyTorch Code]

  20. Deep Predictive Learning: Problems and Methods
    PhD Dissertation (in Chinese) [PDF]
    Outstanding Doctoral Dissertation Award, China Computer Federation (CCF), 2020

  21. Unsupervised Transfer Learning for Spatiotemporal Predictive Networks
    Zhiyu Yao*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang
    ICML 2020 [PDF] [PyTorch Code]

  22. DualSMC: Tunneling Differentiable Filtering and Planning under Continuous POMDPs
    Yunbo Wang*, Bo Liu*, Jiajun Wu, Yuke Zhu, Simon S. Du, Li Fei-Fei, Joshua B. Tenenbaum
    IJCAI 2020 [PDF] [Website] [PyTorch Code]

  23. Probabilistic Video Prediction from Noisy Data with a Posterior Confidence
    Yunbo Wang, Jiajun Wu, Mingsheng Long, Joshua B. Tenenbaum
    CVPR 2020 [PDF] [Website]

  24. Progressive Adversarial Networks for Fine-Grained Domain Adaptation
    Sinan Wang, Xinyang Chen, Yunbo Wang, Mingsheng Long*, Jianmin Wang
    CVPR 2020 [PDF]

  25. A Multi-Player Minimax Game for Diversifying Multi-Discriminators of Generative Adversarial Networks
    Ying Jin, Yunbo Wang, Mingsheng Long*, Jianmin Wang, Philip S. Yu, Jiaguang Sun
    ICME 2020 [PDF]

  26. Multi-Task Learning of Generalizable Representations for Video Action Recognition
    Zhiyu Yao*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang, Philip S. Yu, Jiaguang Sun
    ICME 2020 [PDF]

  27. Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
    Yunbo Wang*, Jianjin Zhang*, Hongyu Zhu, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    CVPR 2019 [PDF] [TensorFlow Code]

  28. Eidetic 3D LSTM: A Model for Video Prediction and Beyond
    Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei
    ICLR 2019 [PDF] [Poster] [TensorFlow Code]

  29. PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
    Yunbo Wang, Zhifeng Gao, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    ICML 2018 [PDF] [TensorFlow Code]

  30. PredCNN: Predictive Learning with Cascade Convolutions
    Ziru Xu*, Yunbo Wang*, Mingsheng Long*, Jianmin Wang
    IJCAI 2018 [PDF]

  31. PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs
    Yunbo Wang, Mingsheng Long*, Jianmin Wang, Zhifeng Gao, Philip S. Yu
    NeurIPS 2017 [PDF] [PyTorch Code]

  32. Spatiotemporal Pyramid Network for Video Action Recognition
    Yunbo Wang, Mingsheng Long*, Jianmin Wang, Philip S. Yu
    CVPR 2017 [PDF]

  33. A Feasibility Study of Ray Tracing on Mobile GPUs
    Yunbo Wang, Chunfeng Liu, Yangdong Deng*
    SIGGRAPH Asia 2014 [PDF]

Teaching

  • JCCX0021: Fundamentals of Artificial Intelligence (Reinforcement Learning), 2022-2025
  • CS0502: Introduction to Computer Science, 2022-2025

Research Group

  • Graduate students: Minting Pan, Huayu Deng (Co-supervised), Xiangming Zhu, Qi Wang (Co-supervised), Haochen Yuan, Yanpeng Zhao, Yumeng He, Yitao Zheng, Yichen Song (Co-supervised), Yuetian Wang (Co-supervised)
  • Alumni: Jingwei Xu (2021), Wendong Zhang (2023), Geng Chen (2023), Shanyan Guan (2024), Siyu Gao (2024), Haijian Chen (2024)

Research Grants

  • PI, Visual Intuitive Learning of Fluid Dynamics, NSFC Original Exploratory Program, 2023-2025
  • PI, Predictive Learning from Spatiotemporal Data, NSFC Young Scientists Fund, 2022-2024