avatar

Rongyu Zhang

Ph.D. Candidate
NJU & PolyU
rongyuzhang@smail.nju.edu.cn


Short Bio

Rongyu is a second-year dual Ph.D. candidate at Nanjing University and The Hong Kong Polytechnic University, where he is co-supervised by Prof. Yuan Du and Prof. Dan Wang. He is now a visiting student at Peking University under the supervision of Prof. Shanghang Zhang. Previously, he received the M.Phil. degree from The Chinese University of Hong Kong, Shenzhen, supervised by Prof. Fangxin Wang. He has published more than 10 papers in top conference journals such as CVPR, AAAI, TMC, and TCSVT, etc, with more than 120 citations. He was recently selected for the First China Association for Science and Technology โ€œYoung Elite Scientist Sponsorship Program (Ph.D.)โ€, 2025-2027.

Research Interests

My broad interests lie in efficient and generalization learning towards open-world multimodal systems, especially for applications of autonomous driving and robotics. Earlier, my work focused on federated learning and neural video delivery.

Education

NEWS ๐Ÿ”ฅ

๐Ÿ“Œ We have several academic intern positions at HMI Lab (Peking University). We actively work on AI4Science and Embodied AI. If you like what we do, don't hesitate to contact me.

[01.2025] ๐Ÿ“š I am selected for the First China Association for Science and Technology โ€œYoung Elite Scientist Sponsorship Program (Ph.D.)โ€, 2025-2027.

[12.2024] ๐ŸŽ‰ One paper BEVUDA++ was accepted by Transaction on Curcuits Systems and Video Technology (CAS-Q1) as first author.

[12.2024] ๐ŸŽ‰ One paper PAT was accepted by AAAI 2025 (CCF-A).

[12.2024] ๐Ÿ† I was named "Outstanding Ph.D. Candidate" by NJU.

[11.2024] ๐Ÿ’ฐ I was offered "Bank of Jiangsu" Scholarship from NJU.

[09.2024] ๐Ÿ’ป The Panasonic Corporation is integrating the VeCAF (MM'24) into its actual business operations.

[08.2024] ๐Ÿ’ผ I joined the Beijing Academy of Artificial Intelligence supervised by Prof. Shanghang Zhang.

[07.2024] ๐ŸŽ‰ One paper VeCAF was accepted by ACM Multimedia 2024 (CCF-A) as first author.

[07.2024] ๐ŸŽ“ I am offered a dual Ph.D. with the Department of Computing at The Hong Kong Polytechnic University supervised by Prof. Dan Wang.

[05.2024] ๐ŸŽ‰ One paper MuPFL was accepted by IEEE Transaction on Mobile Computing (CCF-A) as first author.

[04.2024] ๐Ÿ“š Our project: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation, has been selected as one of the Jiangsu Province Graduate Research and Practical Innovation Project.

[03.2024] ๐Ÿ’ป The Panasonic Corporation is integrating the MoFME (AAAI'24) into its actual business operations.

[01.2024] ๐ŸŽ‰ One paper BEVUDA was accepted by IEEE ICRA 2024 (CCF-B) as first author.

[12.2023] ๐ŸŽ‰ One paper MoFME was accepted by AAAI 2024 (CCF-A) as first author.

[08.2023] ๐Ÿ† We won 2nd place in the SHIFT Challenge 2023 - Continuous Test-time Adaptation for Semantic Segmentation in the challenges of VCL Workshop, ICCV2023.

[06.2023] ๐ŸŽ“ I joined the ISCL lab at Nanjing University and the HMI Lab of the NATIONAL ENGINEERING RESEARCH CENTER OF VISUAL TECHNOLOGY at Peking University as a joint Ph.D. student.

[04.2023] ๐ŸŽ‰ One paper RepCaM was accepted by ACM NOSSDAV 2023 (CCF-B) as first author.

[03.2023] ๐ŸŽ‰ One paper CdFed was accepted by IEEE ICME 2023 (CCF-B) as first author.

[03.2023] ๐ŸŽ‰ One paper FedFHN was accepted by IEEE Network (CAS-Q2) as first author.

[03.2023] ๐ŸŽ‰ One paper FedAB was accepted by IEEE Internet of Things Journal (CAS-Q1).

[03.2023] ๐ŸŽ‰ Two papers BEVSAN and CDCCA were accepted by IEEE CVPR 2023 (CCF-A).

[03.2023] ๐ŸŽ“ I received M.Phil. degree from CUHKSZ.

[09.2022] ๐Ÿ’ผ I joined OPPO research as a Research Intern.

[08.2021] ๐ŸŽ“ I joined INML lab in CUHKSZ as an M.Phil. student.



Selected Publications

  1. TCSVT'25
    BEVUDA++: Geometric-aware Unsupervised Domain Adaptation for Multi-View 3D Object Detection
    Rongyu Zhang, Jiaming Liu, Xiaoqi Li, Xiaowei Chi, Dan Wang, Li Du, Yuan Du, Shanghang Shang
    IEEE Transactions on Circuits and Systems for Video Technology (TCSVT|CAS-Q1), 2025.
    PDF Code

  2. MM'24
    VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness
    Rongyu Zhang*, Zefan Cai*, Huanrui Yang*, Zidong Liu, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Baobao Chang, Yuan Du, Li Du, Shanghang Zhang (* Co-First Authors)
    ACM International Conference on Multimedia (MM|CCF-A), 2024.
    PDF Code

  3. TMC'24
    Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data
    Rongyu Zhang, Yun Chen, Chenrui Wu, Fangxin Wang, Bo Li
    IEEE Transactions on Mobile Computing (TMC|CCF-A), 2024.
    PDF Code

  4. AAAI'24
    Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation
    Rongyu Zhang, Yulin Luo, Jiaming Liu, Huanrui Yang, Zhen Dong, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Yuan Du, Shanghang Zhang
    Proceedings of the AAAI Conference on Artificial Intelligence (AAAI|CCF-A), 2024.
    PDF Code

  5. TMC'25
    Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation
    Rongyu Zhang, Xiaowei Chi, Wenyi Zhang, Guiliang Liu, Dan Wang, Fangxin Wang
    IEEE Transactions on Mobile Computing (TMC|CCF-A), 2025 (under revision).
    PDF Code

  6. TMC'25
    RepCaM++: Exploring Transparent Visual Prompt with Inference-time Re-parameterization for Neural Video Delivery
    Rongyu Zhang, Xize Duan, Jiaming Liu, Li Du, Yuan Du, Dan Wang, Shanghang Shang, Fangxin Wang
    IEEE Transactions on Mobile Computing (TMC|CCF-A), 2025 (under revision).
    PDF Code

Other Publications

  1. AAAI'25
    PAT: Pruning-Aware Tuning for Large Language Models
    Yijiang Liu, Huanrui Yang, Youxin Chen, Rongyu Zhang, Miao Wang, Yuan Du, Li Du
    Proceedings of the AAAI Conference on Artificial Intelligence (AAAI|CCF-A), 2025.
    PDF Code

  2. ICRA'24
    BEVUDA: Multi-geometric Space Alignments for Domain Adaptive BEV 3D Object Detection
    Jiaming Liu*, Rongyu Zhang*, Xiaowei Chi, Xiaoqi Li, Ming Lu, Yandong Guo, Shanghang Zhang (* Co-First Authors)
    IEEE International Conference on Robotics and Automation (ICRA|CCF-B), 2024.
    PDF Code

  3. Network'23
    Optimizing Efficient Personalized Federated Learning with Hypernetworks at Edge
    Rongyu Zhang, Yun Chen, Chenrui Wu, Fangxin Wang, Jiangchuan Liu
    IEEE Network Magazine (Network|CAS-Q2), 2023.
    PDF Code

  4. ICME'23
    Cluster-driven GNN-based Federated Recommendation System with Biased Message Dropout
    Rongyu Zhang*, Yun Chen*, Chenrui Wu, Fangxin Wang (* Co-First Authors)
    IEEE International Conference on Multimedia and Expo (ICME|CCF-B), 2023.
    PDF Code

  5. NOSSDAV'23
    RepCaM: Re-parameterization Content-aware Modulation for Neural Video Delivery
    Rongyu Zhang*, Lixuan Du*, Jiaming Liu*, Congcong Song, Fangxin Wang, Xiaoqi Li, Ming Lu, Yandong Guo, Shanghang Zhang (* Co-First Authors)
    ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV|CCF-B), 2023.
    PDF Code

  6. CVPR'23
    Cloud-Device Collaborative Adaptation to Continual Changing Environments in the Real-world
    Yulu Gan, Mingjie Pan, Rongyu Zhang, Zijian Ling, Lingran Zhao, Jiaming Liu, Shanghang Zhang
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR|CCF-A), 2023.
    PDF Code

  7. CVPR'23
    BEV-SAN: Accurate BEV 3D Object Detection via Slice Attention Networks
    Xiaowei Chi*, Jiaming Liu*, Ming Lu*, Rongyu Zhang, Zhaoqing Wang, Yandong Guo, Shanghang Zhang (* Co-First Authors)
    IEEE Conference on Computer Vision and Pattern Recognition (CVPR|CCF-A), 2023.
    PDF Code

  8. IoTJ'23
    FedAB: Truthful Federated Learning with Auction-based Combinatorial Multi-armed Bandit
    Chenrui Wu, Yifei Zhu, Rongyu Zhang, Yun Chen, Fangxin Wang, Shuguang Cui
    IEEE Internet of Things Journal (IoTJ|CAS-Q1), 2023.
    PDF Code

Preprints

  1. Arxiv'24
    Decomposing the Neural: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation
    Rongyu Zhang*, Aosong Cheng*, Yulin Luo*, Gaole Dai, Huanrui Yang, Jiaming Liu, Ran Xu, Li Du, Yuan Du, Yanbing Jiang, Shanghang Zhang (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  2. Arxiv'24
    Implicit Neural Image Field for Biological Microscopy Image Compression
    Gaole Dai, Rongyu Zhang*, Cheng-Ching Tseng*, Qingpo Wuwu*, Shaokang Wan*, Ming Lu, Tiejun Huang, Yu Zhou, Ali Ata Tuz, Matthias Gunzer, Jianxu Chen, Shanghang Zhang (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  3. Arxiv'24
    Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning
    Yijiang Liu*, Rongyu Zhang*, Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, Shanghang Zhang (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  4. Arxiv'24
    M2Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation
    Xiaowei Chi*, Rongyu Zhang*, Zhengkai Jiang, Yijiang Liu, Yatian Wang, Xingqun Qi, Wenhan Luo, Peng Gao, Shanghang Zhang, Qifeng Liu, Yike Guo (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  5. Arxiv'24
    FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models
    Zhongyu Zhao*, Menghang Dong*, Rongyu Zhang*, Wenzhao Zheng, Yunpeng Zhang, Huanrui Yang, Dalong Du, Kurt Keutzer, Shanghang Zhang (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  6. Arxiv'24
    EVA: An Embodied World Model for Future Video Anticipation
    Xiaowei Chi*, Hengyuan Zhang*, Chun-Kai Fan*, Xingqun Qi, Rongyu Zhang, Anthony Chen, Chi-Min Chan, Wenhan Luo, Wei Xue, Shanghang Zhang, Yike Guo (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

  7. Arxiv'24
    ViML: A Video, Music, Language Unified Dataset for Understanding and Generation
    Xiaowei Chi*, Aosong Chen*, Pengjun Fang*, Yatian Wang*, Zeyue Tian*, Yingqing He, Zhaoyang Liu, Xingqun Qi, Rongyu Zhang, Mengfei Li, Jiahao Pan, Yanbing Jiang, Wei Xue, Wenhan Luo, Qifeng Chen, Shanghang Zhang, Qifeng Liu, Yike Guo (* Co-First Authors)
    Arxiv, 2024.
    PDF Code

Employment

Services

Conference Reviewers

Journal Reviewers

Projects

Awards


Miscellaneous

  1. Family
    I have a beautiful girlfriend, Ziqi Qiao, who is also a Ph.D. student at Peking University. We also own an adorable cat ๐Ÿฑ named QiuQiu. He brings us tremendous fun and happiness.
  2. Friends
    Here are some of my closest academic friends.
    Jiaming Liu, Yulin Luo, Yuan Zhang, and Gaole Dai from PKU; Xiaowei Chi from HKUST; Chenrui Wu from ZJU&SFU; Yili Jin from McGill.
  3. Hobbies
    I am a crazy basketball ๐Ÿ€ & football โšฝ๏ธ fan. I enjoy the games of Kevin Durant ๐Ÿ•ท๏ธ and Kyrie Irving ๐Ÿง™. I also wholeheartedly pledge my allegiance to Chelsea Football Club, KTBFFH! ๐Ÿ’™

Contact

๐ŸŒ Address: Room 2728, Science Building No. 2, Peking University, Beijing, China
๐Ÿ“ฑ Phone: +(86) 188-1305-1303
๐Ÿ“ฎ Email: rongyuzhang@smail.nju.edu.cn
๐Ÿ“ฎ Email (optional): rongyu.zhang@connect.polyu.hk


Webpage credits