Relation understanding in videos
S Zheng, X Chen, S Chen, Q Jin. "Relation understanding in videos." 2019 Proceedings of the 27th ACM International Conference on Multimedia. 2662-2666.
Download CV
Education
- B.Eng. in Computer Science and Engineering, Renmin University of China, China, 2023
- Ph.D. in Computer Science and Engineering, Renmin University of China, China, 2023
Research Interests
- Computer vision and human behavior understanding
- Multimodal understanding with large language models
- Open-world embodied agent learning
Work Experience
- Jul. 2023 - Current: Research Scientist
- Beijing Academy of Artificial Intelligence, Beijing, China
- Duties included: large multimodal model, multi-agent learning
- Apr. 2022 - Oct. 2022: Research Intern
- Microsoft Research Asia, Beijing, China
- Duties included: temporal sentence grounding for long-term videos.
- Nov. 2021 - Apr. 2022: Research Intern
- Beijing Academy of Artificial Intelligence, Beijing, China
- Duties included: multi-lingual language-vision-audio pre-training.
Awards & Honors
- National Scholarship for Ph.D Students.
- 2022 Ranked 3th in CVPR 2022 Ego4D Natural Language Query Challenge.
- 2021 Ranked 3th in NIST TRECVID 2021 Ad-hoc Video Search (AVS) Challenge. (20+ teams)
- 2021 Ranked 4th in CVPR 2021 HOMAGE Scene-graph Generation Challenge.
- 2020 Ranked 2th in ACM MM 2020 Video Relationship Understanding Grand Challenge.
- 2019 Ranked 2nd in ACM MM 2019 Video Relationship Understanding Grand Challenge.
- Best Method Prize in ACM MM 2019 Grand Challenge.
- 2019 First Class Scholarship for Ph.D Students from 2018 to 2021.
- 2015 First Prize in National University Mathematical Modeling Competition of Beijing Area.
Services
- Conference Reviewer for CVPR, ICCV, ECCV, NeurIPS, AAAI, ACM MM, ICME.
- Journal Reviewer for IJCV, TCSVT, TMM, JATS.
Publications
Visual relation detection with multi-level attention
S Zheng, S Chen, Q Jin. "Visual relation detection with multi-level attention." 2019 Proceedings of the 27th ACM International Conference on Multimedia. 121-129.
Skeleton-based interactive graph network for human object interaction detection
S Zheng, S Chen, Q Jin. "Skeleton-based interactive graph network for human object interaction detection." 2020 IEEE International Conference on Multimedia and Expo (ICME). 1-6.
Exploring anchor-based detection for ego4d natural language query
S Zheng, Q Zhang, B Liu, Q Jin, J Fu. "Exploring anchor-based detection for ego4d natural language query." 2022 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop.
VRDFormer: End-to-end video visual relation detection with transformer
S Zheng, S Chen, Q Jin. "VRDFormer: End-to-end video visual relation detection with transformer." 2022 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18836-18846.
Few-shot Action Recognition with Hierarchical Matching and Contrastive Learning
S Zheng, S Chen, Q Jin. "Few-shot Action Recognition with Hierarchical Matching and Contrastive Learning." 2022 European Conference on Computer Vision. 297-313.
Anchor-Based Detection for Natural Language Localization in Ego-Centric Videos
B Liu, S Zheng, J Fu, WH Cheng. "Anchor-Based Detection for Natural Language Localization in Ego-Centric Videos." 2023 IEEE International Conference on Consumer Electronics (ICCE). 01-04.
Accommodating audio modality in CLIP for multimodal processing
L Ruan, A Hu, Y Song, L Zhang, S Zheng, Q Jin. "Accommodating audio modality in CLIP for multimodal processing." 2023 Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence. 7 - 14.
No-frills Temporal Video Grounding: Multi-Scale Neighboring Attention and Zoom-in Boundary Detection
Q Zhang, S Zheng, Q Jin. "No-frills Temporal Video Grounding: Multi-Scale Neighboring Attention and Zoom-in Boundary Detection." 2023 arXiv preprint. arXiv:2307.10567.
Open-Category Human-Object Interaction Pre-Training via Language Modeling Framework
S Zheng, B Xu, Q Jin. "Open-Category Human-Object Interaction Pre-Training via Language Modeling Framework." 2023 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19392-19402.
POV: Prompt-Oriented View-agnostic Learning for Egocentric Hand-Object Interaction in the Multi-view World
B Xu, S Zheng, Q Jin. "POV: Prompt-Oriented View-agnostic Learning for Egocentric Hand-Object Interaction in the Multi-view World." 2023 Proceedings of the 31st ACM International Conference on Multimedia. 2807-2816.
SPAFormer: Sequential 3D Part Assembly with Transformers
B Xu, S Zheng, J Qin. "SPAFormer: Sequential 3D Part Assembly with Transformers" 2023 arXiv preprint. arXiv:2310.08922.
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
S Zheng, J Liu, Y Feng, Z Lu. "Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds." 2023 arXiv preprint. arXiv:2310.13255.
UniCode: Learning a Unified Codebook for Multimodal Large Language Models
S Zheng, B Zhou, Y Feng, Y Wang, Z Lu. "UniCode: Learning a Unified Codebook for Multimodal Large Language Models" 2024 arXiv preprint. arXiv:2403.09072.
EgoNCE++: Do Egocentric Video-Language Models Really Understand Hand-Object Interactions?
B Xu, Z Wang, Y Du, Z Song, S Zheng, Q Jin. "EgoNCE++: Do Egocentric Video-Language Models Really Understand Hand-Object Interactions?" 2024 arXiv preprint. arXiv:2405.17719v1.
LLaMA Rider: Spurring Large Language Models to Explore the Open World
Y Feng, Y Wang, J Liu, S Zheng, Z Lu. "LLaMA Rider: Spurring Large Language Models to Explore the Open World." 2023 arXiv preprint. arXiv:2310.08922.
QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds
Wang, Ye and Mei, Yuting and Zheng, Sipeng and Jin, Qin. "QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds" 2024 arXiv preprint. arXiv:2406.16578.
Languages
Mandarin(Native), English(Fluent)
Skills & Expertise
- Strong interpersonal skills with a good sense of teamwork.
- Programming Skills: Python C/C++ and Html in both Unix and Windows systems.
- Rich experience in large model pre-training.