The proposed InterViLa research project aims to enhance the visual understanding and
reasoning on human-object interactions in procedural videos by leveraging Vision-Language models and
external Knowledge Graphs. Building on the success of previous deep learning approaches, this project
introduces a novel integration of powerful vision-language foundation models, which provide a flexible
and robust framework for visual and semantic interpretation. By utilizing these models, the system will
be able to link visual data with semantic meaning, improving the recognition and reasoning capabilities
for complex human-object interactions in videos. The project also explores the use of external knowledge
graphs for common-sense reasoning, enabling the system to infer unseen or implicit relationships across
subsequent HOI in procedural videos. A key aspect of this research also include an activity graph
generation approach for modelling the observed complex HOI and a visual question answering (VQA)
mechanism that will enable the performance towards challenging downstream visual tasks, such as HOI
prediction, anomaly detection, video summarization and more. The proposed framework will significantly
advance the state of the art in video understanding and human-robot interaction, with broad implications
for autonomous systems, surveillance, and assistive technologies.