Meet our project coordinator, Alain Pagani, the deputy director of the Department of Augmented Vision at the German Research Center for Artificial Intelligence. Learn about his purpose and background as an expert in computer vision, image understanding and artificial intelligence (AI). Also, about his role in CORTEX2 and his vision of its positive impact in the field of extended reality (XR) and collaborative telepresence, as well as the project’s main achievements before its first year and its next steps.
“After the world had to massively use video conference tools due to COVID-19 restrictions (…), we developed the idea of a system that combines video conferencing and eXtended Reality to generate cooperative experiences with better quality and higher impact.”
Dive into our interview with him:
Q: Hello Alain. Can you introduce yourself and describe your background in computer vision, image understanding and AI?
A: I am a senior researcher and deputy director of the Augmented Vision department at the German Research Center for Artificial Intelligence (DFKI). I have been working in the field of 3D computer vision, image understanding and scene analysis for over 20 years in different research institutes.
“My goal is to develop systems that can react or take decisions in real-time based on visual input. To this aim, I use different methods of AI, such as sensor fusion, reasoning or machine learning.”
Q: In particular, what motivated you to want to focus on the field of augmented vision?
A: Augmented reality (AR) has always been an exciting playground for computer vision scientists, as it requires a thorough understanding of the real scene.
“Augmenting the real world with virtual elements opens up endless possibilities for interacting with it.”
Today, with the advent of XR, our research focuses more on understanding complex situations or behaviours.
Q: It is a really exciting area that offers opportunities in more and more sectors. About your work at DFKI, what does it consist of, and what do you like the most about it?
A: Our department has more than 50 PhD students working in many aspects of computer vision and sensor analysis. As a deputy director, I supervise students’ work towards excellent research and lead several interdisciplinary projects related to our research field.
“I particularly like to see how cooperation among researchers can lead to innovation and scientific progress. Research is a competitive field, but there is also a lot of collaboration and exchange of ideas.”
Q: It must be incredible to see the results of this collaborative work come to life! Regarding your experience in academic research and industrial projects, can you give us an example of the positive impact of any initiatives you have been involved in?
A: For example, in the LARA project, we have created a system for visualizing hidden underground infrastructures using AR and sensor fusion. This type of technology can prevent much damage and save costs in the building industry.
And in the Eyes of Things project, we have developed the concept of vision for any object in the spirit of the Internet of Things. Our platform allows for running virtually any computer vision algorithm directly at the object level without the need for cloud-based processing. We could demonstrate its usability in many scenarios, including a museum guide and therapeutic toys for hospitalized children.
Q: Sounds amazing! Now, tell us more about our European-funded CORTEX2 project. How would you describe it? What is the vision and mission behind it?
A: The idea of CORTEX2 – COoperative Real-Time experiences with EXtended reality – came after the world had to massively use video conference tools due to COVID-19 restrictions. We realised they were useful only for certain types of meetings, leaving aside situations where more information and interaction is needed. We, therefore, developed the idea of a system that combines video conferencing and XR to generate cooperative experiences with better quality and higher impact. By adding augmented reality, virtual reality and services such as environment reconstruction, virtual assistants and meeting summarization, we make tele-cooperation much more natural and efficient for users.
Q: And what about DFKI’s role in the project?
A: The DFKI team working in CORTEX2 focuses on the aspects linked to vision, AR and AI. We are currently working on algorithms that capture a user’s environment in order to share it with distant participants. This is very exciting since it will allow our platform users to share the same reality even if they are thousands of miles away from each other.
Q: We can’t wait to see this innovative technology in action. What would you say have been our project’s most important achievements so far?
A: We have already developed XR clients for video conferencing with the possibility to dynamically change the virtual part of the scene, depending on the users’ actions. Another great achievement is our instantaneous appearance adapter, which allows any user to participate in a video conference with another appearance of his or her choice.
Q: Looking forward to seeing this groundbreaking XR platform come to life! How do you think CORTEX2 benefit the XR and collaborative telepresence fields?
A: CORTEX2 is truly at a crossroads between these two worlds. Current XR applications do not focus on telecooperation or are limited to sharing a common virtual room. Telepresence today mainly consist in sharing video and audio streams. The advantage CORTEX2 offers is merging these two worlds for better experiences when meeting with distant persons.
“Our ultimate goal is to make people forget about the distance and feel the real presence of other video conference participants in a common space.”
Q: Can you share a specific example of how it will positively impact remote collaboration in real-world scenarios?
A: In CORTEX2, we focus on examples not typically addressed by video conferencing tools today. This is the case, for example, for workers on the field confronted with technical problems linked to specific equipment.
“Through our CORTEX2 platform, a field worker can seek instantaneous help from a colleague, even if this one is away or working on another site. Together, they can discuss the next steps that often imply physical action on the local equipment.”
Thus, some incidents can be solved within minutes, whereas it would have taken several days for an expert to come on-site and look at the issue himself.
Q: And, specifically, how will the project contribute to sustainability and reduced carbon footprint by enabling remote collaboration?
A: First, using telepresence instead of physical travel reduces the ecological cost in terms of gas emissions and energy consumption. Today, sobriety in energy is a must, and CORTEX2 can help avoid unnecessary travel.
In addition, we work on specific video compression algorithms to limit the bandwidth used when using video streams. This technology, based on generative deep learning, goes beyond general-purpose video compression for specific usages and allows for a drastic reduction of the necessary bandwidth and hence energy costs.
Q: This is undoubtedly a fundamental aspect of the project, without which today’s scientific progress is inconceivable. How do you think XR will shape the future of remote work and education?
A: XR is in constant evolution. As the technology becomes more mature, it will be more natural to wear an AR or VR headset and live in a mixture of real and virtual worlds. With its unique capacity to create virtual objects, scenes and environments, XR will make interaction with digital systems extremely natural. This will have a huge impact on work and education, as it will allow us to generate supporting objects, explanations and animations that seem real and easily understandable.
“XR will make interaction with digital systems extremely natural, which will have a huge impact on work and education, as it will allow us to generate supporting objects, explanations and animations that seem real and easily understandable.”
Q: It will be interesting to see how this technology develops its full potential. Lastly, what are the project’s most important milestones and future plans you can share?
A: We are now reaching the first year of the project and will deliver a so-called ‘minimal viable product’ to early demonstrate the capabilities of our platform. This important milestone allows us to present our achievements to potential stakeholders.
“An important next step is launching our first open call for external partners to contribute to developing our XR platform in October this year.”
We will finance 30 external partners to contribute to the development of the evaluation of the platform, for a total of 4 million €.
We are very excited to share our first results in the upcoming weeks! Stay tuned for further progress and developments.
Learn more about Alain Pagani‘s expertise and scientific publications.
Want to learn more about our open calls? Subscribe to our newsletter and follow us on social media (LinkedIn and Twitter) to stay updated on how to participate!