Develop a user-oriented interface for Splash, our video-mapping software

Intership title

Developing a user-oriented interface for Splash, our video-mapping software

Objective

This internship aims at improving the usability of Splash, our in-house, libre video-mapping software. The current user interface is developer-oriented, and as such very verbose and does not automate much of the process of video-mapping (except for parts like projectors calibration and blending).

The goal would be, using the same technology that is already in place (among which Dear ImGui which has become a standard for user interfaces in the videogame industry), to add a new simplified mode to the existing interface which would hide most of the complexity behind Splash. With the help of Metalab's team, this user interface will also be used to control Splash through the web.

Tasks

  • Participate in the ideation
  • Participate in the maintainance of the software, particularly code related to the user interface
  • Contribute to the source code of open source software
  • Participate in the implementation of a new user interface
  • Document the work and make demos/tutorials of it

Context and software

  • Multidisciplinary team
  • Weekly meetings
  • C++/Python/OpenGL/Linux
  • Gitlab/Git

Development of a plugin for a bibliography management tool to facilitate exploring trends

Title of the internship

Development of a plugin for a bibliography management tool to facilitate exploring trends

Objective

At Metalab and other departments for innovation at the SAT, as in many Research and Development groups, we regularly perform technological watch to keep up to date with recent advances in technologies, in our case enabling the creation of immersive multimedia experiences. We use Zotero to share collections of bibliographical items.

A few Zotero plugins have been created to perform trend exploration tasks on collections using information visualization and text mining, but they are either deprecated (Paper Machines, zotero-voyant-export for Voyant Tools), or in progress.

The goal of this internship is to develop a plugin for Zotero to facilitate exploring trends, for now in collections of PDF files. Restoring deprecated plugins or contributing to in-progress plugins or starting from scratch are viable options.

Tasks

  • Review existing Zotero plugins
  • Understand key techniques from information visualization and text mining to support tasks such as exploring trends
  • Develop a plugin (restore deprecated plugins or contribute to in-progress plugins or start from scratch)
  • (Bonus) Present our joint contributions at an international conference

Context and software

Live crowd animation

Title of the internship

Live crowd animation

Objective

In the context of live virtualization of a symphonic concert, a video capture of the artists and the audience allows to extract movement information from the crowd. The objective of this internship is to prototype a simple approach to model in real time the activity of the concert hall in an online 3d environment.

Tasks

  • Participate in the ideation of the approach.
  • Implement the approach, in collaboration with the research and development teams.
  • Participate in a symphony orchestra capture
  • 3d avatar modeling for performers and public
  • Develop animation strategies
  • Participate and accompany the integration of the animation system in an online 3d environment

Context and software

  • LivePose
  • Data flow mapping
  • Mozilla Hubs through the SAT instance, called Satellite
  • Blender

C++/Python/Linux development for networked audiovisual transmission

Internship title

C++/Python/Linux development for networked audiovisual transmission

Objective

This internship aims to contribute to the development and implementation of Switcher, a software allowing networked low latency multichannel audiovisual transmission. The software is implemented in C++, with a Python wrapper. Different transmission protocols are implemented: SIP/RTP, RTMP, OSC and NDI.

Tasks

  • Participate in the maintenance of the software through the open contribution process
  • Participate in the backlog monitoring under gitlab
  • Contribute to the source code of the open source software
  • Participate in the implementation of Switcher in a multichannel transmission project with Metalab partners
  • Participate in the integration of Switcher in a tool architecture, with a view to deploying services in the Cloud

Context and software

  • Multidisciplinary team
  • Weekly meetings
  • C++/Python/CMake/Linux
  • Network protocols for audiovisual transmission
  • Command line and scripting
  • Network measurement and monitoring tools

Audiodice speaker calibration (filled)

Internship title

Audiodice calibration

Objective

This interships aims to calibrate audiodice speakers, a set of 5 dodecahedron with one speaker per side.

https://vimeo.com/519938720

This type of speaker needs to be calibrated in two ways: - Frequency response - Directivity of the speaker depending it's configuration (single speaker at once, omnidirectional pattern)

From this calibration data, we will be able to create a profile of the audiodice and use it to enhance the sound reproduction. It will also help to create audio spatializer thanks to the directivity knowledge.

Tasks

  • Participate in the measurements
  • Explore existing in-house and other tools for similar tasks
  • Experiment and develop or improve existing in-house tools or pipelines involving other tools
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Linux OS
  • Digital signal processing
  • Python (numpy), bash
  • 3D audio and ambisonics
  • Optional: SuperCollider

3D pose estimation from multiple uncalibrated cameras

Project title/description

3D pose estimation from multiple uncalibrated cameras

More detailed description of the project

LivePose is an open-source pose and action detection software, capable of using multiple cameras to help with creating interactive experiences.

With using multiple cameras comes the need to calibrate them together, to extract 3D coordinates from the 2D pose estimations associated with each camera. This project aims for facilitating the calibration process by using the human body as a calibration tool, hence removing the need to manually place calibration points or any such cumbersome method.

Some preliminary literature describing a human-based calibration:

Expected outcomes

  • Implementation of a sub-module from one of the papers above along with a working prototype (or of similar work found by the beginning of this project)
  • Partial representation of results by training/testing on a subset of a publicly available dataset

Skills required/preferred

  • required: experience with Python
  • required: experience with computer vision
  • required: strong foundation in math
  • if going the deep learning road: experience with machine learning frameworks

Possible mentors

Emmanuel Durand, Christian Frisson

Expected size of project

350 hours

Rating of difficulty

hard

Integration of pose estimation and tracking for multiperson videos

Project title/description

Integration of pose estimation and tracking for multiperson videos

More detailed description of the project

LivePose is an open-source human skeleton and face keypoints detection software. There are many state-of-the-art pose estimation approaches integrated into Livepose.

Often during immersive experiences, the participant moves around the scenes, resulting in frequent occlusions in given timestamps, while fully visible in the neighboring frames. This project aims to combine pose estimation and tracking for improved pose estimation of immersive media.

Our proposed method is inspired by the below studies in this field:

Expected outcomes

  • Complete implementation of one of the papers above (or of similar work appearing or found by the beginning of this project)
  • Partial representation of results by training and testing on one of the publicly available pose datasets
  • (Bonus) Adaptation components for a custom dataset (e.g., custom dataloader, etc)

Skills required/preferred

  • required: experience with Python
  • required: experience with machine learning frameworks (Pytorch preferred)
  • required: understanding of the fundamentals of deep learning
  • preferred: understanding of the fundamentals of computer vision

Possible mentors

Farzaneh Askari, Emmanuel Durand, Christian Frisson

Expected size of project

350 hours

Rating of difficulty

hard

Create a WebUI for data mapping and software control in embedded systems (filled)

Project title/description

Create a WebUI for data mapping and software control in embedded systems

More detailed description of the project

SAT tools have been used for artists and in both artistic residencies and for independent new-media art projects. The Metalab team constantly works to make the deployment, integration, and use of these tools easier for artists. This project consists in creating a WebUI for integrating (mapping) and remotely launching SAT tools in embedded systems based on the Raspberry Pi and, in some use cases, the Jetson Nano.

The applications include setup, maintenance, and control of art installations, sound spatialization, and music performances.

Expected outcomes

  • WebUI capable of launching and control basic functionality of the following software:

Skills required/preferred

  • required: basic knowledge of networking protocols, specially OpenSoundControl (OSC)
  • required: passing knowledge of or willingness to learn Python and C++ for adaptations of the embeddding backend
  • preferred: willingness to learn NodeJS and JavaScript for the frontend (there is some flexibility on the choice of the WebUI frontend language)

Possible mentors

Edu Meneses, Christian Frisson

Expected size of the project

175 hours

Rating of difficulty

medium

Explorable explanations for teaching digital arts concepts in hybrid telepresence campus

Project title/description

Explorable explanations for teaching digital arts concepts in hybrid telepresence campus settings

More detailed description of the project

Explorable explanations add interativity to documents and figures, and associate illustrations and words to account for diverse abilities in learning complex concepts while playing with parameters. Examples are available here: https://explorabl.es

Started in 2020, in response to the health crisis, the Satellite hub building upon Mozilla Hubs is an immersive 3D social web environment developed at SAT to promote various cultural and artistic contents and to create synergy by interconnecting them, including for teaching in hybrid telepresence campus settings.

We want you to help us support teaching in hybrid telepresence settings by prototyping examplar explorable explanations and embedding these in Satellite / Mozilla Hubs including interactivity.

Expected outcomes

  • Prototype explorable explanations in JavaScript
  • Integrate explorable explanations in Mozilla Hubs used for Satellite by adapting Spoke to defining dynamic assets with interactivity.
  • (Bonus) Present our joint contributions at an international conference.

Skills required/preferred

  • required: experience with JavaScript
  • preferred: understanding of the fundamentals of digital arts technologies, including but not limited to: computer graphics, computer vision, deep learning, digital audio signal processing, haptics

Possible mentors

Christian Frisson

Mentors for this project idea can share their experience with potential GSoC contributors to this project idea in developing explorable explanations for various disciplines related to digital arts:

  • information visualization using Idyll

Expected size of project

350 hours

Rating of difficulty

hard

Replace OpenGL by a multi-API rendering library in Splash

Project title/description

The Splash projection mapping software allows for controlling multiple videoprojectors together to build a single projection. It is able to adapt to virtually any real geometry, as long as the surface is diffuse. Its rendering engine is built using the OpenGL API. It currently runs on platforms capable of runing Linux and handling an OpenGL 4.5 context, and has been tested successfully on x86_64 (with NVIDIA, AMD and Intel graphic cards) and aarch64 (with NVIDIA graphic cards)

However to be able to a) optimize it further and b) support more platforms (for example Raspberry Pi), it would be interesting to support more graphics API. To do this it is envisionned to replace direct use of OpenGL with an intermediate, multi-API rendering library. For now bgfx is considered.

More detailed description of the project

For now everything related to graphic rendering is done through direct calls to OpenGL, which means that OpenGL is quite intertwined with the source code of Splash. A first step could be to isolate a bit more the OpenGL calls, which would then facilitate replacing these calls with calls to the multi-API rendering library.

Expected outcomes

  • cleanup of the Splash rendering code
  • added support for at least the Vulkan API
  • possibility to run Splash on new platforms

Skills required/preferred

  • C++
  • OpenGL, other graphics API would be a plus
  • experience with game/rendering engine programming

Possible mentors

Emmanuel Durand

Expected size of project

350 hours

Rating of difficulty

hard