3D pose estimation from multiple uncalibrated cameras

Project title/description

3D pose estimation from multiple uncalibrated cameras

More detailed description of the project

LivePose is an open-source pose and action detection software, capable of using multiple cameras to help with creating interactive experiences.

With using multiple cameras comes the need to calibrate them together, to extract 3D coordinates from the 2D pose estimations associated with each camera. This project aims for facilitating the calibration process by using the human body as a calibration tool, hence removing the need to manually place calibration points or any such cumbersome method.

Some preliminary literature describing a human-based calibration:

Expected outcomes

  • Implementation of a sub-module from one of the papers above along with a working prototype (or of similar work found by the beginning of this project)
  • Partial representation of results by training/testing on a subset of a publicly available dataset

Skills required/preferred

  • required: experience with Python
  • required: experience with computer vision
  • required: strong foundation in math
  • if going the deep learning road: experience with machine learning frameworks

Possible mentors

Emmanuel Durand, Christian Frisson

Expected size of project

350 hours

Rating of difficulty

hard

Add directionality for sources to our audio ray tracing tool (vaRays)

Internship title

Add directionality for sources to our audio ray tracing tool (vaRays)

Objective

Our audio simulation engine, vaRays, uses ray tracing to simulate acoustics in virtual environments. The objective of this internship will be to propose and implement a method that will allow different possible energy distributions for audio sources in vaRays.

Tasks

With the help of the Metalab team:

  • Get acquainted with the vaRays code

  • Experiment with and propose different methods for the implementation of directionality for audio sources

  • Participate in the creation of demonstrations of those implementations

  • Document the work and ensure its reproducibility

Work environment

  • Scrums, code review, etc.

  • C++

  • GitLab, Linux, Free and Open Source Software (FOSS)

  • JIRA / Confluence

Add support of SRT in Switcher, a multichannel low latency streamer (closed)

Project title/description

Switcher is an integration environment, able to interoperate with other software and protocols. Switcher provides low latency streaming of multichannel audio, video and data through IP networks.

Switcher provides managing of several instances of services (called Quiddities). A Quiddity can be created and removed dynamically and can be controlled through property get/set as well as method invocations and information tree monitoring. Switcher provides introspection mechanisms to help write higher level software written in python3 or C++. Switcher can save and load the state of quiddities.

The project consists in writing a new Switcher Quiddity for the support of the SRT protocol. This protocol is regarded as of interest for video communication in the context of artistic telepresence, but also for potential streaming to diffusion platforms, as a replacement to the RTMP protocol. This project will have a positive impact on the Scenes ouvertes (fr) telepresence network of artistic venues in Quebec, Canada.

More detailed description of the project

The projet will consist in writing a Quiddity that handles SRT, with audio and video streams as input. The GStreamer framework is already a strong dependency in Switcher, and already includes an integration of the protocol. The new Quiddity could therefore use the GStreamer implemention for integration. Accordingly, this project does not require strong knowledge of the SRT protocol. The main part of the project will involve C++ interfacing of GStreamer implementation into the Quiddity logic, and writing C++ or python test.

Expected outcomes

  • Add new module (a Quiddity) for SRT streaming in Switcher
  • Add C++ or python test for this Quiddity
  • Demonstration of an audio/video streaming with the new Quiddity

Skills required/preferred

  • C++
  • Ubuntu Linux
  • interest/experience with GStreamer application programming
  • possibly basic Python3

Possible mentors

Nicolas Bouillot, Valentin Laurent

Expected size of project

350 hours

Rating of difficulty

hard

Audio extension for the 3D scene exchange format glTF

Internship title

Definition and implementation of an audio extension for the 3D scene exchange format glTF

Objective

The glTF format is a standardized standard (https://github.com/KhronosGroup/glTF) allowing the exchange of complex 3D scenes including 3D models, animations, cameras, etc. However, this format does not support the audio part. It is therefore necessary to define a dedicated extension to describe the sound aspects of a 3D scene.

Tasks

  • Make a state of the art of audio scene description formats

  • Participate in the definition of the specifications of an audio extension in glTF format

  • Develop an implementation example

  • Add support for the extension in the tools developed/used by the Metalab: EiS, vaRays, Blender

  • Participate in the life of the laboratory: scrums, code review, etc.

  • Document the work and ensure its reproducibility.

Work environment

C++/Python/Linux development for networked audiovisual transmission (closed)

Internship title

C++/Python/Linux development for networked audiovisual transmission

Objective

This internship aims to contribute to the development and implementation of Switcher, a software allowing networked low latency multichannel audiovisual transmission. The software is implemented in C++, with a Python wrapper. Different transmission protocols are implemented: SIP/RTP, RTMP, OSC and NDI.

Tasks

  • Participate in the maintenance of the software through the open contribution process
  • Participate in the backlog monitoring under gitlab
  • Contribute to the source code of the open source software
  • Participate in the implementation of Switcher in a multichannel transmission project with Metalab partners
  • Participate in the integration of Switcher in a tool architecture, with a view to deploying services in the Cloud

Context and software

  • Multidisciplinary team
  • Weekly meetings
  • C++/Python/CMake/Linux
  • Network protocols for audiovisual transmission
  • Command line and scripting
  • Network measurement and monitoring tools

Development of a plugin for a bibliography management tool to facilitate exploring trends (filled)

Title of the internship

Development of a plugin for a bibliography management tool to facilitate exploring trends

Objective

At Metalab and other departments for innovation at the SAT, as in many Research and Development groups, we regularly perform technological watch to keep up to date with recent advances in technologies, in our case enabling the creation of immersive multimedia experiences. We use Zotero to share collections of bibliographical items.

A few Zotero plugins have been created to perform trend exploration tasks on collections using information visualization and text mining, but they are either deprecated (Paper Machines, zotero-voyant-export for Voyant Tools), or in progress.

The goal of this internship is to develop a plugin for Zotero to facilitate exploring trends, for now in collections of PDF files. Restoring deprecated plugins or contributing to in-progress plugins or starting from scratch are viable options.

Tasks

  • Review existing Zotero plugins
  • Understand key techniques from information visualization and text mining to support tasks such as exploring trends
  • Develop a plugin (restore deprecated plugins or contribute to in-progress plugins or start from scratch)
  • (Bonus) Present our joint contributions at an international conference

Context and software

Development of an open-source toolkit for creating haptic effects with ultrasound displays (filled)

Internship title

Development of an open-source toolkit for creating haptic effects with ultrasound displays

Context

Founded in 2002, Metalab is the research laboratory of the Society for Arts and Technology [SAT]. Metalab's mission is twofold:

  1. to stimulate the emergence of innovative immersive experiences,
  2. to make their design accessible to artists and creators of immersion through an ecosystem of free software.

Besides our set of tools for audio- and visual rendering, we are exploring ways to enhance immersive experiences with haptic feedback, to create simulations that stimulate the sense of touch. One first haptic rendering device created at Metalab is the Scalable Haptic Floor that transmits haptic feedback to our bodies through their points of contact with the floor. In addition, ultrasound haptics render air-borne haptic feedback in mid-air, without contact, and without requiring to wear devices [1,2], opening new perspectives for immersion through focused haptic feedback.

Objective

The main objective of this internship is to develop an open-source toolkit for creating haptic feedback with ultrasound displays, reusing recent research works [1,2], and facilitating the authoring of ultrasound haptic effects for artists and creators of immersion.

This toolkit could complement other Metalab tools, for instance using Poire for haptic generation through audio signals as proxy, similarly to the Scalable Haptic Floor, and LivePose for posture or hand detection to support interactive scenarios.

Tasks

  • Review haptic effects that can be generated with ultrasound displays.
  • Develop the open-source toolkit (hardware and software building upon recent research works [1,2]).
  • Create a demo video.
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document our work in a lab notebook and git repositories and ensure its reproducibility.
  • Co-author a publication in the scope of this internship.

Work environment

  • Hardware deployment: Arduino or Raspberry Pi
  • Software development: C++, JavaScript or Python
  • Documentation: BibTeX, LaTeX, Markdown
  • Workflows: Confluence, Gitlab, Jira
  • Ecosystem: Linux, Free and Open Source Software (FOSS)

References

Explorable explanations for teaching digital arts concepts in hybrid telepresence campus

Project title/description

Explorable explanations for teaching digital arts concepts in hybrid telepresence campus settings

More detailed description of the project

Explorable explanations add interativity to documents and figures, and associate illustrations and words to account for diverse abilities in learning complex concepts while playing with parameters. Examples are available here: https://explorabl.es

Started in 2020, in response to the health crisis, the Satellite hub building upon Mozilla Hubs is an immersive 3D social web environment developed at SAT to promote various cultural and artistic contents and to create synergy by interconnecting them, including for teaching in hybrid telepresence campus settings.

We want you to help us support teaching in hybrid telepresence settings by prototyping examplar explorable explanations and embedding these in Satellite / Mozilla Hubs including interactivity.

Expected outcomes

  • Prototype explorable explanations in JavaScript
  • Integrate explorable explanations in Mozilla Hubs used for Satellite by adapting Spoke to defining dynamic assets with interactivity.
  • (Bonus) Present our joint contributions at an international conference.

Skills required/preferred

  • required: experience with JavaScript
  • preferred: understanding of the fundamentals of digital arts technologies, including but not limited to: computer graphics, computer vision, deep learning, digital audio signal processing, haptics

Possible mentors

Christian Frisson

Mentors for this project idea can share their experience with potential GSoC contributors to this project idea in developing explorable explanations for various disciplines related to digital arts:

  • information visualization using Idyll

Expected size of project

350 hours

Rating of difficulty

hard

Integration of pose estimation and tracking for multiperson videos (filled)

Project title/description

Integration of pose estimation and tracking for multiperson videos

More detailed description of the project

LivePose is an open-source human skeleton and face keypoints detection software. There are many state-of-the-art pose estimation approaches integrated into Livepose.

Often during immersive experiences, the participant moves around the scenes, resulting in frequent occlusions in given timestamps, while fully visible in the neighboring frames. This project aims to combine pose estimation and tracking for improved pose estimation of immersive media.

Our proposed method is inspired by the below studies in this field:

Expected outcomes

  • Complete implementation of one of the papers above (or of similar work appearing or found by the beginning of this project)
  • Partial representation of results by training and testing on one of the publicly available pose datasets
  • (Bonus) Adaptation components for a custom dataset (e.g., custom dataloader, etc)

Skills required/preferred

  • required: experience with Python
  • required: experience with machine learning frameworks (Pytorch preferred)
  • required: understanding of the fundamentals of deep learning
  • preferred: understanding of the fundamentals of computer vision

Possible mentors

Farzaneh Askari, Emmanuel Durand, Christian Frisson

Expected size of project

350 hours

Rating of difficulty

hard

IR, Calibration and Visualisation

Internship title

Impulse Response, Calibration and Visualisation of Audio

Objective

This interships aims to continue the development of recipes, approaches and tools around Impulse Response recording and visualisation. The purpose of this work is to identify some acoustic properties of a physical space to help with calibration of audio equipment.

Tasks

  • Participate in the ideation
  • Explore existing in-house and other tools for similar tasks
  • Experiment and develop or improve existing in-house tools or pipelines involving other tools
  • Participate in the development of a demo (video or live)
  • Participate in the life of the laboratory: scrums, code review, etc.
  • Document the work and ensure its reproducibility.

Context and software

  • Linux OS
  • Digital signal processing
  • Python (numpy), bash
  • 3D audio and ambisonics
  • VR/AR
  • Optional: SuperCollider

Contact

Please send your CV to metalab-stage@sat.qc.ca