Unfortunately SAT will not participate in the Google Summer of Code (GSoC) 2024.

We hope to participate again in the future!

Project title/description

Adding custom Satellite features to the hubs blender add-on

More detailed description of the project

SAT's metaverse, also known as Satellite, is a WebXR platform powered by Mozilla Hubs designed for multiple purposes including but not limited to facilitating access to virtual experiences and promoting immersive community interactions.

Satellite operates mostly by injection to add custom features to the hubs platform. Since its creation in 2020, numerous features have been introduced to enrich the user's experience. Although some of those features are available on a GUI, they require support from the SAT team to access.

To aid artists in the creative process, Mozilla Hubs developed a blender add-on that allows 3D creators to make use of their features before exporting the models.

The purpose of this project is to add some of Satellite's main custom features to the add-on, more specifically the zone-based features. The zone-based features rely on a bounding box collision in order to emit a specific event such as a teleportation or a room change.

Expected outcomes

• A Hubs add-on fork with at least 2 added features.
• A Satellite room using the newly added blender features.
• Documentation of the process and the features

Skills required/preferred

• required:
  • Python
• preferred:
  • Javascript
  • Familiarity with Blender and WebXR platforms

Possible mentors

• Rochana Fardon
• Manuel Bolduc
• Jean-Michaël Celerier

Expected size of project

350 hours

Rating of difficulty

medium

Project title/description

Adding gestures to puara-gestures, a library for creating high-level gestural descriptors for IoT devices and New Interfaces for Musical Expression.

More detailed description of the project

Self-contained and IoT systems oftem use sensors to get real-world data and provide methods for users to control and interact with systems. This sensor data is often hard to understand and don't properly represent the way we interact with everyday objects, e.g., it is hard to check if a smartphone is being shaked by looking at the plotted IMU data. In this example, it is be easier when the user or IoT designer gets access to a pre-processed a "shake" parameter.

Puara-gestures is currently a small library within the Puara Framework for creating these kind of high-level descripotrs for sensor data. The library currently has only a handful of gestures, but there are other initiatives that created descriptor algorithms used in academic research to create digital musical instruments in the last 15 years. More specifically, we have access to implementations for instruments developed in the context of The McGill Digital Orchestra and the T-Stick.

This project is about porting sensor fusion and gestural descriptors algorithms from Max patches and plugins, including the Digital Orchestra Toolbox (DOT) into the puara-gestures, ensuring they are compatible with SoCs commonly used in IoT such as the ESP32.

Expected outcomes

• Multiple C/C++ functions (or classes), compatible with commonly used IoT SoCs, that expect specific sensor data and output gestural descriptors.
• Updating the standalone version of the library to include the created functions/classes.
• (BONUS) Refactor current functions into classes. The library currently uses an old Arduino-like organization where a single class makes all functions available while sensor data is updated via call in a loop. The ideal refactoring include create classes for each gesture and ensure both sensor imput and gestural output are thead safe.

Skills required/preferred

• required:
  • C/C++
  • Max
• preferred:
  • PureData
  • PlatformIO

Possible mentors

• Edu Meneses
• Jean-Michaël Celerier
• Joseph Malloch

Expected size of project

350 hours

Rating of difficulty

easy

Project title/description

Integration of YOLOv8 backend in LivePose

More detailed description of the project

LivePose is a command line tool which tracks people skeletons from a RGB or grayscale video feed (live or not), applies various filters on them (for detection, selection, improving the data, etc) and sends the results through the network (OSC and Websocket are currently supported).

Some preliminary work has been done regarding the integration of object detection backends such as YOLOv8 in LivePose (for instance, see the proposed YOLOv5 backend integration).

With this project, we hope to evaluate the latency and the accuracy of multiple object detection in a live video processing framework.

Expected outcomes

• A LivePose fork with the proposed YOLOv8 backend integration
• Some demonstration of the integration of the different YOLOv8 computer vision tasks in LivePose (detect, classify, track, ...)
• Documentation of the integration process + experimentations

Skills required/preferred

• Some familiarity with Python (or a will to learn!)
• Some familiarity with AI frameworks in computer vision (or a will to learn!)
• Documentation skills

Possible mentors

• Manuel Bolduc
• Rochana Fardon
• Sarah Al Mamoun

Expected size of project

350 hours

Rating of difficulty

medium

Project title/description

A cross-platform API for GPU texture sharing

More detailed description of the project

Every desktop operating system provides a way to share GPU textures between multiple processes. This is especially useful in the media arts field: a software can be specialized in particles rendering, another in video mapping, and both can then be combined at run-time with no CPU performance cost for inter-process sharing. On Windows, Spout is the standard way to use these OS APIs and share a texture across applications ; on macOS, it is Syphon, and on Linux, this is achievable through PipeWire which wraps DMA-BUF. This means that cross-platform applications wanting to provide texture sharing have to implement multiple different backends, each matching the different platform APIs which leads to a lot of code duplication. This project is about providing a zero-cost cross-platform abstraction over these useful APIs, which would allow software authors to interoperate with Spout / Syphon / PipeWire without having to duplicate shared abstraction code, in particular by extracting the existing code from ossia score and wrapping it in a separate library.

Expected outcomes

• An open-source library which allows to share a GPU texture across applications.
• The library should be able to import / export the texture multiple graphics APIs, for instance OpenGL, Vulkan, Metal, D3D.
• The open-source media arts ecosystem will become more cross-platform-friendly :-)

Skills required/preferred

• C++20
• Graphics programming experience in at least one modern GPU API.
• Cross-platform experience

Possible mentors

• Jean-Michaël Celerier
• Manuel Bolduc

Expected size of project

350 hours

Rating of difficulty

medium

Project title/description

Towards a full-featured embedded media sequencer

More detailed description of the project

Embedded microcontrollers can be extremely powerful tools in the right hands, but usually require advanced programming capabilities. The ossia and Puara Puara/MPU projects allow media artists to create installations, performances, devices based on embedded systems, ranging from Linux-compatible such as Raspberry Pi, to OS-less microcontrollers such as ESP32 or Raspberry Pi Pico. In particular, ossia score, a media sequencer, allows exporting scores made in it into C++ programs which will run directly on embedded microcontrollers. These scores are composed of processing nodes scheduled in time: automations, sound file players, mappings, audio effects, small scripts, etc. This project is about porting some of these nodes from ossia so that they can run directly on microcontrollers: for instance, to enable playing soundfiles from an SD Card in an ESP32 and not be limited to basic GPIO & LED control or network messages. In practice, this involves rewriting the dataflow nodes implementation to remove dependencies to the existing graph system, by porting them to Avendish as this will make them automatically useable in an embedded context.

Expected outcomes

• Multiple objects of ossia ported so that they can run on OS-less systems, targetting for instance the Arduino platform.
• ossia should be able to export code for these objects through the pre-existing infrastructure.
• In the end, this would allow to write a score that for instance plays a sound file when a GPIO changes state on a desktop computer, and then compile that score to ESP32 without the artist having to write a single line of code.

Skills required/preferred

• C++20
• Qt
• Embedded experience: Arduino / ESP-IDF
• (Bonus) Knowledge of a media art tool: ossia, Max/MSP, PureData, TouchDesigner, ...

Possible mentors

• Jean-Michaël Celerier
• Edu Meneses

Expected size of project

350 hours

Rating of difficulty

medium

SAT is proud to announce we applied for the Google Summer of Code (GSoC) 2024 as a mentor organization.

We will update this website with information as soon as Google publishes the list of accepted mentoring organizations.

About Google Summer of Code

Google Summer of Code (g.co/gsoc) is Google's mentorship program for bringing new contributors into open-source communities. It's happening again for the 19th year in 2023! Over 18,000 developers from 112 countries have participated.

Google Summer of Code is a unique program where new contributors to open source, ages 18 and over, are paired with a mentor to introduce them to the open source community and provide guidance while they work on a real-world open source project over the summer. Projects cover a wide range of fields including: Cloud, Operating Systems, Graphics, Medicine, Programming Languages, Robotics, Science, Security and many more. GSoC Contributors do earn a stipend to work on their medium (~175 hours) or large (~350 hours) projects. This is not an internship but provides an invaluable experience and allows you to be part of an amazing community!

GSoC is a highly competitive program, so don't wait to the last minute to prepare! GSoC Contributors should reach out to the organizations that interest them once organizations are announced on February 22, 2024. Potential GSoC Contributors can apply at g.co/gsoc.

Got questions? Email gsoc-support@google.com.

About The Society for Arts and Technology [SAT]

Founded in 1996, the Society for Arts and Technology [SAT] is a non-profit organization dedicated to digital culture. With its triple mission as a center for the arts, training and research, the SAT is a gathering space for diverse talent, curiosity, and knowledge. It is recognized internationally for its active, leading role in developing technologies for immersive creation, mixed realities and telepresence. The Society for Arts and Technology is a place of collective learning that holds the promise of exploring technology to infuse it with more meaning, magic and humanity.

Please consult the following pages to get more details about our application as an organization for GSoC:

• Organization Profile
• Organization Questionnaire
• Program Application

Please visit sat.qc.ca for more information about the SAT.

Note: This blog post contains the details of the work I did during my GSoC period. For a tldr, please check out the work product!

Introduction

As part of my contribution to Google Summer of Code (GSoC) 2023, I was to clean up Splash's use of graphics code, as well as make it work on the Raspberry Pi (Rpi). After a brief period of slow work due to my finals, work began on reviewing existing graphics rendering libraries. One of my mentors, Jean-Michael Celerier, provided a list of possible libraries to check out. Some were unmaintained, some were maintained but for personal use, etc.

Sidequest 1: Testing

It was apparent that we'd need some automated way to be able to quickly iterate and test code, so I went on a little sidequest to get to know Splash more in depth while reviewing graphics libraries. The existing testing infrastructure didn't support testing if the output was correct. Splash did have a sink, which was a way to capture the output of Splash in case you needed it somewhere else. Using this, I set up a basic test case that used reference images and compared them to Splash's output.

Quest 1: Exploring libraries and pi graphics API support

Continuing on with library exploration, the list whittled down to bgfx, Granite, Magnum. All of them are very capable in their own right, but each carried downsides as well.

At this point, I was feeling a bit sidetracked. So I took some time to refocus by seeing which APIs our "target" (Raspberry Pi 4) supported. At the time of writing, the Pi supports Vulkan 1.2, and OpenGL ES 3.1 along with some parts of 3.2. It seemed like supporting OpenGL ES might be a good first step to making Splash run on the Pi, so I homed in on that. Vulkan was a compelling choice, but it was too verbose and different from existing code. And while Vulkan provides speedups for CPU-bound applications, Splash is mostly GPU bound at the moment.

Quest 2: Transformation

The next few weeks were spent in a haze of segfaults, crashes, error callbacks, and countless hours of renderdoc debugging. "Why isn't anything rendering?" was a frequent question on my mind, but I persisted. I started off by fixing the segfaults. These were due to loading an OpenGL ES context, but using OpenGL 4.x calls, which weren't loaded, leaving their function pointers as null. This was a matter of repeatedly launching Splash in gdb, waiting for it to blow up, then figuring out which OpenGL ES call corresponded to the call that crashed. Some parts of the API needed a bit more work than finding the corresponding OpenGL ES call, like memory mapping, but the docs were always there to guide me.

After that, the errors. GLSL for OpenGL ES (GLSLES from now on) requires that you specify floating point precision for either each variable, or for the whole shader (as I understand it right now). It also doesn't support some implicit conversions that normal GLSL allows. The biggest hurdle here was from Splash's shader construction wizardry. Where a bunch of "modules/imports" where built into the program, each associated with some name. The shader code would then search for include <module> and replace it by the corresponding module's code. Basically replacing the preprocessor. This caused the code handed to the driver to be a bit different from the code I was editing and confusion ensued. But this was fixed with some good old debugging aided by gdb's printf's to print the assembled shader code!

Next was uniform initialization. This is used to provide default values for uniforms in the shader source code so you don't have to keep track of it on the CPU. GLSLES does not support uniform initialization. This caused a few shader compilation errors, as well as a couple bugs related to rendering and gamma correction. By this point, though, I had some momentum going. So I gritted my teeth, analyzed error logs, and debugged my way through it.

And there was light! I finally got something to display using OpenGL ES. Despite not being on the Rpi, this was a great first step. It would be a headache to try and fix things on the Rpi without knowing whether the existing code worked or not.

First "proper" render out of the OpenGL ES code, notice the dark void in the background. This is supposed to be the GUI.

Turns out, the GUI was rendering, although very sneakily.

Quest 3: More testing

Next up was testing. My other mentor and the main maintainer of Splash, Emmanuel Durand, provided a list of basic tests so we can figure out if all basic functionality works properly. The three most related to graphics were loading images, loading videos, and blending. Loading images already worked. Emmanuel provided me with a couple of videos in multiple resolutions and formats to make sure the main codepaths were covered.

And of course, Splash segfaulted on the first video I tried. This as usual was solved with a bit of debugging and documentation scouring. Then came blending, note that since the beginning, I tested and fixed stuff locally, then re-tested and further fixed stuff on the Pi. This was a bit of a footgun when it came to blending, as blending requires some features of OpenGL ES 3.2 that the Pi doesn't quite support yet, so I was quite disappointed when blending worked locally, but refused to work on the Pi citing GL_INVALID_ENUM when creating a tesselation shader. This wasn't really a big issue though, since blending is usually used when you have more than one output. The poor Pi couldn't even handle one output well at the moment.

Sidequest 2: Profiling

Another sidequest that I went on was profiling the pi using Tracy. Tracy was honestly magical when it worked. It was a bit of a pain to run on the pi as it doesn't explicitly support OpenGL ES, but using an older version (v0.7.6) and following the solution here worked well.

Turns out, the pi spends most of its time swapping buffers, which most likely involves waiting for the GPU to finish, so we were quite sure the pi was GPU bottlenecked at this point. There was no other way to figure out exactly what was causing this issue, so we focused on making sure the OpenGL ES code wasn't too slow versus the old OpenGL code. I spent a bit of time profiling on my laptop, but I'll spare you the boring details for in issue (#85). Tl;dr: The new code path doesn't significantly affect performance.

Quest 4: Reconciliation

The next step was blending up the new and existing code so that Splash can still be released as one binary instead of an OpenGL version and an OpenGL ES version. This mainly involves choosing an API, initializing it properly since OpenGL and OpenGL ES required slightly different initializations, then redirecting calls so the appropriate code was called for the chosen API.

I looked at a couple of things for inspiration, with the main thing being QT's RHI API. The main idea with this kind of thing is that we want to redirect calls at runtime from some unified interface. This of course screams OOP/Inheritance (or function pointers for C programmers), so I started by drafting out a Renderer class, specializing that for OpenGL and OpenGL ES, moved on to textures which were the biggest plate of spaghetti I've seen for some time (sorry, Manu), but that couldn't stop me at this point. Gpu buffers were next, and things kinda snowballed from there. After some time, I had Splash running under OpenGL 4.5 (anywhere that supports it) and OpenGL ES (for the pi specifically), with automatic API detection between the two.

The final hierarchy for the renderers is summed up by the following diagram:
Renderer class hierarchy

Each class that touched graphics code got separated into a class that has application code, some methods for shared OpenGL/OpenGL ES graphics code, as well as pure virtual methods for parts that were API specific. An extra class would then inherit from this class and implement graphics API specific methods. The following diagram shows a general example, where Foo can be replaced by Texture_Image for example:
Mixed Foo class hierarchy

This was good enough to make splash work with both OpenGL and OpenGL ES, but what about other graphics APIs? Another suboptimal thing was the mixture of application and graphics code. These were the next things to tackle.

Quest 5: Separation

I spent my last couple of weeks working on MR #628, which aimed to separate application code and graphics API specific code such that adding support for a new API would theortically just require maintainers to implement some interface and not have to fiddle with application code. This adds the benefit that changes to graphics code would have less effect on application logic. This keeps the renderer hierarchy from the previous section, but adds another layer between application logic and graphics code to allow for better maintainability. The following diagram illustrates this:

Separated Foo class hierarchy

Now, instead of each renderer creating instances of Foo for its API, it would instead create gfx::<API>::FooGfxImpl, passing those to Foo objects to manage for duration of the program as unique_ptrs. You can also notice that methods shared between OpenGL and OpenGL ES were moved to gfx::GlBaseFooGfxImpl to remove all remaining API specific code present in Foo, while allowing OpenGL / OpenGL ES classes to only override the methods that are needed. If for example your graphics code works for both OpenGL and OpenGL ES, you can forego this base class.

Conclusion and final words

One important thing that I wish to emphasize to other developers: It's essential to deploy on your target device as soon as possible. One of the biggest surprises that I encountered repeatedly is that if the code runs on my laptop with OpenGL ES, it's not guaranteed to run on this version of the Rpi. This is due to a couple of factors: The drivers on my laptop support some OpenGL calls even when running under an OpenGL ES context; another thing is that the pi doesn't 100% support OpenGL ES 3.2, while my laptop supports it. This has led to multiple instances of me being sure the code works well, testing it on the Rpi, only to find it crashing on some OpenGL call or reporting an error.

Overall, my time at GSoC 2023 was quite enjoyable and elevated my skills as a programmer working with C++, OpenGL, and graphics code in general. It also taught me a few precious skills regarding code separation and documentation. As well as teaching me how to collaborate with and report to others remotely in an efficient and concise manner. It was a blast and I sincerely thank my mentors for their invaluable aid during this time!