Wrapping up GSoC 2024 with VideoLan

Participating in Google Summer of Code (GSoC) 2024 with VideoLan has been an incredibly rewarding experience. This summer, I had the opportunity to work on a project that aimed to bring the power of WebAssembly (Wasm) to VLC by developing a plugin system using Rust and Wasmer. The goal was to enhance VLC’s extensibility by enabling WebAssembly plugins, potentially setting the stage for replacing the existing Lua scripting system in the future. Throughout this journey, I’ve gained valuable insights into Rust, WebAssembly, and the VLC codebase, all while collaborating with an amazing team of developers. I’m thrilled to share my journey and the progress made during this exciting project.

Project Scope

• Rust Integration

  The initial step was to add Rust support to VLC, ensuring seamless integration within the codebase. This work built upon the foundational Rust work done by Loic, which significantly accelerated progress. Without Loic's initial contributions, it would have taken considerably more time to reach the level of integration achieved. This groundwork was essential for the subsequent development of the Wasm Plugin Manager.

• Wasm Plugin Manager Development

  The core focus was on creating a manager capable of loading and handling WebAssembly plugins. The aim was to replicate and eventually extend the Lua scripting functionality, allowing users to develop plugins in any language that compiles to WebAssembly.

  WebAssembly (Wasm) defines a portable binary-code format and a corresponding text format for executable programs as well as software interfaces for facilitating interactions between such programs and their host environment.

• Interface and Extension Module Implementation

  To manage Wasm plugins, both an interface module and an extension module were necessary. The interface module provided the necessary hooks for Wasm plugins to interact with VLC. The extension module, on the other hand, was required to handle the specifics of the extension lifecycle and integration. This involved identifying the required functions and capabilities for these modules, using the Lua extension launcher as a reference. The approach was to start with a minimal, testable set of features and expand from there.

• Testing and Iteration

  Rigorous testing was crucial to ensure stability and functionality. The process began with simple tests using small WebAssembly (WASM) files, which allowed us to verify that each feature worked as intended within VLC. As the project progressed, the scope of testing expanded to cover more complex scenarios and edge cases to ensure comprehensive validation of all features.

Merge Requests

Several key merge requests (MRs) were made to enhance VLC’s functionality and integrate Rust with WebAssembly (Wasm). This section provides an overview of these MRs, including those that have been accepted and those still in progress. MRs requiring further details will be discussed in depth later.

1. extensions_manager: Introduce typed callbacks for extensions_manager

• View MR Details

Status: Merged

This MR introduced typed callbacks as an alternative to the variadic argument pf_control.

Rationale:

The goal of introducing typed callbacks is to have a safer interface to plug into, moving away from the complexities surrounding va_list. The current approach with va_list is prone to errors due to compiler-specific implementations, limited support in safer languages, and the need for va_arg everywhere. Typed callbacks allow programming languages without va_list support to still implement these callbacks.

2. [GSoC 2024] Add WebAssembly Plugin Support and Integrate with Latest Rust Support Changes

• View MR Details

Status: Draft

This MR introduces WebAssembly Plugin System support using Wasmer in Rust. The Rust work is derived from Loic's rust-for-vlc branch.

Detailed Changes:

1. WebAssembly Plugin Implementation:
   • Added core functionality for WebAssembly plugins using Wasmer.
   • Integrated WebAssembly plugin loading, execution processes, and interactions.
2. Rust Support Integration:
   • Merged the latest master branch changes, including Loic's commits for Rust support.

Motivation:

• Extend the capabilities of our system by incorporating WebAssembly plugins.
• Ensure seamless integration of the new WebAssembly functionality with Rust support enhancements.

3. [GSoC 2024] extensions: Add Locking and Access Functions for Managing Extension Array

• View MR Details

Status: Draft

This MR introduces new functions for synchronization and access control in the extension manager. The updates address:

• Ownership Flexibility: When the extension manager is instantiated from C, it directly owns the extensions. If instantiated from Rust or another language, that language retains ownership, which can result in the extension array being null. This flexibility allows Rust code to manage and own the extensions while ensuring compatibility with existing C code.

• Synchronization and Access Control: The new functions provide necessary synchronization and access control mechanisms to manage the extension array consistently, regardless of the instantiating language.

4. macosx: Fix potential deadlock in VLCExtensionsManager

• View MR Details

Status: Merged

Ensured mutex is unlocked if the extension index exceeds the array size to prevent deadlock.

5. rust: cargo: Fix Build Command Error

• View MR Details

Status: Merged

Fixed a build issue related to the cargo command.

6. rust: Add vlcrs-core Crate and Abstractions for vlc_object_t

• View MR Details

Status: Pending

This Merge Request adds the vlcrs-core crate, which helps create safe ways to work with VLC's C functions and definitions. This crate makes it easier to use VLC features in Rust without directly dealing with unsafe C code.

It also includes Rust abstractions for vlc_object_t, making it simpler and safer to handle VLC objects in Rust.

Technical Detail

In this section, I'll dive into the technical aspects of the project and how various challenges were addressed.

The work began by branching from the master branch, but since the Rust integration hadn't yet been merged into the master, I needed to rebase Loic's Rust work onto my new branch. This was one of the reasons why the Merge Request included both the WebAssembly work and the Rust integration.

At this stage, a key decision revolved around how to generate the FFI bindings to C/C++. We had two options:

1. Use bindgen: This tool automatically generates the Rust FFI bindings.
2. Manual Binding: Writing the bindings manually to maintain more control over the integration.

After extensive discussions, it was decided that manually written bindings would be more beneficial in the long run. This approach, although more labor-intensive upfront, offered greater flexibility and precision, ensuring that the integration would be robust and adaptable as the project progresses.

However, writing everything from scratch was not necessary. We leveraged bindgen to generate initial code and then refined it manually as needed. This hybrid approach streamlined the development process, making it easier to create and maintain the FFI bindings. For this purpose, we utilized the vlcrs-sys-generator, which facilitated the generation of the required code pieces, allowing us to focus on fine-tuning and integrating the bindings effectively.

Interface and Extension Submodule

The first task was to create the Interface and Extension modules. Interfaces are how you interact with the VLC media player. Creating the Interface module was relatively straightforward at this stage, as there wasn't much code to write. We needed to develop the Rust FFI for the interface and wrap it in a way that ensured safety in Rust. The implementation details can be found here. With this in place, defining an interface module in Rust became a straightforward task.

use vlcrs_core::error::Result;
use vlcrs_submodules::interface::InterfaceModuleLoader;
use vlcrs_macros::module;
use vlcrs_submodules::{interface::{InterfaceCapability, ThisInterfaceThread}, ModuleArgs};

use vlcrs_messages::{Logger, debug};

pub struct Wasm;

impl InterfaceCapability for Wasm {
    fn open<'a> (
        _this_interface: ThisInterfaceThread<'a>,
        logger: &'a mut Logger,
        _args: &mut ModuleArgs,
    ) -> Result<()> {

        debug!(logger, "Wasm interface module loaded");

        Ok(())
    }
}

module! {
    type: Wasm (InterfaceModuleLoader),
    capability: "interface" @ 0,
    category: INTERFACE_MAIN,
    description: "Wasm Interpreter",
    shortname: "Wasm",
    shortcuts: ["wasmintf"],
}

The first major challenge, however, was adding the Extension capability as a submodule of the interface. Unfortunately, the module! macro lacked support for submodules, which required additional development to accommodate this new structure.

The Module stores its information in the ModuleInfo struct, and since submodules are almost like modules, it made sense to include ModuleInfo within the SubmoduleModuleInfo struct. This approach allowed submodules to be parsed using the same logic as modules. However, nested submodules were not permitted, as confirmed by my mentor, so this needed to be handled separately with a clear error message to guide users. The detailed changes required to implement this can be found in this commit.

With this completed, the next step was to include the FFI bindings for the extension module. The logic for this was almost identical to what was done for the interface module, with one key difference: the extension module included control functions that were variadic. This necessitated the introduction of typed callbacks. The merge request related to this work can be found here. With this change merged, the control functions could now be easily defined in Rust.

The Extension Manager submodule can be added using the following syntax:

module! {
    type: Wasm (InterfaceModuleLoader),
    capability: "interface" @ 0,
    category: INTERFACE_MAIN,
    description: "Wasm Interpreter",
    shortname: "Wasm",
    shortcuts: ["wasmintf"],
    submodules: [
        {
            type: WasmExtensionModule (ExtensionModuleLoader),
            capability: "extension" @ 2,
            category: UNKNOWN,
            description: "Wasm Extension",
            shortcuts: ["wasmextension"],
        }
    ]
}

Given the current architecture of the VLC codebase, it's only possible to retrieve a single extensions manager via module_need. If additional managers are required, module_match would need to be used to monitor multiple managers. Because of this limitation, we needed to ensure that our module had a higher priority than the Lua module to allow for proper testing. This is why the priority for our module is set to 2.

Extension

The Extension struct encapsulates all the relevant information about an extension loaded into the VLC media player. It serves as a Rust wrapper around the extension_t from the C codebase, bringing with it several fields that are essential for managing and interacting with the extension.

Fields

1. sys: Refers to the internal object that handles the specific operations and state of this extension.
2. logger: A pointer to the logger object, facilitating logging within the extension.
3. name: The extension's name.
4. title: The title of the extension, typically displayed in user interfaces.
5. author: The author of the extension.
6. version: The version number of the extension.
7. url: The URL associated with the extension, possibly linking to documentation/website.
8. description: A detailed description of what the extension does.
9. shortdescription: A brief summary of the extension’s purpose.
10. icondata and icon_size: These fields store the icon for the extension and its size, used for visual representation.

Ownership and Lifetime Management

In an ideal scenario, Rust would have full control over the lifetime of the Extension struct and its associated sys field. However, the current implementation of the extensions manager within the VLC codebase presents challenges in achieving this. The existing architecture limits Rust’s ability to fully own and manage the lifecycle of the extension_t.

There has been draft proposal aimed at addressing this issue and making the extension system more future-proof. After several discussions, a more refined approach has been identified, which will be implemented in the near future to better align with Rust's ownership principles. This work is ongoing and will pave the way for a more robust and maintainable extension management system in VLC.

For more detailed information of Extension, you can explore the current implementation here

Extension Manager Module

With the foundational work completed, the development of the Wasm Extension Manager could begin. The WasmExtensionManager struct is central to this module and encapsulates several key fields, including ThisExtensionsManager, which provides the ability to add extensions, logger, a wasmer::Store (which will be discussed later), variables and a playlist.

struct WasmExtensionManager<'a> {
    extension_manager: ThisExtensionsManager,
    logger: &'a mut Logger,
    store: Arc<Mutex<Store>>,
    variables: Variables,
    playlist: Playlist,
}

The first step in setting up the Extension Manager involves scanning a specified location for Wasm extensions. This is accomplished using the vlcwasm_scripts_batch_execute function, which lists Wasm scripts from the extension path and executes a callback function provided as an argument. The scan_callback function processes each script path, reads the Wasm extension description, sets up the extension, and appends it to the list of available extensions. It also assigns the sys field, which is of type WasmExtension.

WasmExtension Struct

The WasmExtension struct represents an individual Wasm extension and contains several fields crucial for its operation:

struct WasmExtension {
    extension: Extension,

    store: Arc<Mutex<Store>>,
    instance: Instance,

    capabilities: Capabilities,

    thread_handle: Option<thread::JoinHandle<Result<()>>>,

    tx_command: mpsc::Sender<Command>,
    rx_command: mpsc::Receiver<Command>,

    state: Mutex<WasmExtensionState>,
    thread_running: bool,
}

Field Breakdown

1. Extension: This field holds the Extension object, which contains detailed information about the extension.
2. Store: Represents all global state that can be manipulated by WebAssembly programs.
3. Instance: A WebAssembly Instance is a stateful, executable instance of a WebAssembly Module. An instance contains all the exported WebAssembly functions, memories, tables, and globals, allowing interaction with the WebAssembly code. This is one of the most critical components when working with Wasm files.
4. capabilities: This field defines various capabilities of the extension, such as HAS_MENU, TRIGGER_ONLY, and others. These capabilities are represented using bitflags, which are defined as follows:
   rust

   bitflags! {
   #[derive(Default, Debug)]
   struct Capabilities: u32 {
       const HAS_MENU = 1 << 0;
       const TRIGGER_ONLY = 1 << 1;
       const INPUT_LISTENER = 1 << 2;
       const META_LISTENER = 1 << 3;
       const PLAYING_LISTENER = 1 << 4;
   }}
   

5. thread_handle: Stores the thread handle returned by thread::spawn. This is used to ensure that the thread has completed its work. Currently, the mechanism for dropping WasmExtension is not yet implemented due to pending work related to the extension's lifetime.
6. tx_command and rx_command: These channels facilitate message passing between the main thread and the extension thread.
7. WasmExtensionState: Describes the current state of the extension (e.g., Activated, Deactivated, Exiting).
8. thread_running: Indicates whether the thread has already been spawned. If the thread is already running, re-activation code is executed instead of spawning a new thread.

Reading Extension Description from Wasm File

In the Wasm extension system, the extension description provides essential metadata about the extension. This metadata includes details like the title, version, author, and descriptions. Here's a simple example of how an extension description can be defined in Rust:

extension::Description {
    title: "test".to_string(),
    version: "0.0.1".to_string(),
    author: "VideoLAN".to_string(),
    shortdesc: "Test example".to_string(),
    description: "Test description".to_string(),
    capabilities: vec!["menu".to_string(), "input-listener".to_string()],
}

Binding Generation with wai-bindgen-rust

To facilitate communication between the Wasm guest and the host, we use wai-bindgen-rust. This crate generates the necessary bindings, allowing the host to interact with the guest's functions.

Internally, the code above is called by a function named __wai_bindgen_extension_descriptor. This function is defined as follows:

#[export_name = "descriptor"]
unsafe extern "C" fn __wai_bindgen_extension_descriptor() -> i32 {
    // code
}

Memory Management between Host and Guest

One of the challenges of working with Wasm is that the guest (Wasm) and the host (extension manager module) do not share the same memory space. As a result, the address returned by __wai_bindgen_extension_descriptor is in the Wasm memory space, which the host cannot directly access.

To overcome this, we use wasmer::Memory, which provides a wasmer::MemoryView for reading and writing data between the host and guest memory spaces.

Reading the Extension Description

The functions for interacting with Wasm memory have been defined here. These functions enable us to read the extension description from the Wasm memory by accessing the specific memory location returned by __wai_bindgen_extension_descriptor.

Using these functions, the extension manager module can retrieve and utilize the extension description as demonstrated here. This allows the extension manager to understand and manage each Wasm extension's metadata effectively.

Extension Manager Control

With the setup of the Extension Manager module complete, the next step involves implementing the control functions necessary for managing extensions. These functions allow for the activation, deactivation, and general control of the extensions. The key control functions are:

1. activate: Activates a specific extension when invoked by the manager.
2. deactivate: Deactivates the extension.
3. is_activated: Checks if the extension is currently activated.
4. has_menu: Determines whether the extension provides a menu.
5. get_menu: Retrieves the extension's menu, if available.
6. trigger_only: Indicates whether the extension can only be triggered without activation.
7. trigger_menu: Triggers a specific entry within the extension's menu.
8. set_input: Sets the input item for the extension.
9. playing_changed: Handles changes to the playing state.
10. meta_changed: Handles metadata changes.

Activating an Extension

The activate function plays a central role in the lifecycle management of extensions. It takes an extension as an argument and performs the following steps:

• Check Activation Status: First, the function checks if the extension is already activated. If it is, no further action is taken.
• Send Activation Command: If the extension is not yet activated, a Command::Activate is sent through the communication channel.
• Set State: The extension's state is updated to WasmExtensionState::Activating.
• Thread Management: The function then checks if the thread responsible for running the extension is active. If the thread is not running, a new thread is spawned using thread::spawn, passing the run_extension_thread function to it. The handle of this thread is stored in the WasmExtension structure, and the thread_running flag is set to true.

Running the Extension Thread

The run_extension_thread function, defined here, is responsible for managing the execution of the extension within a loop. Within this loop, the function listens for commands from the channel and acts accordingly:

• Activate Command: When a Command::Activate is received, the Wasm extension's activation function is executed, and the state is updated to reflect the activation.
• Deactivate Command: Similarly, when a Command::Deactivate is received, the deactivation function is executed, and the extension's state is updated accordingly.

Executing Wasm Functions

To execute a function within the Wasm environment, the execute_function method of the WasmExtension structure is used, as seen here. This method retrieves the target function from the instance.exports and then invokes it. The output from the Wasm function is encapsulated in a Box<[Value]>, representing the return values of the function.

Exposing VLC APIs to WebAssembly

To enable WebAssembly (Wasm) modules to interact with VLC’s core functionality, specific libraries are exposed to the Wasm environment. This is achieved using the register_functions_with_namespace macro, which simplifies the process of registering and managing these APIs. Below is a detailed explanation of the macro, its usage, and the supported libraries.

register_functions_with_namespace Macro

The register_functions_with_namespace macro streamlines the process of exposing multiple functions to Wasm by automating the creation of Exports objects, registering functions, and adding namespaces. Here’s a breakdown of how it operates:

macro_rules! register_functions_with_namespace {
    ($namespace:literal, $store:expr, $env:expr, $import_object:expr, { $( $name:literal => $func:expr ),* $(,)? }) => {{
        use wasmer::{Exports, Function};

        // Create a new Exports object
        let mut exports = Exports::new();

        // Register the functions
        $(
            exports.insert($name, Function::new_typed_with_env($store, $env, $func));
        )*

        // Register the namespace with the Imports object
        $import_object.register_namespace($namespace, exports);
    }};
}

Macro Description

• Purpose: Creates a new Exports object, registers multiple functions, and registers the namespace with the Imports object.
• Parameters:
  • $namespace:literal: The name of the namespace to register.
  • $store:expr: The Store used for creating the functions.
  • $env:expr: The environment (FunctionEnv) passed to the functions.
  • $import_object:expr: The Imports object where the namespace will be registered.
  • { $( $name:literal => $func:expr ),* $(,)? }: List of function names (as string literals) and their corresponding implementations.

Example Usage:

register_functions_with_namespace!(
    "vlc_msg",
    &store,
    &env,
    &mut import_object,
    {
        "msg_dbg" => vlcwasm_msg_dbg,
        "msg_warn" => vlcwasm_msg_warn,
        "msg_err" => vlcwasm_msg_err,
        "msg_info" => vlcwasm_msg_info,
    }
);

In this example, the vlc_msg namespace is created, and functions for various logging levels are registered.

How States Are Saved

State management in Wasm involves using the Env struct, which holds the environment for functions. This struct is encapsulated within wasmer::FunctionEnv, providing an opaque reference to the function environment. The Env struct looks like this:

pub struct Env {
    pub instance: Option<Instance>,
    pub extension: Extension,
    pub variables: Variables,
    pub playlist: Playlist,
}

Functions receive a wasmer::FunctionEnvMut handle, allowing temporary access to and modification of the environment. Here’s an example function demonstrating this:

fn get_msg_and_logger<'a>(env: &'a mut FunctionEnvMut<Env>, ptr: u32, len: i32) -> (String, &'a mut Logger) {
    let (env_data, store) = env.data_and_store_mut();
    let instance = env_data.instance.as_ref().expect("Should have instance");
    let msg = vlcwasm_read_string(&store, &instance, ptr, len as usize).expect("Failed to read string from memory");
    let logger = env_data.extension.get_logger();
    (msg, logger)
}

fn vlcwasm_msg_dbg(mut env: FunctionEnvMut<Env>, ptr: u32, len: i32) {
    let (msg, logger) = get_msg_and_logger(&mut env, ptr, len);
    debug!(logger, "{}", msg);
}

In this example, get_msg_and_logger retrieves a message and logger from the environment, while vlcwasm_msg_dbg uses this information to log a debug message.

Supported Libraries

Currently, the following libraries are exposed to the Wasm environment, each serving a specific purpose:

1. Messages: Provides logging functionality to the Wasm guest. This library includes functions for various logging levels such as debug, warning, error, and information. View Implementation
2. Variables: Manages module parameters and callbacks registered with the object. View Implementation
3. Configuration: Provides access to configuration settings and directory paths like user, system, home, cache, and data. It also allows adding and retrieving configuration values. View Implementation
4. Playlist: Exposes player and playlist functionalities to the Wasm guest. This library includes APIs for interacting with and manipulating the playlist and player. View Implementation

Adding New Libraries

Adding additional libraries to expose VLC APIs can be done by following a similar pattern to the ones outlined above. Each new library should define the appropriate functions and register them using the register_functions_with_namespace macro. For each library, ensure that the corresponding system (sys) files are also added. These files can be found at /src/rust/vlcrs-core/src

Testing WebAssembly Plugin

The code for generating WebAssembly output that serves as a plugin for VLC can be found here. This plugin is intended for testing and demonstrates the full functionality of the WebAssembly integration.

Conclusion

As I conclude my Google Summer of Code (GSoC) 2024 journey with VideoLAN, I am filled with both gratitude and excitement. This summer has been an incredible adventure, allowing me to contribute to VLC by integrating Rust support and developing a plugin system using WebAssembly (Wasm) with Wasmer.

I couldn’t have done this without the incredible guidance from Alexandre Janniaux and the amazing support from the VideoLAN team. Their help has made this project not just a success but a truly enriching experience.

Looking ahead, I’m eager to see how this project evolves and excited to receive feedback from the community. Thank you to everyone who supported me throughout this journey. I look forward to continuing to contribute to VLC and exploring new possibilities in open-source development.