Neural Audio 2: Audio Transformers

In our previous post, we introduced some of the fundamentals of AI and Machine learning and their relevance in modern audio technology, software and game audio. This article takes a deeper dive into transformers, a popular architecture that has significantly improved various machine learning tasks, including some that are pivotal in audio processing. We aim to highlight how audio transformers could enhance game audio development, crafting experiences that are more immersive and interactive.

If you missed the previous article, catch up on it here.

The Basics of Transformers

Neural Networks form just a piece of the puzzle in the architecture of complex models, spanning a wide spectrum from a single neuron to intricate, multi-layered networks known as Deep Learning. Enter transformers: these advanced neural network architectures excel in managing sequential data such as text, audio, and, to some extent, images. What sets transformers apart from earlier models, which tackled data in a sequential manner—one piece at a time—is their ability to process entire data sequences simultaneously. This capability significantly enhances their efficiency in tasks that require an understanding of the context or relationships within the data [4, 5].

The Components of a Transformer

A typical transformer model consists of two main parts: the encoder and the decoder.

Encoder: The encoder reads and processes the input data, applying the attention mechanism to understand the relationships and context within it. It then converts the input into a rich, contextualised representation.
Decoder: The decoder uses the representation provided by the encoder and additional inputs (if any) to generate the output. In tasks like translation, the decoder generates the translated text one word at a time, paying attention to the input text and what it has generated so far.

This great diagram by DeepLearning.AI provides an overview of how the architecture of a Transformer model encapsulates both of the aforementioned components [2].

Transformers in Practice

Transformers have been pivotal in advancing performance across a wide range of tasks [1, 3], including but not limited to:

Natural Language Processing (NLP): Transformers are behind many state-of-the-art models for language translation, text summarisation, sentiment analysis, and more.
Speech Recognition and Processing: These technologies have improved the accuracy of converting spoken language into text and vice versa, enabling more natural interactions with voice-assisted technologies.
Image Processing: Though initially designed for text, transformers have also been adapted for image recognition and generation tasks. They show promising results by treating parts of images as sequences.

Why Transformers?

Transformers stand out for several reasons, making them a preferred choice for handling complex data (such as audio):

Parallel Processing: Transformers have a distinct advantage over older models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs) by processing all parts of the input data at once. This ability not only speeds up training times but also enhances efficiency significantly. [1].
Scalability: With the capability to efficiently manage large datasets, including those with thousands of tokens in input sequences, transformers excel in addressing long-range dependencies. This scalability is crucial for processing extensive and complex information.
Flexibility: Beyond their success with text, the versatile architecture of transformers extends to processing audio and images. This adaptability allows them to be used across a wide range of applications, demonstrating their broad utility.

Transformers in Audio

Audio transformer models lie at the core of modern audio processing, especially in tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) synthesis. Despite the diversity of applications and implementations, these models share a common foundation: the transformer architecture, which is well-known for its attention layers. This underlying similarity extends to the utilisation of the model components; some models primarily leverage the encoder section of the transformer, while others employ both the encoder and decoder for more comprehensive processing [1, 3, 4].

A pivotal aspect of working with audio transformers is handling audio data. The process involves adapting the model's input and output layers, whether for speech recognition, voice generation, or any other audio-related task. This adaptability allows for transforming raw audio data into embeddings through preprocessing layers and, subsequently, converting predicted embeddings back into audio output. However, the transformative capability of the audio transformer remains constant, serving as a robust backbone for diverse audio processing tasks.

Key Concepts in Transformer Audio Processing

Building on the foundation laid out in our previous article, here are some of the main areas which Transformers are used in audio processing:

Audio Representation: Understanding how audio is digitally represented is crucial. Audio signals are typically captured and stored in a digital format as waveforms, with the waveform amplitude varying over time representing sound. Machine learning models, however, often require different representations, such as spectrograms or Mel Frequency Cepstral Coefficients (MFCCs), to effectively process and analyse audio data [5].
Preprocessing Techniques: Preprocessing is essential to preparing audio data for machine learning. This can include normalisation to ensure consistent volume levels across recordings, trimming or padding audio clips to a uniform length, and converting audio files into formats suitable for machine learning models [3].
Feature Extraction: Machine learning models don't work directly with raw audio data but rather with features extracted from the audio. Spectrograms provide a visual representation of the spectrum of frequencies in a sound over time, and MFCCs, which represent the power spectrum of an audio signal, are standard features used for audio processing tasks [3, 4].
Machine Learning Models for Audio: Various machine learning models can be applied to audio processing tasks, including traditional models like Support Vector Machines (SVMs) for classification tasks and more advanced deep learning models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), which are particularly effective for handling the temporal nature of audio data. Transformers, known for their success in NLP, are also being increasingly applied to audio tasks because they can model complex dependencies in data [3, 4].

Mel spectrograms representing various speech emotion states are utilized for inference by a model proficient in image processing, enabling comparison with an established training dataset [5].

Applications in Audio Processing

Speech Recognition is the Conversion of spoken language into text. This complex task involves recognising the spoken words and understanding the context in which they're used [3, 4].
Audio Classification involves identifying the type of sound or classifying the genre of a piece of music. It involves training models on labelled audio data to recognise patterns associated with different categories [4, 5].
Sound Generation: Creating new audio clips, such as music or synthetic speech. Deep learning models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) can be trained to generate new sounds that mimic the properties of actual audio samples.

How Audio Transformers could be utilised in Game Audio

Here are just a few ideas and explored concepts for changing up gameplay with Transformer technology such as Speech Recognition and other audio processing tasks:

Enriching Multiplayer Dynamics: Imagine stepping into a game where language barriers vanish, allowing players from different linguistic backgrounds to collaborate seamlessly. Speech recognition can transform spoken language into text, bridging communication gaps in real-time. This has the potential to open up multiplayer games to global teams, where players can strategise and bond without the hurdle of native language differences. The technology recognises spoken words and grasps the context, ensuring that the essence of every message is accurately conveyed, enriching the multiplayer experience [6].
Enhanced NPC Interactions: Moving beyond traditional dialogue trees, speech recognition provides players the ability to converse naturally with non-playable characters (NPCs). This can make each interaction unique, as players use their voices to influence storylines, solve in game tasks, or build narrative-led relationships within the game world and its characters. Speech recognition systems' nuanced understanding of language can allow NPCs to respond in more complex, engaging ways, making the game world feel alive and responsive to the player's voice [2, 6].
Innovative Music and Rhythm Gameplay: As is well known, the realm of audio extends into the heart of rhythm and music-based games, where speech recognition and sound generation could unlock new forms of interaction. Players could solve puzzles, play virtual instruments, or even control game elements through the rhythm and pitch of their voices. By classifying different sounds and genres of music, the game could offer a tailored experience, responding dynamically to the player's vocal input. This offers the ability to create a rich, immersive environment where music and sound are not just background elements but integral parts of gameplay.
Real-Time Voice Synthesis for RPG Immersion: Role-playing games (RPGs) can achieve unparalleled immersion with real-time voice synthesis. Players could hear their spoken words transformed to match their in-game characters, from noble knights to cunning sorcerers. This synthesis adds depth to character development and enhances role-playing, allowing players to embody their characters through voice fully. Whether in critical dialogue moments or casual banter with fellow adventurers, the player's voice becomes a powerful tool in storytelling [5].

The image depicts the user interface on convai.com for customizing Non-Playable Characters (NPCs) with advanced conversational AI. This technology enables AI characters to not only converse but also interpret and react to various inputs across virtual and real-world settings [2].

Through these applications, speech recognition and audio processing technologies show promise to transform the landscape of the interactivity of audio and gameplay. By breaking down language barriers, introducing novel interaction mechanisms, and increasing role-playing immersion, they open up new game development and player engagement horizons.

Given the transformative potential of transformer audio processing in game development, exploring audio transformers presents an exciting avenue for developers. These advanced neural network models excel at understanding and generating human-like speech, offering nuanced control over audio interactions. Their ability to process and generate audio in real-time, adapt to different languages, and create immersive soundscapes is ideally suited for enhancing game audio systems and tooling. Audio transformers can serve as the backbone for developing more interactive, immersive, and inclusive gaming experiences, marking a significant step forward in the evolution of game design and player engagement.

References

[1] Alammar, J. (2018). The Illustrated Transformer. jalammar.github.io. Last Updated: 27 Jun 2018. Available at: <https://jalammar.github.io/illustrated-transformer/>.

[2] Convai Technologies Inc. (2024). Embodied AI Characters ‍For Virtual Worlds. convai.com. Available at: <https://convai.com/>.

[3] DeepLearning.AI. (2023). A Complete Guide to Natural Language Processing. [Online]. deeplearning.ai. Last Updated: 11 Jan 2023. Available at: <https://www.deeplearning.ai/resources/natural-language-processing/>.

[4] HuggingFace. (2021). Transformers, what can they do?. huggingface.co. Last Updated: 14 Jun 2021. Available at: <https://huggingface.co/learn/nlp-course/chapter1/3?fw=pt>.

[5] Day-West, O. (2023). An investigation into the feasibility of implementing Speech Emotion Recognition. London: University of West London. [6] Thompson, T. (2024). How AI is Actually Used in the Video Games Industry. Last Updated: 28 Feb 2024. Available at: <https://youtu.be/j3LW5no-5Ao?si=QZxUjshgIRReoXv8>.