Composer Guy Jones talks adaptive music for games

Ever hear a song on Spotify and think, “this would make amazing BGM” while wondering why developers aren’t already jumping all over it? The reality is that there’s far more to the art of choosing music for game play than simply liking a particular track. To work within the dynamic, interactive game play environment, music and audio must be what we call “adaptive.”

To start, the songs you hear on Spotify are almost always linear in form, which means you’ll have start and end points with the music ramping up until you hit the outro and end. Linear music is more typical of film use. With video game music, however, the structure is non-linear, meaning there is no defined end, and tracks are designed to loop so they can adapt to the players’ actions.

During a recent video podcast interview with longtime professional game composer Guy Jones, I got the opportunity to delve into the topic of adaptive music and the some of the key techniques involved.

When Guy writes music for video games, which means he has to tell a story within a minute or two minutes of a music loop, he tries to imagine the gamer who sits down to play and is at it for a while. A loop has started when the gamer gets up to make a cup of tea and is still going while they’re away and disengaged from the game. The moment they sit back down is crucial, as the music needs to help reinvest them in the game’s world.

“The loop has to have a real impact on the world that it’s trying to sell. Not only that, but it has to be ready to shift gears whenever that character or player changes to a level of intensity,” he explained. “Say they walk through a door and a boss is there. All of a sudden, the music has to have an impact and it has to change. And that's why it's non-linear, because those moments can happen at any point, and the player decides when it happens.”

In its most basic form, creating audio for video games is about getting the loops right and making sure they work, but there are different techniques that are used to adapt the music to the game play.

Adaptive Video Game Music’s Vertical Technique

“A more modern version of this is using a vertical technique where you have different layers of a loop that all get triggered whenever something happens. And if anybody's familiar with Elias or FMOD, it allows you to do that really easily,” Guy told me. “So, you'll have what's called a base layer, and that will be maybe a drum kit. (Just sort of talking in a band sense.) And that's to say you've got your drums that are going, and they're kind of chilled and they're chugging along, and you're kind of drifting around the game and doing your own thing. Then, all of a sudden, an enemy pops out, and you might have an electric guitar kick in. That's all triggered within the software that can be used within Unity. The way I know how to do it is Unity showing a path to FMOD, but that path can be created in any form of middleware that people want to use.”

Digging in a little further, I asked him to visually describe the vertical music techniques.

“Vertical is exactly what it is, which is stacks of stems,” he said. “Where it is basically just a base layer and those stems that are underneath, it can all be triggered based on what's happening in the gameplay. That's why it truly is adaptive music.”

For those who are unfamiliar with stems, Guy explained that stems are individual parts of music. Looking at a band like Nirvana as an example, the stems would be Dave Grohl’s drums, Krist Novoselic’s bass, Kurt Cobain’s guitar and Kurt Cobain’s vocals.

“So, in that band, you would probably have about four stems. And it gets a bit more complicated than that but, in essence, stems are just the track that you hear in full, broken down into multiple different files,” he explained. “And this is why having loopable stems is such an asset to have as a game developer, because not only do you have a full track, but you also have the amount of stems that are given to you and you can have different variations within that track very easily.

“And it doesn't even matter if you want to trigger those stems based on a bus appearing or anything like that. You can just add variation to the loop over the course of 10 minutes by kind of letting those different stems fade in and out, because they're always running together. Even when say four stems are muted of the five, they're always running in unison and they'll always loop back together, and you can just have them kind of triggering… And you can even fade stems in and out as well underneath what would be a base layer.”

One good example of this vertical technique is in a game called Cities: Skylines. Basically, you can zoom in and out, and as you zoom out, the music gets more and more sparse. Likewise, when you zoom in, the different looping stems are faded in to make it feel like you're getting closer and closer to civilization. It's quite a cool effect.

The Horizontal Technique for Adaptive Video Game Music

The horizontal technique is another way to make music adaptive for game play.

“Horizontal can be tricky, 'cause there you're basically creating chunks of audio that all have to work in unison together, so there's almost like sharp cuts into certain aspects,” Guy said. “So, you have like an intro that cuts into a whole loop and then an outro that will be sharply cutting to that as well. And there are techniques that you can use to kind of fade and smooth those transitions over, but that's the basic audio editing level. It is going from one audio clip, jump into the next one and you have to make sure that they all fit perfectly. And I did a massive project where an enormous amount of music had to all work in that sense. And you had to make sure that outro to the loop, or that transition out of the loop could happen pretty much anywhere in the track.

That was a real challenge but, based on that challenge, we developed a technique with ALIBI where we could create loops with stacked stems that not only allowed us to have those horizontal techniques in place, but also allow us to create adaptive music through vertical techniques, because those stems could all work in unison and loop, and you could fade in and out. We kind of managed to get the best of both worlds through experiences with other game developers, which has been incredible.”

Using Tempo Maps to Smooth Game Music Transitions

To help with both the horizontal and vertical techniques Guy described, game developers can use tempo maps to make the transitions smooth.

“The BPM is always at the top of every file, and there's always a way to integrate it into the software that you're using to create the game. Sometimes it can be easier to have something like Elias or FMOD or Wwise, because it's a very big feature because BPM goes a long way to making sure that music runs really smoothly, particularly if it's music that has a lot of sharp transitions,” he said. “So, a transition is when you get the initial hit of a snare drum or the very second a guitar is strung. Just like those initial clips, which are the very start of the beat. And when that first click of the BPM comes in, it's really crucial to making sure that that's really clean. And is why, to the editor's dismay, we actually go through every single bar and we make sure everything is smooth no matter what bar the tracks come out of. And the only way we do that is by matching it with BPM and tempo. I think some people call it compass markers as well, or is that maybe the time signature, but we do include that as well.”

Beyond its looping format, vertical and horizontal techniques and use of tempo maps to smooth transitions adaptive music for video games can also use other techniques like drones and stingers, a topic Guy and I delved into further during our chat. You can learn more about that and listen to the full podcast interview HERE.