Chime – A Game of Musical ManipulationIn 2010 Brighton based game studio Zoë Mode in conjunction with One Big Game, a video game based charity, released a remarkably original game through the Xbox Live Arcade. After some success it found its way on the PC via the Steam platform and earlier in the year a new enhanced “Super Deluxe” version arrived on the PlayStation 3. The game is called “Chime” and what’s remarkable is that it’s invented a whole new way of using and manipulating music in an interactive gaming environment. We popped down the south coast to talk to audio director Ciaran Walsh and lead audio engineer Joe Hogan about how the game works. Getting Into The Game Chime in full flow The simplest way to describe Chime would be to imagine if Tetris got trapped inside an ambient matrix sequencer while downing a cocktail of illegal substances. It’s a puzzle game but one that uses music in a unique way. Ciaran explains it this way; “Chime evolved out of an idea I had several years ago for generating music out of gameplay interactions in a fixed musical space. The principle was to create a game in which the gameplay is the embodiment of a set of musical rules and whatever you throw at it it’s always going to be transformed into something coherent and fed back to you. Our Creative Director, Ste Curran, introduced the pentominoes (Ed- shape made up of five squares) and really made the gameplay work over a slow-burn prototyping period in the Zoë Mode “Lab”. So it’s pretty unusual in that the musical mechanics came first and the game was built around them.”  Ciaran Walsh (right) and Joe Hogan in one of their studios Lead audio engineer Joe Hogan takes it from there; “Once the concept was proven, a small team was assembled (1x artist, 1x designer, 2x programmers and myself handling audio), to turn the prototype into a game. From then on the game design and the music system were developed at the same time, with a lot of collaboration between the audio, art, design and programming disciplines. The aim was to make a game that built the music organically according to how the player plays, ideally never sounding the same twice, but also being fun, and well balanced from a gameplay perspective too.” The game involves filling a grid with the aforementioned pentominoes to create “Quads” or squares and rectangles of at least 3 grid spaces a side, which then disappear. The aim is to create Quads over the entire playing area within a time limit. As the shapes are placed and Quads formed the music is triggered and remixed both in terms of melody and structure and so evolves differently as you play. Ciaran: “Chime has a different feel to it than other games – it feels very gentle, it’s not manic or stressful, although it can get there a bit. Usually when playing a puzzle game there’s a stressful intensity to it which the music is designed to emphasise – high energy, high tempo – intended to ramp up the tension. With Chime there’s a gentle atmospheric feeling to the music and it becomes a meditative experience as much as an against the clock feeling.” Sourcing The MusicCiaran explains where the music came from: “At the very beginning we were just focused on making a game that’s really fun to play. Once we knew we had that we started thinking more about what to do with it, how we would release it, what it should sound like. Around that time we hooked up with the One Big Game charity initiative and agreed to develop the game for XBLA (Xbox Live Arcade) for them free of charge. That’s where music licensing came in – previously we had been working with material created by Joe and the team in-house but with the game being developed for One Big Game we saw using high profile artists as a way to maximize benefit for the charity. Also the charity angle gave us a better chance to get artists on board for what was a completely unknown IP (intellectual property). We drew up a target list with a lot of names and worked with One Big Game and Nimrod Productions, our music licensing partner, to approach them.Music licensing is a complicated business. You need to have a lot of different people on board to make something happen. It’s not as simple as just talking to an artist and seeing if they like the game. Also because of the charity angle, everybody involved in the project was donating their services for free, which meant there was no money on the table for the artists, labels and publishers. It’s not easy to approach a major label and ask for something for nothing. But we got some great artists involved and where we had direct communication with them about the game they loved the idea – it’s exactly the sort of thing technology-loving musos like!” The finished game ended up with 6 levels and 6 artists who will no doubt be familiar:
 Brazil by Philip Glass in the Chime Deluxe version for PSN in multiplayer craziness Finding The Sounds That WorkJoe: “The tracks were mostly delivered as a folder full of stems or multitrack wavs/aifs, plus a reference mix, and in some cases a bunch of separate additional sounds that we can use. The normal process was to rebuild the song in Cubase and exactly recreate the mix, with correct reverb, compression, and mastering. We then start pulling the track to bits, and isolating layers, and certain combinations of sounds, and then structuring the parts in such a way that the game can use it. Kind of like doing a re-mix. Once the song is running in the game we can then see which bits need re-editing or structured differently, it takes a few iterations. In game it’s often surprising which sounds work really well, and which sounds don’t!”So, how well did it cope with a Philip Glass composition? Joe:“Brazil by Philip Glass proved interesting, and it really put the new music system through its paces! Quite literally testing every part of it … It has interesting chord changes, a constantly fluctuating tempo, and an unusual musical structure based on changing meters of 5/8 and 6/8. Amazingly the system held up well … mostly!” Dismantling The AssetsOpening up the Paul Hartnoll “For Silence” project in Cubase and it becomes clear how they break the material down into samples. Joe takes us through the procedure:“We recreate the mix as closely as possible. In this one I had to recreate a piano part that we didn’t have a stem for. Then it’s broken up into loops – the loop region is wider than the actual loops as it has reverb tails in it, so all the samples overlap. Some loops have notes that carried across and so that was also useful for letting them trail into the next sample which would sound odd looped otherwise. With the Portal song (Jonathan Coulton – Still Alive, as heard in the Steam game “Portal”) it had vocals and every single vocal phrase goes over the sweep so for some rather than having one bar extra they have two or three in some. There are 24 loops in this one, some have a lot more.” The For Silence track by Paul Hartnoll dismantled in Cubase.Showing the original mix recreated from multitrack/stems on the left. Separated into loops in the middle, and the shapes and quad sound to the right, and the music preview (used in the level select menu) at the far right. Ciaran: “The general principle of what’s going on is that the song gets dismantled into all its constituent parts including rules and definition of tempo and harmony and configured in the game data in such a way it can be reassembled dynamically every time you play the game. What’s different every time is the precise sequence of the backing and the notes triggered by the shapes and the quads.” Joe: “The vocal line gets broken down into words and I export them all as a single wave file. In Sound Forge I open the wave file and create a region list which defines how each sample is saved. This means that if when applying it to the game we realise we need to change something, change the length of reverb etc, rather than having to reapply and re-export individually we can add the reverb to the whole vocal line and use the same playlist to extract the samples – also makes it easy to replace samples in the playlist.” Ciaran: “One of the characteristics of making game audio that’s different to other fields is because what we generate is a whole load of assets that have to be integrated with systems, code and game data – all these things have to fit together and work and be flexible – we have to be really meticulous about how everything is organised, and the way the work flow is organised with things like naming conventions. If you’re working with some kind of pure creative process and ending up with a product it doesn’t matter how you get there, but with us it matters a lot how you get there.” Joe:“I think that’s something crucial that I’m learning more and more and that people outside the games industry often don’t have any appreciation of. You have to have a really consistent way of working. Especially when you have multiple audio people sharing the same projects on different machines, who have to pick up exactly where you are. And with thousands of files and multiple stages in the process everything has to be super consistent, named and structured in a logical way, and be back trackable." Ciaran: “One error in a file name and the game doesn’t work. Another point is where you’ve built stuff and you’re making a change and feeding new content back through that pipeline into the game - you make a mistake and it breaks.” Do they find the restraints of the system hurts or encourages creativity? Joe: “Both in different ways – with the restrictions you have to think in a creative manner to get the best of it.” Ciaran:“We have to be a lot more nuts and bolts and understand the techy context. But then composers 20 years ago would be utterly bewildered by Cubase.” The samples and their behaviour is defined by a script called a CLV file. It’s similar to an XML script or a style sheet and is essentially a text file describing every aspect of how the audio relates to the game. Ciaran: "There’s an awful lot of musicality that’s gone into this text file and is ultimately described by it; abstract events, harmony, dynamic content etc. We define the rules. Some of those rules are defined in a fixed way by us as opposed to by the player but the way things are assembled in accordance to those rules are defined by the player – what progressions occur, what melodies are triggered etc." Joe: "This, for instance (pointing to a section of text), defines the coverage percentage and whenever this is reached it goes onto a text string that’s defined there and each playlist is defined with certain properties, which links to the name of the sound in the sound bank." Get WwiseThe integration is achieved through the use of a piece of software called Wwise from AudioKinetic.Ciaran: “Wwise is a third party game audio middleware tool, it’s an audio engine and content authoring tool so there’s kind of a code element to it which gets integrated to the game and a designer tool which lets you define behaviours such as volumes and pitches etc. It’s a tool which is becoming an industry leader. Fmod (from Firelight Technologies) is the alternative player in game audio, most widespread and been around longest, but Wwise is much more in tune with the way we think as sound designers and musicians. There are similarities with software modular synths like Reaktor or MaxDSP – the thing is constructed out of hierarchies of objects and the objects are the things that the game understands. So you’ve got a bunch of sounds at the bottom of the hierarchy, we’ve got folders containing things, different sorts of containers containing sounds, like a random based one here so if you trigger this object it will randomly decide which of its child objects will be triggered. It can all be nested in quite complicated hierarchies but at the end of the path there’s always an audio file. The thing that the game does, the way that it interacts, is that it triggers an event. The event is the starting point of generation of audio so for the backing tracks for “For Silence” there’s one that starts it, one that stops it – the one that starts it triggers an interactive music object which contains all of the loops that Joe exported from Cubase. It gets triggered and defaults to playing the first one. Then there are game states, which are set by the events, which change what gets played. The thing’s playing round and round in a loop and the game says it needs to be on loop 2 and so it changes the state to loop 2. That will cause the switch on its next pass to trigger the next one."  Wwise by AudioKinetic, audio engine and content authoring tool. This shows Wwise (on a pc) synchronised to the game running on a PS3 over lan.
 In this Wwise screenshot you have:
Spending time playing the “Free Mode” in Chime it becomes clear how creatively useful the audio engine could be. Markus Shultz comments “It’s opened my eyes... when i’m doing tracks now i’m going to start thinking like this in terms of how to recreate stuff and create a different vibe just by changing pitches and the cuts of tracks.” Joe says that they have no plans to open it up to allow users to import their own tracks, it would require a fair bit of reorganisation but he’s certainly not adverse to the idea “The team here were very enthused to find a youtube clip of a user who had managed to drop in some of his own samples, and we are certainly not opposed to users modding the game or creating custom content.” Ciaran adds: “It could be adapted to function that way, the underlying nuts and bolts are similar to a matrix sequencer. Where the audio concept came from in the first place was thinking about the ability to conform any kind of input to harmony and rhythm, creating patterns out of chaotic inputs. I think that principle can be interesting creatively because you can think in terms of shapes and contours rather than note-by-note performance.” Well ReceivedReactions to the game by the artists have been overwhelmingly positive. Markus Schulz, in a YouTube interview says “When I first heard about the game I thought there would be different parts of my song played in different levels or whatever, but I had no idea that the song would be cut up and completely changed around like that in so many different ways. Each time you play this game you come up with different sounds, different patterns – it’s really a trip… it’s really stunning and beautiful and I couldn’t be more proud and more happy to be involved in this project.”Here's a few more interviews with some of the artists involved: In playing the game the musicality of it makes an immediate impression. The seamlessness with which the music builds, changes, evolves and dies as play progresses is so good that it feels completely natural and organic. Ciaran: “Nothing is more universal than music so that’s naturally been a big part of what we do. Even though Chime is a completely different kind of game to something like SingStar, there’s a common thread in the joy and escapism of music. We’re really passionate about that. It’s not all we do by any means but we’re always thinking about new ways to create deeper musical experiences through games.“ The StudiosZoë Mode has three studios on their Brighton premises. The main Studio A is based around a Mac Pro running Pro Tools HD and Cubase, the other studios and each engineer has Cubase and M-Powered Pro Tools running on a PC. We asked Joe about their choice of platform – “This is because the majority of the game development tools, and middleware that we use on a day to day basis are PC only. Also the vast majority of the rest of Zoë Mode (Programmers, Designers, Producers, Artists) are PC based, it makes a lot of things much simpler. We use a Mac in Studio A mainly for recording and surround work and because we have had a number of compatibility and stability issues with the Pro Tools HD hardware and PCs. All our problems disappeared once we switched to using a Mac. Chime was produced entirely in Cubase and we have found both Cubase and Pro-tools M-Powered to be reliable on our PCs though.” Studio ABe A Games Composer?For people wanting to write music for games Ciaran has this advice: “One of the difficulties for newcomers is understanding how the rest of development works. You can’t just exist in your own world creating masterpieces and sending in the end result – music in games is woven into the fabric of the game, it’s technically challenging in all sorts of ways, it’s interdependent with the design of the game, it has to have functionality as well as form. And game development is a fast-paced, high-pressure process with unpredictable twists and turns in every project, and rarely a single consistent vision from day one.”“As someone who commissions music, I have to know that the composer can deliver high quality at high speed, can understand a technical as well as a creative brief, and can deal with a bumpy ride over the course of a project. And they have to understand the medium – a film composer who doesn’t watch films would seem pretty bizarre, but a surprising number of composers think they can break into game music without any passion for games. My advice to would-be game composers is to play a lot of games, understand how the music works and who makes it, and try to build a network of contacts. Keep show-reels short, sweet and focused – if necessary make bespoke versions for different potential clients. And don’t ramble in covering emails or attach big audio/video files – those are my pet hates! Finally, don’t have any illusions that the games industry is awash with cash – it’s not!” Chime is available from Xbox Live Arcade, Steam for the PC and on the Playstation Network for around £3.99 http://marketplace.xbox.com http://www.steampowered.com http://www.zoemode.com http://www.onebiggame.com http://www.chimegame.com/ |