The Problem:
Computer Choreography
The Ladder is, at its core, a theatrical experience. Just like traditional stage theatre, it involves performing story beats for you that are cued, acted out, and spruced up with dynamic lights and audio sequences. All of these things being cued together creates a wondrous multi-sensory thing that is greater than the sum of its parts.
But what’s doing the “acting out”? In our case, besides the lights and audio, what carries each story moment is the interactive media displayed on screens – characters talking to you, menus offering choices, and so on – as well as several moving set pieces run by microcontrollers. And with the cues themselves happening automatically on timers and based on player interaction, our experience mostly becomes a play directed and performed by computers. So how do you get these computers to act on cue?
As the designer in charge of building the interactive media, this question was critical to my work, and I happened to be the first on the team to need an answer.
The Solution:
Take the Leap & Learn MQTT
This aspect of The Ladder was tackled primarily by myself and lead creative technologist Matthew McClain, with help from Robert Twombly, Joe Pisanzio, Peter Helstrom, and Michael Pammit. I believe it was Peter, who had experience with MQTT, who first suggested it.
MQTT is an events-based messaging protocol used heavily in IoT-related technologies. And when we started putting time into learning it, I was terrified, because I understood approximately zero words in that sentence. But in the spirit of “just-in-time learning” (which is corporate-speak for blind hope and lots of YouTube tutorials), I was able to grasp it soon enough.
MQTT uses a publish-subscribe model, which means that you don’t have to connect to another machine to send them a message, you essentially just shout the message into the air to be received by any party who happens to be listening for it. There’s a “topics” system that keeps messages organized into channels and sub-channels for easy filtering. There are a few other features, like quality-of-service levels that ensure messages are sent and not double-sent, but in general this is about as bare-bones a system as you can find for this kind of purpose. Better yet, MQTT is widespread enough in the industry that it can work with essentially any program or device that can run some code. That meant that for every device in The Ladder, we could assume with reasonable confidence that it would be able to use MQTT to communicate with home base and vice versa.
The Process
Planning Out the System
Matthew and I worked out a system of nomenclature for how MQTT topics and messages would be treated. We split the messages into two primary categories:
- Commands: Messages sent from home base to an app.
- Reports: Messages send from an app back to home base.
There were more categories than this as time went on, but the bulk of The Ladder’s wireless communication is commands and reports. Commands are generally related to cues (playing sequences to start and end decades, going to pre-show, etc), and reports ended up generally being used for Unity sequences to trigger lights and audio and other things from home base. This system worked really well for us and survived the whole project pretty much unchanged, which is something I most definitely cannot say for the messages themselves.
Every Unity app has a number of different cues or functions it can perform, as well a certain number of things it needs to report back to home base. MQTT messages take the form of a piece of data – which for our purposes means a number or a short line of text (called a “string”). This presented us with a problem: how do we send messages to and from these Unity apps that can be interpreted?
Our answer was eventually to give each command and report a corresponding string tag (e.g. “cue-start-decade”, “timer-stop”, “video-finished”) and to keep track of all of these tags in a giant spreadsheet. Every time I added a new command that needed cueing or report that needed to be accounted for on the other side, I would make sure to let Matthew know and to put it in the spreadsheet. I’m sure I never forgot to do this.
Integrating MQTT with Unity
So I lied when I said that I understood none of the words in the sentence “MQTT is an events-based messaging protocol used heavily in IoT-related technologies”. I did understand one crucial word, and that was “events”.
Unity is a game-development platform first and foremost, not a smart-home app development platform, but game code and most other types of code have a concept of events. Events are momentary bursts of data that can be sent out into the air, or “invoked”, just like MQTT messages can, and you can write code that listens for specific events and acts on them when they happen. So it wasn’t too much of a stretch for me to see how MQTT could fit into my workflow, at least if I could figure out how to make Unity and MQTT work together. And to my delight, someone already had.
From there, it was a simple matter to have Unity apps connecting to the MQTT broker automatically, connecting commands with pre-programmed events, and sending reports. The biggest challenge, as hinted above, was staying connected with the team and making sure everyone was aware what the apps were capable of doing and what commands were available.
Other Uses of MQTT
Another use of MQTT that bears mentioning is interactive menu input. Logic would have you believe that the screen showing a menu and the peripheral gadget controlling your selection would be running locally on the same machine, but in all sorts of instances that turned out to not be the case. Here I started really appreciating the different levels of quality control MQTT provided. A command is one single very important message, but sending the movement of a knob to Unity means sending a stream of thousands of messages, each representing the knob position at a particular millisecond, such that Unity receives a smooth model of the knob’s movement over time and can reflect that in the movement of the menu on-screen. So the luxurious amount of quality control I gave to these commands, for which a single missed message would visibly impact the players’ experience, would be ridiculous to assign to each of these tiny knob-movement messages and would probably cause a strain on the system’s performance.
Lastly, I eventually reserved a separate MQTT topic for each Unity app to send logs to anyone who wanted them. Logs are helpful messages sent internally by the app for debugging purposes, including error messages and status updates. I tend to use logs heavily in my own work, and eventually I wanted everyone else to have access to them as well. This also gave me an excuse to whip up a simple log-catching app for the team.
The Final Result
I came into this MQTT journey very hesitantly. Even the acronym scared me. It just felt like a massive leap into “deep computer stuff” land, a land only IT professionals knew how to explore, where you have to know what your IP address is and programs don’t have any buttons and you have to type stuff directly into the terminal. I’m proud of myself for getting over the hump and doing it, and like so many things like this it turned out to not be that bad. As I move forward onto other projects, I’m sure I’ll learn other equally-daunting technologies that will exist alongside MQTT in my toolbox.
