Building a Fault-Tolerant Show Choreography System for a Physical Light Installation

A music-reactive light installation can make a room feel alive. It can brighten with a chorus, pulse with a kick drum, or scatter color on a sharp onset. But reacting to sound is not quite the same as choreography.

Choreography is what lets the room feel as though it understands the shape of a song. It can gather energy ahead of a change, hold tension through a breakdown, send a buildup through a physical route, and release it without turning every light into the same flashing meter.

DungeonOPC is the Unity-based system at the center of this work. It renders visuals, samples them into OPC fixture output, and ultimately drives the physical installation. The design challenge is to make it more musically deliberate without making it dependent on any remote service being healthy at exactly the right moment.

The guiding principle is simple: upstream systems provide useful context; DungeonOPC keeps final local authority.

The room should not be remotely puppeted

Several systems can know something valuable about a track.

MixMan owns the authoritative playback state: what is playing, whether it is paused, and the current playhead and rate. Queen Mary analysis can add musical facts such as tempo, beats, bars, phrases, sections, and likely transitions. The MixMan Coordinator can shape that information into a small, time-bounded visual plan. Realtime audio can tell the installation what the music feels like in the room right now. And a local operator may need to request a temporary action.

None of those sources should directly own the rendered scene.

Networks can lag or fail. Analysis can be unavailable or wrong. A person at the installation may need to intervene. And an auxiliary source without rich metadata should still produce a good audio-reactive experience. DungeonOPC therefore treats incoming data as evidence rather than commands. It chooses from its local pattern catalogue, applies local safety behavior, and remains responsible for fixture output.

Different information answers different questions

The system combines four kinds of knowledge:

• Musical structure tells it where the track is: beat, bar, phrase, section, and transition.
• Realtime audio tells it what is happening at this instant: energy, spectral balance, and onset.
• Physical topology tells it where an action should occur in the installation.
• The local conductor decides what the room should do with all of that evidence.

The enriched path looks like this:

Player playback → MixMan → Coordinator visual plan → DungeonOPC conductor → Unity rendering → OPC fixtures
                         ↑
                 Queen Mary analysis

Realtime audio ──────────────────────────────────────┘
Operator requests ───────────────────────────────────┘

Coordinator is a forwarding and shaping layer in this design. It does not render visuals, and it does not bypass MixMan to obtain its own version of playback or analysis state.

Keep current playback separate from a visual suggestion

Two related APIs need different jobs.

The first is now-playing. It is the authoritative handoff for the current track and playback state. It answers basic present-tense questions: what is playing, is it active or paused, and what is the current local show state?

The second is a visual plan. This is a separate advisory message for the current track. It can include a revision, source, confidence, BPM, beat/bar/phrase context, a suggested local mode, and strict issued, effective, and expiry times.

That separation matters because future-looking structure is not the same thing as authoritative playback. A plan that was helpful ten seconds ago can become actively wrong after a seek, a track change, a delayed Coordinator refresh, or an outage.

DungeonOPC should accept a plan only while it is current, ordered correctly, and tied to the track it believes is playing. A fresh plan may help seed a local tempo and phase clock so that a change can land on a sensible beat, bar, or phrase. It must never be allowed to force a specific pattern remotely.

Audio is the living edge

Transport gives structure, but it is not a substitute for immediate sound. Realtime audio supplies the living edge: the information that makes fine detail breathe and helps a visual hit align with what people hear.

The planned audio contract uses one explicit versioned frame containing sequence, source time, confidence, full-band energy, low/mid/high bands, and onset. The metadata is as important as the energy values. Sequence numbers reveal gaps and reordering. Source time makes timing inspectable. Confidence expresses signal quality. Onset can create a local tap or transient action.

The listener must be skeptical. It should reject malformed, duplicate, and reordered frames; recognize a bounded source restart; record diagnostic counters; and let reactive influence fade after a stale timeout. If audio disappears, the room should not keep displaying the last bright pulse as though the beat were still there. It should return safely to a quiet but intentional state.

Modes give the room an identity; motifs make a gesture

The local conductor is the proposed decision layer that turns these inputs into a coherent room.

A mode is the room’s sustained identity. It can establish a palette, density, intensity range, speed, baseline pattern behavior, and the physical zones it is willing to lend temporarily.

A motif is a short musical gesture: a build moving through the room, a release, a call-and-response, or an orbit. It is temporary, bounded by duration and confidence, and scheduled to a sensible musical boundary when the available evidence supports one.

A motif does not take the whole room by default. It asks for a zone lease: a temporary reservation of named physical areas with an owner, priority, and expiry. When the action completes, loses its source, expires, or conflicts with a higher-priority request, the lease is released and the underlying mode resumes cleanly.

This creates a useful priority order:

1. A live, non-expired operator request.
2. A safe local choice informed by fresh Coordinator and MixMan context.
3. Audio-only local selection.
4. A known ambient fallback.

The result is controlled handoff instead of a fragile contest between remote recommendations, audio pulses, and manual controls.

The physical room must be shared data

A directional visual only has meaning if the system agrees on where the physical fixtures are. The roadmap treats that layout as shared topology rather than scattered texture coordinates hidden in individual effects.

A RoomTopologyProfile can describe named fixture routes and zones, their direction, sample counts, metric polyline points, and the conversion from physical metres to Unity coordinates. The camera-to-OPC path can then sample equal distances along a named physical route and project those positions into the render texture.

This makes the preview and physical output speak the same language. A sweep can move from one real area of the installation to another because both Unity and OPC output use the same measured route data.

A practical fallback remains essential. When no profile is installed, the legacy camera-sampling path should continue to work. When an enabled profile is malformed or out of view, the failure must be visible in diagnostics rather than silently mapping fixtures to some unrelated route.

Zone awareness is not yet zone compositing

There is an important boundary to state plainly. A conductor can understand zones and authorize a motif to lease them before the renderer can truly render different visuals into different physical areas at the same time.

The production rendering path is still largely a single composited visual. To keep a baseline mode visible across the room while a motif affects only its leased zones, the system will need a deliberate rendering architecture: masks or layers, separate cameras or render textures, and a deterministic composition stage before OPC sampling.

That work should not be faked by adding zone labels to a whole-room effect. The topology, leases, bindings, and diagnostics provide the right model for the future implementation; they do not magically create per-zone output on their own.

Failure should be understandable and safe

Live installations are hard to debug after the fact. Timing, networks, audio, Unity rendering, and real fixture wiring can all interact in ways that are invisible once the moment has passed.

The roadmap therefore calls for compact diagnostics and replayable traces. Operators should be able to see source freshness, plan revision, audio-frame health, active mode or motif, zone leases, fallback state, and the selected topology profile. Recorded events should make it possible to replay a Coordinator loss, a stale-audio reset, a pause or seek, an expired plan, or a track change against a real local orchestrator.

The goal is not only to show that parsers accept valid messages. It is to prove that a failure produces the intended local behavior.

Where the work stands

The current roadmap reports a separately validated foundation: strict visual-plan contracts, v2 realtime-audio work, topology primitives, and the upstream MixMan-to-Coordinator handoff have been developed in stages. It also makes clear that this is not yet a complete room choreography system.

The remaining delivery work is deliberately ordered. First comes closing known baseline and diagnostics gaps. Then the topology profile must be connected to the active OPC sampling path and calibrated against the real room. The optional live playback path needs end-to-end proof in an actual session. The local conductor, timed intents, modes, motifs, leases, and the first build-to-release gesture then need focused implementation and validation. Finally, true per-zone compositing, replay, resilience, and physical-room acceptance testing complete the picture.

This is not a compromise. It is the difference between an impressive demo and a system that can be trusted in a real installation.

When the full stack is healthy, the room can become musically structured and spatially expressive. When only audio is available, it can remain alive. When a network, analysis source, or calibration profile fails, it can explain the fallback and keep operating safely. That is the real purpose of keeping the conductor local: the intelligence may be distributed, but the show remains where it is happening.