Improving Software-Based AV Deployments with Virtualization
7 minutes read •
AV and the Growth of Software
In the professional AV world, more and more of the devices we use to design and deploy complex and larger scale systems are just software. Dante Domain Manager plays a key role in managing and securing large AV deployments. Crestron VC and QSYS vCore offer much more flexible centralized control options, where adding more processing power is just a computer upgrade away.
Now these systems were originally expected to live on many of the same systems IT uses, sometimes as part of their Data Center. How it works out in reality often isn’t that way though, with complications landing these systems somewhere between air-gapped audiovisual networks and a separation of management / personnel between AV and IT. I’ve seen deployments on a NUC or Lenovo Tiny - and to be fair, the hardware used is plenty for the task.
My complaint is with how these tools are installed, all too often just right to the bare metal. AV has a lot to benefit from the IT world, and the approach to server deployments is just one of them. So let’s talk a bit about virtualization technologies, backups, and highly available, highly resilient solutions.
Even for One Server
Even with just one machine, virtualization provides substantial benefits.
- Backups and Disaster Recovery (DR): Incremental backups, data deduplication, automated copies of disk images - with a VM, your options are wide. With a standalone deployment to bare metal, you may need third-party software and complex configurations, while with a VM its a few files. The entire system can be fully restored from a backup within minutes to the same or different physical hardware.
- Hardware Agnostic: By running AV software (such as QSYS vCore or Crestron VC-4) in a virtual machine (VM), the underlying hardware can be swapped out easily. Even if there is a hardware failure, the VM disk image can be moved to a new host with minimal to no configuration changes.
- Snapshots: Full backups take time, and sometimes changes happen throughout the day. Snapshots really save the day here - you can take point-in-time snapshots of the running systems before running that software update or making major configuration changes. Any issue that breaks the system or functionality can be reverted instantly, minimizing troubleshooting time and potential downtime.
- Resource Management: Do you really need 32 cores to run that control system? How much RAM are you really using? With virtualization, you can precisely allocate CPU cores, RAM, storage, or even allow the system to balloon resources to get more if it needs it. Dedicate the resources it needs for stable real-time performance, and isolate it from background processes that can slow down button presses and frustrate a user.
Even just one of these reasons would be enough see virtualization as sensible, even for a single machine with a single service. As we all know, Day 1 and Day 2 can have some big differences in requirements, and having a solution you can grow makes that Day 2 effort a much smoother transition.
More Servers, More Options
Why limit yourself to just one server? If its a critical service, it deserves a cluster. Lets talk briefly about what this means…
A server cluster is a group of nodes that can work together as a unified system. You can have as few as two and have a cluster, with three you begin to add some important features. The cluster can have a quorum - this is where we get to the big advantages. Each node gets a “vote” for resource management, allowing us to add additional features that avoid a single point of failure.
That is where we hit enterprise-class territory.
Automatic High Availability (HA) Failover - This is the key resilience feature. If one physical node experiences a hardware failure (whether its a power supply, motherboard, or CPU failure), the remaining nodes can continue operation.
- Failure Detection: The HA Manager constantly monitors the health and presence of all nodes using a reliable cluster filesystem and network communication (typically Corosync).
- Automatic Restart: If a node fails, the HA manager automatically restarts the designated “HA-enabled” VMs on a healthy and available node within the cluster. The system waits a short time (about 2 minutes) to see if this is a temporary network interruption, then sets failover to ensure the failed node is properly fenced - forcefully and safely disconnected from shared resources - to prevent it from interfering.
- Reduced Downtime: While the VM experiences a brief power-off/on cycle on a different machine, this is an automated process that is significantly faster than repairing or reconfiguring a dedicated physical PC.
Live Migration - Multiple nodes, multiple VMs or containers, and you can move them between servers. There may be maintenance being performed on a system, a network switchover, or other operation which will make that server unable to do its job (or may be unable to do its job, and you want to take the safe and supported route). So you move the VM from one node to another node and let it keep working!
- Seamless Transfers: Live migration allows a running VM to be moved from one physical host to another within the cluster without interrupting its operation.
- Proactive Maintenance: Perform maintenance - whether its to apply host OS patches, a scheduled reboot, or to upgrade hardware on a node during production hours - without affecting the running systems. The VMs are seamlessly moved off the node getting its maintenance work done.
- Load Balancing: Live migration can also be used to manually balance the workload across nodes for optimal performance. Two VMs may be running higher CPU cycles, rather than have that processing on a single machine, you can move the VM to one with a lighter CPU load to keep CPU cycles available as well as keep CPU power requirements down, reducing power and heat impact.
Converged, Redundant Storage - For HA and live migration to work effectively, shared storage is a must. It could be on Network Attached Storage (NAS) or a Storage Area Network (SAN), but my favorite option is Ceph. Ceph is a a software-defined, distributed storage platform where compute and storage are unified on the same nodes. It provides a unified system for object, block, and file storage, and is designed to be highly scalable, self-managing, and fault tolerant, allowing for data to be distributed across commodity hardware.
- Ceph replicates data across multiple drives on the different nodes in the cluster. If a disk or even an entire node fails, data integrity is maintained through the remaining replicas, and the system can fully self-heal once the fault is corrected and the node is available again.
- What I specifically like about this approach is it eliminates the need for a NAS or SAN, which are usually expensive and potentially complex, since the storage is built directly into the nodes being used in the cluster.
High availability,
But Why Male Models Virtualize?
AV traditionally has not looked to these sorts of solutions - though some companies have, and I commend them for it. After all, virtualization can be seen as a layer of complexity, an unnecessary risk, or something that requires a new specialist for professional services companies that have run “just fine” for decades without them. That said, given the flexibility and resiliency of virtualization, can it really continue to be ignored, or should the ProAV world step up its game?
The growth of software solutions in professional audiovisual systems truly highlights the need for the AV industry to adopt standard IT deployment practices. We need to leverage virtualization technologies rather than installing software directly onto bare metal. Virtualization offers benefits even for single servers, like simplified backups, hardware independence, snapshots for recovery, and efficient resource management. For critical services, clustered solutions provide enterprise-class resilience through automatic high availability failover, live migration, and converged, redundant storage solutions like Ceph.
We can transform AV system deployments from a fragile single point of failure into a robust, scalable, and highly available architecture ideal for critical AV infrastructure. The ease of issue resolution, the simplification of upgrade paths, and the continuous operation of systems make the extra effort needed all the more worth it.