Lesson 5

Command and control, automation & orchestration

Miriam Webster defines orchestration as “the arrangement of a musical composition for performance by an orchestra”.
 

The computer industry describes it as “the automated arrangement, coordination, and management of complex computer systems, middleware, and services”.
 

Actually they are not very different, but what does any of this have to do with broadcast? Quite a lot!
 

Since the 1950’s automation has been an integral part of broadcast systems. Automation has evolved as broadcast and production systems have become more complex and sophisticated. Each aspect of production and broadcast uses automation tools to create, manage, transport, distribute, protect and archive program content. File based workflows are dependent on automated processes to manage and move files throughout the entire media architecture.


The transition to an IP architecture has created significant changes in command and control and in how systems are monitored and managed. Command and control is the all-encompassing automated set of processes that control the acquisition, file movement, handling and delivery of media. Monitoring is more than a set of scopes and meters. Dashboards and browsers provide the system monitoring tools to manage the handling and Quality Control of media and metadata in the entire facility.


Automation was the system that master control used to integrate the traffic system to the play out system. The automation controls all the program source devices i.e. tape machines, servers, plus routers and master control switches to originate programming following the instructions of the traffic schedule. As the industry has moved to multi-channel delivery, the requirements for automation systems has become more complicated. This translates into receiving multiple instructions sets (commands) i.e. traffic schedules and playlists, then issue multiple commands the various devices it controls to playout content. And while automation has evolved to support it, it is now only one element in the overall automation system of an IP broadcast center.
 

Editing systems control source machines, switchers and mixers. This is one of the primary examples of command and control. The editing system issues a set of commands to the source device each time an element is selected. It issues a different set of commands when the finished program has to be rendered for each delivery format.
 

Command and control has evolved from Serial RS232, 422, 485 with GPI/O relay closure to an IP layer in the stream and file-based architecture that also handles all the management and movement of media. 


  

This shows the layered topology in the IP environment and how command and control is one layer within a single IP transport stream that carries media, metadata and communication in addition to command and control.
 

The stream and file based technologies and workflows have introduced new requirements for broader command and control systems. One that takes control of the entire media lifecycle from the time of concept to the point of distribution and beyond with all processes along the way.
 

In the file-based broadcast and production environment, for each process there is a need for control. Ingest devices need to know to start recording plus the format profiles and direction where to place the media and metadata once it’s created. The production and media management systems need to be notified that the file is ready for use. There is a media handling process that controls the movement of the files across the different business and production units and into different storage areas for production, media management, archive and delivery.
 

Every aspect of the file-based environment is managed by the command and control layer. And each process and device is typically controlled by an automated process. Even when a process is manual, the controller is the dashboard or a separately integrated control system that handles the manual process and shows the status and movement of files and streams within the entire media management environment.
 

Orchestration is the new term for an all-encompassing command and control management system. These systems are also called conductors. Conductors provide a unified dashboard to manage the command and control system. They host the rules and policies that manage all the devices and processes.  

The conductor is the dashboard that is the command and control center. It shows all the active processes, device status and where the files are in the system. The conductor controls the ingest processes and devices, handles media movement, interfaces with media management and controls the master control automation system for delivery. Following the schedules, rules and policies, the conductor will trigger processes then track their progress managing priorities and insuring continuity of the flow of media and metadata.


The Enterprise IT world realized that the interaction required between software applications and the movement of files between systems and processes needed to be managed, controlled, monitored and access protected. Business Process Management tools or BPM was introduced as way to resolve this problem. BPM is a layer of middleware applications that sits between other applications (in the middle) to integrate them or facilitate file movement and access.
 

As cloud services entered the picture and hosting application development in the cloud became commonplace, the need developed to have a single application to control and manage all these different and disparate processes and file movement.
 

Enter Orchestration, the next tier in automation; and command and control. Instead of each process having its own control, monitoring and management layer, this uses a single tool that unifies them into a single control application. This application manages and controls the processes by sending commands to each application and subsystem, reading and reporting on the status of each process in a clean and easy to monitor user interface.
 

As broadcast and production technology has become IP and file based and become dependent on software applications and servers, there are many new application and middleware tools on the market to handle this.
 

A majority of vendors that are selling file based solutions have multiple products to capture, encode, manage and distribute content. Within their own products they all have tools to control each of the processes and to handle the content. They also all have tools or interfaces - API’s - Application Program Interface, a set of programming tools that enable their own developers or their customers to develop the necessary software interfaces that will allow their products to integrate with third party or customer developed systems. This might be database integration for asset management or transferring media files from encoders to transcoders to editing systems.
 

As the broadcast and production industry are adopting and adapting tools from the Enterprise IT community, some of the more critical tools are management and monitoring. 


Orchestration is the next generation of automation and command and control. The stream and file based workflow are not single thread processes. There are multiple devices and applications at each stage of production and broadcast. There are multiple encoders, processing different formats and bitrates with files moving down one path, streams another. Asset management associates metadata with media and allows business applications access for protection and monetization. Files move between production, edit and distribution systems. While many vendors offer “end to end” solutions, there really are no “beginning to end” products or services.   

  

It is too complex to try and manage all this using separate dashboards (User Interface) for each application. Orchestration tools streamline this and add the “single screen” interface to monitor all the processes.


The master control room (MCR) is more of a Network Operations Center (NOC) and orchestration is a needed and critical component. The following list captures some of the responsibilities that the orchestration technology must monitor and manage:

  • Make sure devices are to the correct parameters (ie: Encoders and transcoders)
  • Configure each device based on traffic and schedule data
  • Manage the transfer or transport of the encoded file and/or stream to each stage in the process
  • Assign resources to each process

                  - Fiber or Satellite circuit

                  - Encode chain

                  - Storage location

                  - Play out chain

                  - QC chain

  • Monitor each of the processes to address any conflicts such as over allocating resources
  • Monitor network utilisation
  • Balance the distribution of processes across applications to optimise performance
  • Monitor systems status for any Alerts and Alarms
  • Monitor automated QC processes


The need for orchestration evolved as a natural next generation of automation technology. As devices, applications and systems all communicated using a command and control layer within the IP infrastructure orchestration enabled integration with each other. Orchestration is also critical as more and more applications are becoming cloud based. The integration of cloud services into the production environment will depend on orchestration tools to manage and control the processes in a seamless way.

image113