Fail-Safe Broadcasting

This means that such a complex system can always be put in a situation which is unexpected, has never been tested, shows poor performance or the operator expects a different reaction/behaviour from the automation.

«So, for best reliability it is important to implement best practices when using automation, let me briefly describe what those best practices include:

• 100% redundancy by using main and backup sets of equipment (servers and operator workstations),
• Use multiviewer with video/audio issue detection such as still frame, black field, audio loss, audio overload, notify the operator by means of displaying alarms, sending messages,
• Use an automated failover switch (SL NEO Multiviewer can switch supported routers based on the detected video/audio signal issues),
• Put notes for the operators what they are supposed to do in case of emergencies, how to switch to backup manually, how to put an emergency clip on air etc.,
• In production always use the same media files types, codecs, bitrates,
• Before using new file types, new graphics, new codes, changing bitrates etc. test playback on the backup server,
• Create a backup of the configuration files each time you change the configuration,
• Create a backup of the configuration and media-database files before upgrading to new software versions,
• Test new configurations, new software versions etc on the backup server before putting them on the main server,
• Put all the servers and client workstations on a separate network from the rest of the facility,
• Don’t allow operator to use any usb thumb-drives, installing software or accessing internet from the servers and workstations,
• Do periodic system checks, monitor system error logs, clean fans etc.
• Monitoring what the operators do by recording the desktop of the workstations."

Regards and wishes for reliable hardware and software work,
Alex Sadovskiy, CTO and President of SkyLark Technology Inc.

Our experience and practice show that fears related to possible accidental hang-ups of computer playout systems are unfounded.

Just 3 simple rules will minimize failures and interruptions and will ensure many years of system operation:

1. At the design stage and the choice of the manufacturer of equipment and software it is necessary to take into account all the nuances: related to power supply, cooling, peak loads, redundancy, reliability of equipment manufacturers and system integrators.
2. Technical support is mandatory during the operation of any professional system, even the most compact one.
3. Qualification, training and motivation of personnel is the most important rule, as the absolute majority of failures are due to human errors.

Based on the great experience of our customers, we prepared a detailed list of requirements and actions to ensure failure-free broadcasting:

The choice of system structure and components

• Based on the type of programming and delivery options (just files + logo, or files + many live feeds + graphics, many output lines, etc.) it is necessary to choose the optimal functional scheme: either dispersed, with discrete hardware elements, or integrated (using Channel In A Box e.g.) Select the number of ports and channels in each server with at least 30% reserve CPU, disk and network resources.
• When choosing hardware and software manufacturers, pay attention to the availability of qualified technical support and the possibility of operative delivery of spare components.
• Server platforms, disk arrays and accessories should be well-known and generally available brands, without exclusive «firmware» disks and other limitations.

Redundancy of communication lines, power supply and critical nodes

• When planning input and output communication lines, provide main and redundant, with different paths.
• Provide 100% hot redundancy of all devices involved in signal processing, files, program signal generation and distribution. When choosing the system automation software, pay attention to the functions of constant automatic synchronization of playlists and content of the main and backup sets.
• Provide for 100% hot standby of local network equipment.
• 100% redundancy of all client workstations is not necessary. It is important that all workstation configurations be stored and any of them can be quickly restored on any less loaded station. Client workstations should not store important data (content, schedules).
• All signal processing and shaping devices must have 2 power inputs with automatic switching. Power to the inputs must be supplied from different sources with load balancing on the line.
• It is desirable to install an additional server that works as a sound alarm generator (for each channel or group of channels). The emergency screensaver must be switched on by means of a mechanical button at the workplaces of the on-air operators. The switchboard of the alarm signal on the diagram must be located after the switchboard of the transition to the standby half-completion.

Monitoring & Logging

• A system of on-line monitoring of the condition of hardware and software modules of the system via SNMP, an alarm system of problems with signals, files (light, sound, in GUI) for operators and engineers on duty is necessary.
• Automation system software, in addition to us run logs, must generate detailed technical logs.
• Video surveillance system with recording of operators' actions, video monitors' status, audio recording of operators' conversations, recording of intercom conversations, recording of client stations' desktop workstations is desirable to be provided in the production control room.
• In addition to recording signals from the system output, the monitoring recording system should record signals distributed by the telecom companies (signal delivery).