You awake to your phone ringing.
It’s the middle of the night and your Network Operations Centre (NOC) is on the other end of the call saying everything is down for a customer. You know you need to get an engineer there to fix it but the nearest is hours away. The customer is terrified of disruption and you are hoping this is all just a nightmare…
While this may be an extreme case, it raises an important question: how can you prepare for a scenario like this? The answer may lie in out-of-band management – a critical yet often overlooked aspect of data centre design that enables teams to access and control critical remote infrastructure, even during outages or cyber incidents.
What is out-of-band management?
Let’s start by exploring the two approaches for managing a customer’s environment. This isn’t limited to network devices but also includes the various appliances and servers that might be part of a data centre, campus, or remote office.
In-band management uses the same data path as production traffic to manage the customer environment, while logically isolating management traffic from production data. Although this approach can be more cost-effective, it introduces certain risks. If a problem occurs with the production network, it can also disrupt management access to the infrastructure, a situation referred to as “fate sharing.” In these cases, the only viable solution may be to send an engineer onsite to diagnose and resolve the issue. This can result in significant costs and delays, potentially impacting the customer’s business operations.
Out-of-band management, on the other hand, uses a separate network to provide independent access for managing the infrastructure, completely isolating management traffic from the production network. This separation is crucial during major disruptions like provider outages or security breaches, as it guarantees continuous access to network devices and servers, even if the primary production network is down or compromised.
Unpacking the different elements of out-of-band management
Nowadays, most appliances and servers are designed with dedicated, purpose-built interfaces to provide out-of-band management. These interfaces are generally separate from the data plane, offering exclusive access to the management plane of the device. Additionally, many servers on the market come integrated with out-of-band management tools or baseboard management controllers, enabling streamlined system provisioning, server management, and monitoring capabilities.
Here’s how out-of-band should be built from a network point of view:
● Out-of-band management should be an independent network, equipped with its own set of switches, to which all infrastructure management ports are connected.
● Moreover, the out-of-band network must have autonomous access to external networks. Its security is paramount, and it should be governed by a robust security policy – one that is at least as stringent as, if not more rigorous than, the policies enforced within the production environment.
● Many network devices and servers used in production environments are still equipped with legacy console ports (RS-232), typically used for the initial set-up of those devices. However, these interfaces also provide an alternative access point for managing devices within the customer infrastructure. To simplify this management and enable access to multiple devices through their console ports, a dedicated terminal server can be integrated into the out-of-band infrastructure.
● Remote power switches can be integrated with terminal servers to remotely power equipment in a customer’s data centre. Such functionality is crucial, particularly during security incidents because engineers can quickly reboot or power down a device, isolating it from the rest of the environment.
Building a robust out-of-band network
In many instances, a customer’s IT infrastructure is located far away from support personnel, making remote management essential. Take, for example, the following scenario: the command centre infrastructure – consisting of VDIs, management and monitoring servers – is in one of the cloud providers, from where support engineers are using VDI or other kinds of jump servers to reach the customer environment.
A secure connection links this cloud infrastructure to the customer’s on-premises IT setup, usually through a dedicated private network connection, SD-WAN, or an IPSEC VPN. This connection typically terminates at an on-premises router or firewall, safeguarding access to the out-of-band management network. Since this connection is considered primary, it should be designed with the appropriate level of resilience.
On the customer premise, a dedicated management LAN infrastructure should be established to facilitate the management connectivity of IT infrastructure. This setup must include a terminal server, granting access to devices with console ports. To enhance resilience, the terminal server should be able to support a secondary connection to the network management infrastructure in case the primary link becomes unavailable. This backup connection can be established using dial-up, an LTE cellular modem, or a secondary internet link with VPN connectivity. Incorporating secondary console connectivity, ensures a fail-safe mechanism, maintaining uninterrupted management capabilities of the infrastructure even in the face of multiple network disruptions.
At first glance, out-of-band networks might appear to be an unnecessary expense. However, as we saw last year when millions of Windows systems failed as a result of an outage, they are essential for maintaining business continuity and protecting a company’s reputation during network disruptions. For this reason, out-of-band management should be viewed as a necessary investment to prepare for the next big IT outage where every second matters.
About the Author
Jamie Moore is Architecture Manager at Systal. Systal is a global managed network, cloud and security transformation specialist. We manage complex and tailored technology services for enterprise businesses across 93 countries. These services help our customers achieve strategic technology transformation and maximise the business value, security and innovative potential of their IT infrastructure.
