BGP anycast and dual datacenter: using 1 IP at multiple locations

In times where the expectation level of users is higher than ever before and downtime is no option, offering applications at multiple physical locations through the same IP by BGP anycast is an absolute condition to keep your users happy. After all, if things just are at only one physical datacenter location they are prone to hardware, power, network or any other physical imperfections. Beside of this, today our IP data should be close to our users and network system administrators should be able tweak their network to their users and should also be able to perform maintenance without downtime. Therefore the use of multiple datacenters is key for every serious IP application.

Some call it BGP network anycast, others dual datacenter IP setup and others twin datacenter colocation housing. All different naming with the same goal: using the same IP subnets on the same moment at multiple colocation facilities. By financial reasons most people use the same colocation datacenter company for their dual datacenter network setup. Others are using many datacenters round the globe. In case of a failing physical location, users will be automatically forwarded by BGP routing in milliseconds to the next nearest datacenter. No matter from what part of the world your users request want to access your IP’s.

Gijs van Gemert blog

This technical blog is written by
Gijs van Gemert
Mob: +31(0)657887267
gijs@serverius.net
Skype: serverius.bv

The definition of anycast routing

First of all the word “anycast” routing needs be explained because it’s naming is used in many cases where it should not be used. At Serverius we say that anycast is a collection of IP services which are present using the same public IP address on the internet to offer their content to by different BGP networks to the nearest user. Data traffic from a user (source computer user) will be routed to the nearest BGP IP network. This will boost latency and improves load time, cut down bandwidth costs, improves availability and boosts the scalability. It’s important to know that “topographically closer” does not inherently mean geographically closer, although this is mostly the case.

Anycast is linked with the BGP protocol which ensures that all of a router’s neighbors are aware of the networks that can be reached through that router and the topographical distance to those networks. The IP address range is advertised by BGP at multiple routers which are using multiple carriers which are at multiple phisical datacenter locations. As this propagates across the Internet, routers become aware of which of their neighbors provides the shortest topographical path to the advertised IP address.

With Anycast routing, nobody knows upfront which server will receive an incoming request. In this case, all hardware which share the same IP address should have the same or similar content and should be ready to reply to all requests. So, with anycast routing client should take a care about synchronization of data between hardware. The cool thing of Serverius is the we provide our clients a transparent L2 channel between our datacenter locations. It makes data synchronization easy and fast.

Cost calculation

Like many other things in life, “less risk will you cost more money”. When people want to start using a multi datacenter IP setup they would like to go for the best possible setup. Of course that’s logical, but in reality I see that allot of them cannot afford the best possible setup because the total cost of routers, IP subnets, uplinks, colocation racks, fibers between datacenters racks, time and knowledge to manage, are simply to much.

To understand the basics is sometimes a bit hard, especially in addition to their existing network infrastructure. I’s complicated but with nice new technical options that helps our financial smaller budget users to get an acceptable anycast alternative. This way these dual datacenter IP setups are not only toys from large enterprises anymore. 🙂

The principle of a dual datacenter setup is always “the higher the costs, the better the solution”. And the other way around. That’s why the first step all everyone have to do is to make a financial risk calculation to see if the cost covers the expenses:

  • What’s the total financial damage in case of downtime.
  • What’s your profit to deliver lower latency.
  • Do we need to adjust the network to our users in detail.
  • Are our IP services capable to share the same content at multiple locations.
  • Does your certification require separated physical data storage on multiple locations.
  • What’s the value of less risk of downtime.
  • What’s the value to perform hardware maintenance without downtime.
  • Are my clients asking for it (indirectly).

Secondly there are also less important arguments what can be important:

  • Does a twin datacenter IP setup add value to your services (to beat competition).
  • Are your technicians like to have a anycast infrastructure.
  • Can you achieve better IP security (DDoS protection, firewalling).
  • Does it keep my high availability simple or more complex?

The outcome will show you if it’s a “nice to have” or if it’s essential to have. If the answer is (almost) “yes” the Serverius NOC department can assists to setup a high availability network setup at both our SDC1 and SDC2 datacenters and even others. The principle “start small, expand later when you need it” is used to give a honest and affordable advice.

Let’s discuss some possible examples to create a default setup in a way how Serverius is offering it to its users by default. The 4 default configuration examples below are not all possible setups because there are much more ways to get an even higher availability and lower latency. Although the principles will still be the same: creating an optimal solution on anycast or default dual datacenter network technology.

Example 1: Anycast on personal routers with multiple carriers

This is the best situation for anycast routing when a client has its own BGP routers with its own ASn and at least one /24 IP range to announce to the internet (client can rent IP range from Serverius or elsewhere). In both Serverius facilities you can establish multiple BGP connection to Serverius and other IP-transit carriers. In this case, IP traffic for this IPs will be delivered to the closest datacenter near the user (the shortest traceroute possible). In case of and network outage at one location, the other locations will automatically take over all IP traffic.

You can manage IP data traffic to your requirement 100%, so specific BGP adjustments, special BGP community preferences, disable routes for maintenance, disabling carriers by partial carriers outages, etc. You are in full control to act where needed.

In most of these cases people will use only 2 or more physical datacenter facilities where  they will even make the network redundant with multiple carries per location. For example, they use Serverius and the NL-IX at SDC1 and Serverius and Retn at SDC2 and in case of a 3rd facility on or two other carriers. When using multiple carriers, a full routing table router will be needed. Otherwise a more simple layer 3 switch what can do a partial routing table will do.

Anycast option 1

Advantage:

  • Best availability and redundancy.
  • Full management and routing control.
  • Ability to use any datacenters at the same time.
  • Ability to add extra carriers to a router on any locations.
  • Use BGP network preference to add quality to the network and raise/lower the amount of users per datacenter location.
  • High routing quality by the use of multiple carriers and physical datacenter facilities.

Disadvantage:

  • Extra cost of own routers, BGP ASn and IP ranges.
  • A bit more complex network setup.
  • Router monitor maintenance time/cost.

Service and hardware cost:

  • Your personal (non redundant) hardware infrastructure at all colocation facilities to host your applications.
  • BGP routers on all datacenter locations.
  • Colocation space at all datacenter locations.
  • A carrier what will announce your IP subnet(s) from their router(s).
  • (free) layer 2 transport vlan between your hardware at 2 locations.
  • A /24 IP subnet with own route object what is able to be routed over multiple networks.
  • A private ASN where you can be used to route your IP’s.

Example 2: Anycast on carrier routers with multiple carriers

If you do not (yet) have your own BGP routers with a private ASn, Serverius and another carrier can announce your /24 or larger IP subnets under the carrier its ASn for you. This way you are using all advantages of anycast with a limited amount of own control. IP traffic for client’s will be delivered to the closest carrier/datacenter and in case of an carrier outage at one location the data traffic will be automatically re-routed to another carrier and another datacenter facility. You can still use the Serverius facilities and others, but please be ware the not all carriers will announcing IP subnets from their ASn routers (some will force you to use your own routers and ASn).

You can’t fully manage your IP data-traffic yourself. Adjustments and routing tweaks are simply not possible because you do not have access to the routers yourself. The only thing you can do during network problems is disable a carrier in total.

In most of cases people will use this setup to use anycast as cheap and simple as possible. Route optimizing is done by the carrier quality, during partial outages they disable the carrier and for support they will fully rely on fast carrier response.

Anycast option 2

Advantage:

  • Good availability and redundancy.
  • Ability to use any Datacenters at the same time.
  • High routing quality by the use of multiple carriers and physical datacenter facilities.

Disadvantage:

  • Impossible to manage your routing yourself.
  • Not able to change personal BGP network preference to add quality to the network and raise/lower the number of users per datacenter location.

Service and hardware cost:

  • Your personal (non redundant) hardware infrastructure at all colocation facilities to host your applications.
  • Colocation space at all datacenter locations.
  • A carrier what will announce your IP subnet(s) from their router(s).
  • (free) layer 2 transport vlan between your hardware at 2 locations.
  • A /24 IP subnet with own route object what is able to be routed over multiple networks.

Example 3: Dual datacenter routing by 1 carrier (Serverius) with a /24 or larger

If you don’t have (or don’t want have) your own BGP router with own ASn to setup your BGP infrastructure (like example 1 and 2) you can also use a /24 or larger subnet to setup a fully redundant dual datacenter solution by only the Serverius BGP network. In case of an outage at one physical datacenter location the BGP will stop advertising the /24 and the other locations will automatically take over. Like this you will have almost an anycast solution, but because the BGP is managed by the Serverius routers and you do not use any other carrier.

In normal situation, your IP traffic will be delivered to the closest Serverius datacenter. For example, if someone in the south of the Netherlands, from London/America use to the IP, it will probably end up in SDC1. If it is someone from central/north Netherlands, Germany, Northern Europe or Russia, it will most likely come out in SDC2.

The nice thing here is that with this option you can use an IP address on only one location or use the same IP on the same time at SDC1 and SDC2. But the downside of this setup is that you will have a little bit less personal control. For example, when you like to make personal routing changes it’s not possible or if you want to disable one /24 on one location during your maintenance, you need to contact the Serverius NOC department or disable your uplink. Also, because of the use of only 1 network carrier a general network error of Serverius will affect both datacenter locations.

This example is a good way to start your own network to upgrade to real anycast later on.

Anycast option 3

Advantage:

  • Easy setup by the Serverius NOC department by hour fee.
  • No VLAN cost between your racks, colocation clients can use free data transport between both Serverius datacenters.
  • Good availability and redundancy, when 1 of your Serverius colocation facilities is down, all traffic automatically goes to the other side.
  • Flexibility to (real-time) swap IP’s between datacenters.
  • Easy to use the Serverius IPC security cloud.
  • Disabling all uplinks at one datacenter facility will automatically forward all data traffic to the other facility.
  • A good way to start and change it later on to a real anycast network: add more non Serverius datacenter facilities later on, swap the Serverius network at one facility by another carrier.

Disadvantage:

  • Partly to manage the IP announcing at one datacenter yourself. Disabling one subnet can only be done by the Serverius NOC (and will cost money).
  • In case of general Serverius network problems, both locations will be effected.
  • It works only at the Serverius datacenters.
  • Without owned BGP routers you are unable to add extra carriers like AS-IX, NL-IX (open peering), Retn, GTT, etc.

Service and hardware cost:

  • Your personal (non redundant) hardware infrastructure at both Serverius colocation facilities to host your IP applications.
  • Colocation racks space at both Serverius datacenters.
  • (free) layer 2 transport vlan between both Serverius datacenters.
  • A /24 or larger subnet with own route object.

Example 4: Dual datacenter routing by carrier Serverius with small subnets like a /27

If you are using small subnets which are smaller than a /24 and you want use a dual datacenter solution, the default Serverius IP infrastructure will can offer you it. In case of an outage at one physical datacenter location the BGP will stop advertising the /24 where your smaller subnets are part of and the other locations will automatically take over. But in case of a partial Serverius network outage on one location, data traffic at the affected facility could not be automatically forwarded to your rack(s) at the other location. It really depends on the type of outage, in sometimes it does flap over, sometimes not. Therefore this is not an real redundant or anycast solution and in this case you will take the risk of an partial outage during Serverius network outages at one location.

In normal situation, your IP traffic will be delivered to the closest Serverius datacenter. For example, if someone in the south of the Netherlands, from London/America use to the IP, it will probably end up in SDC1. If it is someone from central/northern Netherlands, Germany, Northern Europe or Russia, it will most likely come out in SDC2. You need to create one VLAN between all your racks at both datacenter facilities because when your uplink is up, data traffic for your IP’s are forward to you, even when the IP is on a hardware device at the other datacenter facility. Therefore you have to forward your traffic to the other location by the VLAN. This “transport VLAN” is offered by a extra physical fiber/copper cable (from another hardware infrastructure) and is offered by Serverius in most cases for free.

Also with this option you can use an IP address on only one location or use the same IP on the same time at SDC1 and SDC2. But the downside of this setup is that you will have a no personal control. For example, when you like to disable your /27 on one location during your maintenance, you need to contact the Serverius NOC department or disable your uplink. Also, because of the use of only 1 network carrier a general network error of Serverius will affect both datacenter locations.

This example is a good way to expand your colocation racks and use it as a whole. Upgrading to real anycast later on is not possible.

Anycast option 3

Advantage:

  • This setup is available by default, without extra cost, available at Serverius IP-transit.
  • No VLAN cost between your racks, colocation clients can use free data transport between both Serverius datacenters.
  • Good availability and redundancy, when 1 of your Serverius colocation facilities is down, in most cases all traffic automatically goes to the other side.
  • Flexibility to (realtime) swap IP’s between datacenters.
  • Easy to use the Serverius IPC security cloud.
  • Disabling all uplinks at one datacenter facility will automatically forward all data traffic to the other facility.

Disadvantage:

  • Not able to manage the IP announcing yourself. Disabling one subnet can only be done by the Serverius NOC (and will cost money).
  • In case of general Serverius network problems, both locations will be effected.
  • It works only at the Serverius datacenters.
  • Impossible to use the same IP’s for anycast later on. Therefore no upgrade to anycast possible.

Service and hardware cost:

  • Your personal (non redundant) hardware infrastructure at both Serverius colocation facilities to host your IP applications.
  • Colocation racks space at both Serverius datacenters.
  • (free) layer 2 transport vlan between both Serverius datacenters.
  • Small subnets like /25 up to /32.

Flexible NOC assistance

The Serverius NOC department assists its users to setup a high availability network setup at both our own datacenters and also external datacenters. We can advice, buy, finance, build, monitor and maintain it by a Noc as a Service (NOCaaS). In the past years we did this for more than a few clients, where the combination with the client and the NOC is perfectly arranged. Sometimes people like us to perform the whole network and hardware setup, others only want to use the NOC to fall back on if needed.

FYI: The options above are default configuration examples. Keep in mind these are not all possible setups because there are even more advanced versions for even higher availability and lower latency. Although the principles will still be the same, creating an optimal solution on anycast or default dual datacenter network technology.