BGP anycast and dual datacenter: using 1 IP at multiple locations
“In times where the expectation level of users is higher than ever before and downtime is no option, offering applications at multiple physical locations through the same IP by BGP anycast is an absolute condition to keep your users happy”
The definition of BGP anycast routing
First of all the word “anycast” routing needs be explained because it’s naming is used in many cases where it should not be used. At Serverius we say that anycast is a collection of IP services that are presently using the same public IP address on the internet to offer their content to by different BGP networks to the nearest user. Data traffic from a user (source computer user) will be routed to the nearest BGP IP network. This will boost latency and improves load time, cut down bandwidth costs, improves availability and boosts the scale-ability. It’s important to know that “topographically closer” does not inherently mean geographically closer, although this is mostly the case.
Anycast is linked with the BGP protocol which ensures that all of a router’s neighbors are aware of the networks that can be reached through that router and the topographical distance to those networks. The IP address range is advertised by BGP at multiple routers which are using multiple carriers that are at multiple physical datacenter locations. As this propagates across the Internet, routers become aware of which of their neighbors provides the shortest topographical path to the advertised IP address.
With Anycast routing, nobody knows upfront which server will receive an incoming request. In this case, all hardware that shares the same IP address should have the same or similar content and should be ready to reply to all requests. So, with anycast routing client should take care of synchronization of data between hardware. The cool thing of Serverius is we provide our clients with a transparent L2 channel between our datacenter locations. It makes data synchronization easy and fast.
Cost calculation
Like many other things in life, “less risk will you cost more money”. When people want to start using a multi-datacenter IP setup they would like to go for the best possible setup. Of course, that’s logical, but in reality, I see that a lot of them cannot afford the best possible setup because the total cost of routers, IP subnets, uplinks, colocation racks, fibers between datacenters racks, time and knowledge to manage, are simply too much.
To understand the basics is sometimes a bit hard, especially in addition to their existing network infrastructure. It’s complicated but with nice new technical options that help our financially smaller budget users to get an acceptable anycast alternative. This way these dual datacenter IP setups are not only toys from large enterprises anymore. 🙂
The principle of a dual datacenter setup is always “the higher the costs, the better the solution”. And the other way around. That’s why the first step all everyone has to do is to make a financial risk calculation to see if the cost covers the expenses:
- What’s the total financial damage in case of downtime.
- What’s your profit to deliver lower latency.
- Do we need to adjust the network to our users in detail.
- Are our IP services capable to share the same content at multiple locations.
- Does your certification require separated physical data storage on multiple locations.
- What’s the value of less risk of downtime.
- What’s the value to perform hardware maintenance without downtime.
- Are my clients asking for it (indirectly).
Secondly, there are also less important arguments what can be important:
- Does a twin datacenter IP setup add value to your services (to beat competition).
- Are your technicians like to have a anycast infrastructure.
- Can you achieve better IP security (DDoS protection, firewalling).
- Does it keep my high availability simple or more complex?
The outcome will show you if it’s a “nice to have” or if it’s essential to have. If the answer is (almost) “yes” the Serverius NOC department can assists to setup a high availability network setup at both our SDC1 and SDC2 datacenters and even others. The principle “start small, expand later when you need it” is used to give a honest and affordable advice.
Let’s discuss some possible examples to create a default setup in a way how Serverius is offering it to its users by default. The 4 default configuration examples below are not all possible setups because there are many more ways to get even higher availability and lower latency. Although the principles will still be the same: creating an optimal solution on anycast or default dual datacenter network technology.
Example 1: BGP anycast on personal routers with multiple carriers
This is the best situation for anycast routing when a client has its own BGP routers with its own ASn and at least one /24 IP range to announce to the internet (client can rent IP range from Serverius or elsewhere). In both Serverius facilities, you can establish multiple BGP connections to Serverius and other IP-transit carriers. In this case, IP traffic for these IPs will be delivered to the closest datacenter near the user (the shortest traceroute possible). In case of and network outage at one location, the other locations will automatically take overall IP traffic.
You can manage IP data traffic to your requirement 100%, so specific BGP adjustments, special BGP community preferences, disable routes for maintenance, disabling carriers by partial carriers outages, etc. You are in full control to act where needed.
In most of these cases, people will use only 2 or more physical datacenter facilities where they will even make the network redundant with multiple carries per location. For example, they use Serverius and the NL-IX at SDC1 and Serverius and Retn at SDC2 and in case of a 3rd facility on or two other carriers. When using multiple carriers, a full routing table router will be needed. Otherwise, a more simple layer 3 switch that can do a partial routing table will do.
Advantage:
- Best availability and redundancy.
- Full management and routing control.
- Ability to use any datacenters at the same time.
- Ability to add extra carriers to a router on any locations.
- Use BGP network preference to add quality to the network and raise/lower the amount of users per datacenter location.
- High routing quality by the use of multiple carriers and physical datacenter facilities.
Disadvantage:
- Extra cost of own routers, BGP ASn and IP ranges.
- A bit more complex network setup.
- Router monitor maintenance time/cost.
Service and hardware cost:
- Your personal (non redundant) hardware infrastructure at all colocation facilities to host your applications.
- BGP routers on all datacenter locations.
- Colocation space at all datacenter locations.
- A carrier what will announce your IP subnet(s) from their router(s).
- (free) layer 2 transport vlan between your hardware at 2 locations.
- A /24 IP subnet with own route object what is able to be routed over multiple networks.
- A private ASN where you can be used to route your IP’s.
Example 2: BGP anycast on carrier routers with multiple carriers
If you do not (yet) have your own BGP routers with a private ASn, Serverius and another carrier can announce your /24 or larger IP subnets under the carrier its ASn for you. This way you are using all advantages of anycast with a limited amount of own control. IP traffic for clients will be delivered to the closest carrier/datacenter and in case of a carrier outage at one location the data traffic will be automatically re-routed to another carrier and another datacenter facility. You can still use the Serverius facilities and others, but please beware the not all carriers will announcing IP subnets from their ASn routers (some will force you to use your own routers and ASn).
You can’t fully manage your IP data-traffic yourself. Adjustments and routing tweaks are simply not possible because you do not have access to the routers yourself. The only thing you can do during network problems is to disable a carrier in total.
In most cases, people will use this setup to use anycast as cheap and as simple as possible. Route optimizing is done by the carrier quality, during partial outages they disable the carrier and for the support, they will fully rely on fast carrier response.
Advantage:
- Good availability and redundancy.
- Ability to use any Datacenters at the same time.
- High routing quality by the use of multiple carriers and physical datacenter facilities.
Disadvantage:
- Impossible to manage your routing yourself.
- Not able to change personal BGP network preference to add quality to the network and raise/lower the number of users per datacenter location.
Service and hardware cost:
- Your personal (non redundant) hardware infrastructure at all colocation facilities to host your applications.
- Colocation space at all datacenter locations.
- A carrier what will announce your IP subnet(s) from their router(s).
- (free) layer 2 transport vlan between your hardware at 2 locations.
- A /24 IP subnet with own route object what is able to be routed over multiple networks.
Example 3: Dual datacenter routing by 1 carrier (Serverius) with a /24 or larger
If you don’t have (or don’t want have) your own BGP router with own ASn to setup your BGP infrastructure (like example 1 and 2) you can also use a /24 or larger subnet to set up a fully redundant dual datacenter solution by only the Serverius BGP network. In case of an outage at one physical datacenter location, the BGP will stop advertising the /24 and the other locations will automatically take over. Like this you will have almost an anycast solution, but because the BGP is managed by the Serverius routers and you do not use any other carrier.
In a normal situation, your IP traffic will be delivered to the closest Serverius datacenter. For example, if someone in the south of the Netherlands, from London/America, use to the IP, it will probably end up in SDC1. If it is someone from central/north Netherlands, Germany, Northern Europe or Russia, it will most likely come out in SDC2.
The nice thing here is that with this option you can use an IP address on only one location or use the same IP at the same time at SDC1 and SDC2. But the downside of this setup is that you will have a little bit less personal control. For example, when you like to make personal routing changes it’s not possible or if you want to disable one /24 on one location during your maintenance, you need to contact the Serverius NOC department or disable your uplink. Also, because of the use of only 1 network carrier a general network error of Serverius will affect both datacenter locations.
This example is a good way to start your own network to upgrade to real anycast later on.
Advantage:
- Easy setup by the Serverius NOC department by hour fee.
- No VLAN cost between your racks, colocation clients can use free data transport between both Serverius datacenters.
- Good availability and redundancy, when 1 of your Serverius colocation facilities is down, all traffic automatically goes to the other side.
- Flexibility to (real-time) swap IP’s between datacenters.
- Disabling all uplinks at one datacenter facility will automatically forward all data traffic to the other facility.
- A good way to start and change it later on to a real anycast network: add more non Serverius datacenter facilities later on, swap the Serverius network at one facility by another carrier.
Disadvantage:
- Partly to manage the IP announcing at one datacenter yourself. Disabling one subnet can only be done by the Serverius NOC (and will cost money).
- In case of general Serverius network problems, both locations will be effected.
- It works only at the Serverius datacenters.
- Without owned BGP routers you are unable to add extra carriers like SpeedIX, NL-IX (open peering), Retn, etc.
Service and hardware cost:
- Your personal (non redundant) hardware infrastructure at both Serverius colocation facilities to host your IP applications.
- Colocation racks space at both Serverius datacenters.
- (free) layer 2 transport vlan between both Serverius datacenters.
- A /24 or larger subnet with own route object.
Example 4: Dual datacenter routing by carrier Serverius with small subnets like a /27
If you are using small subnets that are smaller than a /24 and you want to use a dual data center solution, the default Serverius IP infrastructure will offer it. In case of an outage at one physical data center location, the BGP will stop advertising the /24 where your smaller subnets are part of and the other locations will automatically take over. But in case of a partial Serverius network outage on one location, data traffic at the affected facility could not be automatically forwarded to your rack(s) at the other location. It really depends on the type of outage, sometimes it does flap over, sometimes not. Therefore this is not a real redundant or anycast solution and in this case, you will take the risk of a partial outage during Serverius network outages at one location.
In a normal situation, your IP traffic will be delivered to the closest Serverius datacenter. For example, if someone in the south of the Netherlands, from London/America, use to the IP, it will probably end up in SDC1. If it is someone from central/northern Netherlands, Germany, Northern Europe or Russia, it will most likely come out in SDC2. You need to create one VLAN between all your racks at both datacenter facilities because when your uplink is up, data-traffic for your IP’s are forward to you, even when the IP is on a hardware device at the other data center facility. Therefore you have to forward your traffic to the other location by the VLAN. This “transport VLAN” is offered by an extra physical fiber/copper cable (from another hardware infrastructure) and is offered by Serverius in most cases for free.
Also with this option, you can use an IP address on only one location or use the same IP at the same time at SDC1 and SDC2. But the downside of this setup is that you will have no personal control. For example, when you like to disable your /27 on one location during your maintenance, you need to contact the Serverius NOC department or disable your uplink. Also, because of the use of only 1 network carrier a general network error of Serverius will affect both datacenter locations.
This example is a good way to expand your colocation racks and use them as a whole. Upgrading to real anycast, later on, is not possible.
Advantage:
- This setup is available by default, without extra cost, available at Serverius IP-transit.
- No VLAN cost between your racks, colocation clients can use free data transport between both Serverius datacenters.
- Good availability and redundancy, when 1 of your Serverius colocation facilities is down, in most cases all traffic automatically goes to the other side.
- Flexibility to (real-time) swap IPs between datacenters.
- Disabling all uplinks at one datacenter facility will automatically forward all data traffic to the other facility.
Disadvantage:
- Not able to manage the IP announcing yourself. Disabling one subnet can only be done by the Serverius NOC (and will cost money).
- In case of general Serverius network problems, both locations will be effected.
- It works only at the Serverius datacenters.
- Impossible to use the same IP’s for anycast later on. Therefore no upgrade to anycast possible.
Service and hardware cost:
- Your personal (non redundant) hardware infrastructure at both Serverius colocation facilities to host your IP applications.
- Colocation racks space at both Serverius datacenters.
- (free) layer 2 transport vlan between both Serverius datacenters.
- Small subnets like /25 up to /32.
Flexible NOC assistance
The Serverius NOC department assists its users in setting up a high availability network setup at both our own datacenters and also external datacenters. We can advise, buy, finance, build, monitor and maintain it by a NOC as a Service (NOCaaS). In the past years, we did this for more than a few clients, where the combination with the client and the NOC is perfectly arranged. Sometimes people like us to perform the whole network and hardware setup, others only want to use the NOC to fall back on if needed.
More info: https://serverius.net/network-operations-center-as-a-service/
FYI: The options above are default configuration examples. Keep in mind these are not all possible setups because there are even more advanced versions for even higher availability and lower latency. Although the principles will still be the same, creating an optimal solution on anycast or default dual datacenter network technology.