Migrating Services to IPv6


Google is a pioneer in terms of enabling IPv6 for its services. Practically all of Google’s services are now reachable via IPv6. However, one small piece of the puzzle is still missing. Google is not returning any IPv6 addresses when you ask its standard DNS servers.  Unless you explicitly ask for ipv6.google.com or ipv6.youtube.com your host will communicate over the old IPv4 network. Other large content providers such as Facebook and Yahoo! are doing the same. No IPv6 address unless you explicitly ask for it. Why is this?

The reason is that enabling IPv6 causes some breakage and timeouts for HTTP traffic. There are clients that “think” that they have IPv6 connectivity but in reality they don’t. This causes the client to get a painful long timeout before the browser reverts to connect via IPv4. The number of broken clients have been reported to be somewhere around 0.05%. This is unacceptably high for Google and other large content providers.  If you would like to read more about this breakage, I strongly recommend Tore Anderson’s report on the subject, available at http://www.fud.no/ipv6/.

So here we are, about eight months before the IANA depletion and about a year and a half before it will get really messy, and we still have to figure out how to enable IPv6 for HTTP, one the most common protocols on the Internet.  But let us put HTTP aside for a minute and focus on some other protocols. Do we have the same problem there? Or is the problem Google is experiencing with HTTP a problem unique to that particular protocol? In this article, I will examine HTTP and two other very common protocols that nearly all organizations are using, namely SMTP mail and DNS.

Examining HTTP

First, let’s take a closer look at HTTP and examine the algorithm the client is using to select a destination.  At the end of the day, this is what is causing the breakage.

The IPv4 address is defined by the so called A-record in DNS whereas an IPv6 address is defined by an AAAA-record. For example, a dual stacked host could have something similar to this, configured in the DNS:

www.example.com.       IN           A   

www.example.com.       IN           AAAA    2001:db8:1108:12::80

Once the user enters a hostname in the address field or clicks on a link, the system call to getaddrinfo()will be issued. The getaddrinfo() will in turn issue DNS queries for the A and AAAA records and return all available IPv4 or IPv6 addresses to the browser.

Note that there is no way for getaddrinfo() to send one DNS query for both the A and AAAA at the same time. Instead, two separate queries must be issued.

There is no RFC that is dictating if an A query should be issued before an AAAA query or the other way around. But the list of addresses returned to the client is sorted according to RFC 3484. The sorting mechanism will put IPv6 addresses first on the list. This will cause the browser to first try the IPv6 address when the actual HTTP connection is made. That is a very noble way of selecting a destination address that will increase the IPv6 traffic on the Internet. Doing it the other way around (sorting IPv4 addresses before IPv6) would prevent dual stacked hosts from using IPv6 until they have to access IPv6 only content. Such behavior would limit the usage of IPv6.

The problem with HTTP is clearly the timeouts.  Users are accustomed to the web being more or less instant.  A lot of the current IPv6 networks are less reliable than their IPv4 counterparts because they are still experimental. Another reason for the decreased reliability of IPv6 networks is the fact that IPv6 is currently being tunneled over IPv4. Tunnels will wrap the IPv6 packet in an IPv4 packet and send it over the IPv4 network. This causes the IPv6 Internet to be slower and less reliable than the IPv4 network in many cases.

If a timeout happens, the web browser will sit there doing apparently nothing from 20 and 75 seconds (depending on the platform and browser).  The fact that HTTP makes use of many short-lived connections further exacerbates the problem.  To illustrate the problem, imagine a broken client accessing a web site with 120 different elements (images, iframes, javascripts and so on).  Then assume that the web browser limits the number of parallel connections to 10 and that the platform uses 40 seconds to time out a failed connection.  The result?  120 / 10 * 40 = 480 seconds page load time!  Naturally, content providers are weary of subjecting revenue-generating visitors to such a horrible experience.

This breakage is really not anything new and unique for IPv6. Similar breakage can also exist in pure IPv4 networks. Take an organization with a busy web server. An easy way of sharing the load among multiple web servers is to put multiple A-records in the DNS. The DNS server will randomize the order of the records and the client will pick the first A-record (simplified). Let’s assume one server goes down because a network administrator pulled the wrong cable in the datacenter. The portion of the clients that was unlucky enough to select the address associated with that server will experience a timeout and a long delay.

There are companies specializing in hardware and software to make HTTP more resilient. Companies like Radware, Brocade and A10-networks come in mind. The fact that the retry mechanism in HTTP doesn’t kick in until after 20+ seconds has created a huge demand for load balancing products.

Examining SMTP

Now, let’s focus on SMTP mail for a while.  SMTP is using DNS to figure out how to route emails.  If you ever configured a DNS for mail you know that the MX record is used for this purpose. The MX record contains the receiver’s mail server names as well as an additional piece of information called the preference. The preference stipulates in what order the receiver’s mail servers should be tried. If the lowest preference server fails, the sending mail server will try the second lowest preference servers, and so on. If all servers fail, it will report breakage and send a bounced email back to the client.  An example of how this is configured in DNS is shown below.

Example.com.   IN           MX         10           barcelona.example.com.

Example.com.   IN           MX         20           madrid.example.com.

Barcelona.example.com.               IN           A   

Madrid.example.com.                   IN           A   

In the example above, the sending SMTP server would first try to deliver e-mails to the host Barcelona. If Barcelona is down for any reason, it would try the Madrid server. So how does SMTP’s selection algorithm choose the destination when we throw IPv6 into the mix? What if the DNS configuration looked like this instead?

Example.com.   IN           MX         10           barcelona.example.com.

Example.com.   IN           MX         20           madrid.example.com.

Barcelona.example.com.               IN           A   

Barcelona.example.com.              IN           AAAA    2001:db8:10:133::1

Madrid.example.com.                   IN           A   

Madrid.example.com.                   IN           AAAA    2001:db8:10:affe:babe::1

It turns out that there is an informational RFC in this case; describing who an email agent should work with (RFC 3974). What happens is that the sending server first picks the lowest preference mail server, then it picks AAAA before A for that mail server, and finally it randomizes, if multiple records of one type are available. In the example above, the address 2001:db8:10:133::1 would be tried first, then then 2001:db8:10:affe:babe::1 and finally

The good thing about SMTP is that redundancy is built into the protocol, so it will try all addresses available before it gives up and returns an error. This is a substantial difference as compared with HTTP which only tries the first address and then gives up.  Therefore, SMTP mail is a great initial target to migrate to IPv6.  There is no HTTP-type of breakage even if your, the ISP’s or the sender’s IPv6 connectivity is unreliable.

Examining DNS

How does DNS handle IPv6? The most obvious usage of DNS is when another protocol needs to lookup a name or an address to connect to. SMTP and HTTP are two such protocols and their behavior is described above. But DNS is also used by DNS itself to find delegation information. This is done so that the recursive name server can find the authoritative name server that contains the answer it is looking for. Those records are called NS records. The NS records are used to delegate a sub domain. For instance, if example.com is a university that would like to delegate the math.example.com domain to another server, we might see something like this in the DNS:

Math.example.com.       IN           NS          euler.math.example.com.

Math.example.com.      IN           NS          phytagoras.math.example.com.

euler.math.example.com.                          IN           A   

euler.math.example.com.                           IN           AAAA    2001:db8::1

Phytagoras.math.example.com.                               IN           A   

Phytagoras.math.example.com.                               IN           AAAA    2001:db8:0:1::2

So, then the question becomes:  how would DNS select the destination address? Would it act like HTTP and use getaddrinfo() and then pick AAAA before A? Or would it act like SMTP and try AAAA first for each DNS server, and then fall back to A? Or perhaps there is some other algorithm in place here?

It turns out that DNS made it easy for itself. What happens is that ALL servers are tested randomly and an algorithm called RTT banding is used to figure out which server has the lowest response time. I have personally tested this with the most common recursive DNS servers (Secure64, Unbound and Bind). This server (could be several if they have similar response time) will then be used for consecutive queries. The algorithm is independent of A and AAAA records. It does not matter if you configure the DNS as the example above suggests, or if you configure the server like this:

Math.example.com.       IN           NS          euler.math.example.com.

Math.example.com.      IN           NS          phytagoras.math.example.com.

Math.example.com.       IN           NS          euler6.math.example.com.

Math.example.com.      IN           NS          phytagoras6.math.example.com.

euler.math.example.com.                          IN           A   

euler6.math.example.com.                         IN           AAAA    2001:db8::1

Phytagoras.math.example.com.                               IN           A   

Phytagoras6.math.example.com.             IN           AAAA    2001:db8:0:1::2

The way DNS selects a name server to query will be identical regardless of how you configure DNS. So, this is quite different compared to HTTP and SNMP, that at least tried AAAA before A if a host had both.

The fact that more municipalities in Sweden have IPv6 enabled for more DNS servers than they have for  HTTP servers is probably not a coincidence. (http://www.kommunermedipv6.se/kipv6/ http://www.myndighetermedipv6.se/mipv6/)


There has been a lot of focus lately on the HTTP protocol and the breakage that IPv6 can create for HTTP.  The breakage is obviously a large problem for providers of content over HTTP. However, the breakage is specific for HTTP and does not affect all protocols. Two protocols that are less affected by the breakage are SMTP mail and DNS. Those protocols have built in redundancy, and if needed, are able to quickly adjust and avoid any broken IPv6 servers.

Based on my research, I recommend that organizations consider SMTP and DNS as their first services to migrate to IPv6.  This way, organizations can gain experience with IPv6 without potentially exposing the integrity of their network. As the next step, they can migrate their more problematic HTTP servers.


A special thanks to Torbjorn Eklov and Tore Anderson for insightful comments and suggestions .

Source: The IPv4 Depletion site

Yahoo proposes ‘really ugly hack’ to DNS

Network engineers from Yahoo are pitching what they admit is a “really ugly hack” to the Internet’s Domain Name System, but they say it is necessary for the popular Web content provider to support IPv6, the long-anticipated upgrade to the Internet’s main communications protocol.
Complete info at NetworkWorld, ComputerWorld and CIO.