Will US Government Directives Spur IPv6 Adoption?

Yesterday, the National Telecommunications and Information Administration of the U.S. government hosted a workshop discussing the state of IPv6 in the United States and its impact on industry, government, and the Internet economy. I was asked to be a panelist, along with industry executives from ARIN, ISOC, ICANN, Comcast, Akamai, Verizon, Google, VeriSign, DOE, NIST, and DREN. Moderated by Aneesh Chopra, Chief Technology Officer of the United States and Vivek Kundra, Chief Information Officer of the United States, this was the first event in the past few years to truly shine a spotlight on IPv6 adoption (or lack thereof) and introduce key directives to move this issue forward.

Expecting an IPv6 service that matches IPv4 is asking a lot—the industry overall is still some distance away. John Curran from ARIN and Danny McPherson from VeriSign provided excellent detail on the adoption and usage rates of IPv6 and IPv4, predicting an IPv4 Exhaustion day of May 29, 2011.

A snapshot of IPv4 exhaustion statistics for September 29, 2010. Hurricane Electric has a series of applications, widgets and JavaScript snippets to display the estimated exhaustion time and rate of IPv4, and the growing adoption of IPv6.Afilias’ registry and DNS services support over 17 million domains. All our registries support IPv4 and native IPv6, and have done so for years. We have spent significant time and money in migrating our core services to be fully IPv6 compatible. Despite this attention and effort, and similar to other companies on the front lines of IPv6 deployment, neither the systems nor user adoption are quite there yet.

Usage drives demand – current IPv6 usage is limited

While our systems support IPv6, very few customers are actually using it. Here are a few key statistics:

The .ORG registry has over 8.6 million domain names registered in its system. Of these, about 17,000 names have both an IPv4 and IPv6 address in their glue records; and 99 names have an IPv6 only address. The .INFO registry has nearly 7 million domain names registered in its system. Of these, only 58 names have both IPv4 and IPv6 addresses in their glue records and 25 names have IPv6 only. The .MOBI registry has over 910,000 domain names registered in its system. Of these, no names have both IPv4 and IPv6 addresses in their glue records, and only 2 names have IPv6 only addresses. We also manage the technical systems of 10 national sovereign country code registries, with close to 1 million domain names registered. Of these names, only 14 domains have both IPv4 and IPv6 addresses in their glue records, and 6 domains have IPv6 only addresses.

In contrast, .ORG has over 2.3 million domain names that have an IPv4 only address; .INFO has close to 300,000 IPv4 only addresses, MOBI has 3,660 IPv4 only addresses and the 10 countries whose domains we provide technical services for have over 60,000 IPv4 only addresses.

Just to be clear, these numbers measure the number of “glue records” that have an IPv6 address. A more technical way of saying that would be the number of subdomain DNS servers within the Top-Level Domain (TLD) that can be queried over IPv6. That is not necessarily the same as the number of servers/hosts within those TLDs that are IPv6 enabled.

Aneesh Chopra called this adoption rate “lame.” It seems an apt description.

When “IPv6 Compatible” Isn’t Good Enough

Hardware vendors for critical pieces of load balancing equipment claim to have “Support for IPv6”. Recently, we evaluated equipment from various competing vendors. Quickly, it became evident that there is a remarkable difference in the way IPv6 flows are processed by the different security appliances currently in the market. IPv4 packets are processed by dedicated hardware built into the appliance—this allows the device to handle filter lists of several thousand entries with no or very little impact to performance. In contrast, IPv6 flows are processed in software and therefore impact the CPU directly, consequently affecting the forwarding rate of the appliance as the filter list grows in size. This presents a significant risk of performance degradation and there is no guarantee that software performance will match the service level standards we have been able to meet with our current IPv4 configuration. To solve for some of these limitations, Afilias has provisioned a separate set of front end firewalls dedicated to handle IPv6 only traffic in order to minimize risk on the IPv4 infrastructure. We have also beefed up our evaluation criteria because “support for IPv6” does not equate to “equivalent performance with IPv6”. My experiences were supported by other panelists, and Doug Montgomery from NIST spoke eloquently about having to reject many pieces of so-called IPv6 compatible equipment because of performance issues.

About the same time as we were looking at load balancers, we also evaluated vendors who could offer equipment for rate limiting traffic inflows into our global networks that would match our current IPv4 policies. As of today, there are are no vendors in the market that offer an IPv6 solution with sufficient performance to match that required by our current IPv4 supported policy. Therefore the likelihood of having a single appliance with a unique and enforceable policy for rate limiting in both IPv4 and IPv6 is very unlikely. This is an increased cost, management and usability burden. Vint Cerf from Google explained that his organization circumvents some of these issues because Google builds much of their own systems, but to the extent that they buy from external vendors, they too face similar issues.

Key Directives for IPv6 Adoption

At yesterday’s second panel that focused on Federal efforts on IPv6, US CIO Vivek Kundra unveiled the U.S. Office of Management and Budget’s memorandum to all CIOs of Executive Departments and Agencies of the US government that contained key actionable directives for IPv6 deployment and adoption.

These Federal government directives can be summarized as:

  1. All agencies across the US Government must deploy IPv6 on their public facing websites before September 30, 2012.
  2. Agencies must upgrade their entire internal infrastructure to native IPv6 before September 30, 2014
  3. Agencies must designate an IPv6 Transition Manager
  4. The equipment that agencies procure for networked IT must comply with requirements for the completeness and quality of their IPv6 capabilities.

George Conrades from Akamai suggested that for IPv6 adoption to be taken seriously, it needed to become a reported-upon item at Board Risk Management Committees. This suggestion resonated strongly with both the panelists and the audience, and Aneesh Chopra asked George to lead a “coalition of the willing” to deliver a Board Risk template that could be widely adopted in industry in the next 90 days.

My perspective that, prior to training technical departments, organizational procurement departments also require training on IPv6 found broad acceptance in the room. Teaching the procurement department that best fit calculations based on IPv4 defaults do not translate to a v6 environment, and that incompatibility of IPv4 and IPv6 operations will lead to poor and degraded performance for customers will mean that a different kind of procurement and technology checklist for IPv6 purchases is now needed. Chopra took that on, as a second deliverable to find a way to create a technology IPv6 requirements template to mirror the Board Risk template.

IPv6 performance and requirements must conform to a uniform template across the industry—something that does not exist today. Both Jason Livingood from Comcast and Nabil Bitar from Verizon said that Internet consumers first demand high performance, and IPv6 based networks have to deliver such performance above all else. Given the volume of data that flows between the companies whom the panelists represented—there was a gentle nudge to the Federal government to try to gather publicly available data on IPv6 usage, and publish them in an accessible manner—a national IPv6 Usage Dashboard, of sorts.

As Leslie Daigle from ISOC said, staying with the current Internet addressing protocol is not an option. We are rapidly moving to a world where new devices will be IPv6 accessible only. So far, IPv4 depletion has resulted in scarcity economics rather than prompting a massive IPv6 migration. The eventual adoption of IPv6 is likely to be driven not just by scarcity, but by the direct and tangible benefits organizations believe they can obtain—economic self interest will be the critical motivating factor. The US government directives could spur those economic forces further forward towards broader IPv6 adoption.

Written by Ram Mohan, Executive Vice President & CTO, Afilias

Migrating Services to IPv6

Introduction

Google is a pioneer in terms of enabling IPv6 for its services. Practically all of Google’s services are now reachable via IPv6. However, one small piece of the puzzle is still missing. Google is not returning any IPv6 addresses when you ask its standard DNS servers.  Unless you explicitly ask for ipv6.google.com or ipv6.youtube.com your host will communicate over the old IPv4 network. Other large content providers such as Facebook and Yahoo! are doing the same. No IPv6 address unless you explicitly ask for it. Why is this?

The reason is that enabling IPv6 causes some breakage and timeouts for HTTP traffic. There are clients that “think” that they have IPv6 connectivity but in reality they don’t. This causes the client to get a painful long timeout before the browser reverts to connect via IPv4. The number of broken clients have been reported to be somewhere around 0.05%. This is unacceptably high for Google and other large content providers.  If you would like to read more about this breakage, I strongly recommend Tore Anderson’s report on the subject, available at http://www.fud.no/ipv6/.

So here we are, about eight months before the IANA depletion and about a year and a half before it will get really messy, and we still have to figure out how to enable IPv6 for HTTP, one the most common protocols on the Internet.  But let us put HTTP aside for a minute and focus on some other protocols. Do we have the same problem there? Or is the problem Google is experiencing with HTTP a problem unique to that particular protocol? In this article, I will examine HTTP and two other very common protocols that nearly all organizations are using, namely SMTP mail and DNS.

Examining HTTP

First, let’s take a closer look at HTTP and examine the algorithm the client is using to select a destination.  At the end of the day, this is what is causing the breakage.

The IPv4 address is defined by the so called A-record in DNS whereas an IPv6 address is defined by an AAAA-record. For example, a dual stacked host could have something similar to this, configured in the DNS:

www.example.com.       IN           A             192.0.2.1

www.example.com.       IN           AAAA    2001:db8:1108:12::80

Once the user enters a hostname in the address field or clicks on a link, the system call to getaddrinfo()will be issued. The getaddrinfo() will in turn issue DNS queries for the A and AAAA records and return all available IPv4 or IPv6 addresses to the browser.

Note that there is no way for getaddrinfo() to send one DNS query for both the A and AAAA at the same time. Instead, two separate queries must be issued.

There is no RFC that is dictating if an A query should be issued before an AAAA query or the other way around. But the list of addresses returned to the client is sorted according to RFC 3484. The sorting mechanism will put IPv6 addresses first on the list. This will cause the browser to first try the IPv6 address when the actual HTTP connection is made. That is a very noble way of selecting a destination address that will increase the IPv6 traffic on the Internet. Doing it the other way around (sorting IPv4 addresses before IPv6) would prevent dual stacked hosts from using IPv6 until they have to access IPv6 only content. Such behavior would limit the usage of IPv6.

The problem with HTTP is clearly the timeouts.  Users are accustomed to the web being more or less instant.  A lot of the current IPv6 networks are less reliable than their IPv4 counterparts because they are still experimental. Another reason for the decreased reliability of IPv6 networks is the fact that IPv6 is currently being tunneled over IPv4. Tunnels will wrap the IPv6 packet in an IPv4 packet and send it over the IPv4 network. This causes the IPv6 Internet to be slower and less reliable than the IPv4 network in many cases.

If a timeout happens, the web browser will sit there doing apparently nothing from 20 and 75 seconds (depending on the platform and browser).  The fact that HTTP makes use of many short-lived connections further exacerbates the problem.  To illustrate the problem, imagine a broken client accessing a web site with 120 different elements (images, iframes, javascripts and so on).  Then assume that the web browser limits the number of parallel connections to 10 and that the platform uses 40 seconds to time out a failed connection.  The result?  120 / 10 * 40 = 480 seconds page load time!  Naturally, content providers are weary of subjecting revenue-generating visitors to such a horrible experience.

This breakage is really not anything new and unique for IPv6. Similar breakage can also exist in pure IPv4 networks. Take an organization with a busy web server. An easy way of sharing the load among multiple web servers is to put multiple A-records in the DNS. The DNS server will randomize the order of the records and the client will pick the first A-record (simplified). Let’s assume one server goes down because a network administrator pulled the wrong cable in the datacenter. The portion of the clients that was unlucky enough to select the address associated with that server will experience a timeout and a long delay.

There are companies specializing in hardware and software to make HTTP more resilient. Companies like Radware, Brocade and A10-networks come in mind. The fact that the retry mechanism in HTTP doesn’t kick in until after 20+ seconds has created a huge demand for load balancing products.

Examining SMTP

Now, let’s focus on SMTP mail for a while.  SMTP is using DNS to figure out how to route emails.  If you ever configured a DNS for mail you know that the MX record is used for this purpose. The MX record contains the receiver’s mail server names as well as an additional piece of information called the preference. The preference stipulates in what order the receiver’s mail servers should be tried. If the lowest preference server fails, the sending mail server will try the second lowest preference servers, and so on. If all servers fail, it will report breakage and send a bounced email back to the client.  An example of how this is configured in DNS is shown below.

Example.com.   IN           MX         10           barcelona.example.com.

Example.com.   IN           MX         20           madrid.example.com.

Barcelona.example.com.               IN           A             192.0.2.1

Madrid.example.com.                   IN           A             192.0.44.23

In the example above, the sending SMTP server would first try to deliver e-mails to the host Barcelona. If Barcelona is down for any reason, it would try the Madrid server. So how does SMTP’s selection algorithm choose the destination when we throw IPv6 into the mix? What if the DNS configuration looked like this instead?

Example.com.   IN           MX         10           barcelona.example.com.

Example.com.   IN           MX         20           madrid.example.com.

Barcelona.example.com.               IN           A             192.0.2.1

Barcelona.example.com.              IN           AAAA    2001:db8:10:133::1

Madrid.example.com.                   IN           A             192.0.44.23

Madrid.example.com.                   IN           AAAA    2001:db8:10:affe:babe::1

It turns out that there is an informational RFC in this case; describing who an email agent should work with (RFC 3974). What happens is that the sending server first picks the lowest preference mail server, then it picks AAAA before A for that mail server, and finally it randomizes, if multiple records of one type are available. In the example above, the address 2001:db8:10:133::1 would be tried first, then 192.0.2.1 then 2001:db8:10:affe:babe::1 and finally 192.0.44.23.

The good thing about SMTP is that redundancy is built into the protocol, so it will try all addresses available before it gives up and returns an error. This is a substantial difference as compared with HTTP which only tries the first address and then gives up.  Therefore, SMTP mail is a great initial target to migrate to IPv6.  There is no HTTP-type of breakage even if your, the ISP’s or the sender’s IPv6 connectivity is unreliable.

Examining DNS

How does DNS handle IPv6? The most obvious usage of DNS is when another protocol needs to lookup a name or an address to connect to. SMTP and HTTP are two such protocols and their behavior is described above. But DNS is also used by DNS itself to find delegation information. This is done so that the recursive name server can find the authoritative name server that contains the answer it is looking for. Those records are called NS records. The NS records are used to delegate a sub domain. For instance, if example.com is a university that would like to delegate the math.example.com domain to another server, we might see something like this in the DNS:

Math.example.com.       IN           NS          euler.math.example.com.

Math.example.com.      IN           NS          phytagoras.math.example.com.

euler.math.example.com.                          IN           A             192.0.2.2

euler.math.example.com.                           IN           AAAA    2001:db8::1

Phytagoras.math.example.com.                               IN           A             192.0.4.5

Phytagoras.math.example.com.                               IN           AAAA    2001:db8:0:1::2

So, then the question becomes:  how would DNS select the destination address? Would it act like HTTP and use getaddrinfo() and then pick AAAA before A? Or would it act like SMTP and try AAAA first for each DNS server, and then fall back to A? Or perhaps there is some other algorithm in place here?

It turns out that DNS made it easy for itself. What happens is that ALL servers are tested randomly and an algorithm called RTT banding is used to figure out which server has the lowest response time. I have personally tested this with the most common recursive DNS servers (Secure64, Unbound and Bind). This server (could be several if they have similar response time) will then be used for consecutive queries. The algorithm is independent of A and AAAA records. It does not matter if you configure the DNS as the example above suggests, or if you configure the server like this:

Math.example.com.       IN           NS          euler.math.example.com.

Math.example.com.      IN           NS          phytagoras.math.example.com.

Math.example.com.       IN           NS          euler6.math.example.com.

Math.example.com.      IN           NS          phytagoras6.math.example.com.

euler.math.example.com.                          IN           A             192.0.2.2

euler6.math.example.com.                         IN           AAAA    2001:db8::1

Phytagoras.math.example.com.                               IN           A             192.0.4.5

Phytagoras6.math.example.com.             IN           AAAA    2001:db8:0:1::2

The way DNS selects a name server to query will be identical regardless of how you configure DNS. So, this is quite different compared to HTTP and SNMP, that at least tried AAAA before A if a host had both.

The fact that more municipalities in Sweden have IPv6 enabled for more DNS servers than they have for  HTTP servers is probably not a coincidence. (http://www.kommunermedipv6.se/kipv6/ http://www.myndighetermedipv6.se/mipv6/)

Conclusion

There has been a lot of focus lately on the HTTP protocol and the breakage that IPv6 can create for HTTP.  The breakage is obviously a large problem for providers of content over HTTP. However, the breakage is specific for HTTP and does not affect all protocols. Two protocols that are less affected by the breakage are SMTP mail and DNS. Those protocols have built in redundancy, and if needed, are able to quickly adjust and avoid any broken IPv6 servers.

Based on my research, I recommend that organizations consider SMTP and DNS as their first services to migrate to IPv6.  This way, organizations can gain experience with IPv6 without potentially exposing the integrity of their network. As the next step, they can migrate their more problematic HTTP servers.

Acknowledgements

A special thanks to Torbjorn Eklov and Tore Anderson for insightful comments and suggestions .

Source: The IPv4 Depletion site

NTIA Holding Workshop on IPv6

The National Telecommunications and Information Administration (NTIA) is hosting a workshop today discussing the state of IPv6 in the U.S. and its impact on the industry, government, and the Internet economy. The moderators for the workshop are Aneesh Chopra, Chief Technology Officer of the United States and Vivek Kundra, Chief Information Officer of the United States.

Participants include:

  • • John Curran, President & CEO, ARIN
  • • Leslie Daigle, CITO, ISOC
  • • Peter Dengate Thrush, Chairman of the Board, ICANN
  • • Jason Livingood, Executive Director, Comcast
  • • George H. Conrades, Chairman of the Board, Akamai
  • • Ram Mohan, Executive Vice President, Afilias
  • • Dr. Nabil Bitar, Principal Member of Technical Staff, Packet Network Technology, Verizon
  • • Vint Cerf, VP & Chief Internet Evangelist, Google
  • • Pete Tseronis, Chairman of the U.S. IPv6 Task Force, Department of Energy, Associate CIO (acting)
  • • Doug Montgomery, Manager, Internet & Scalable Systems Metrology Group, National Institute of Standards and Technology (NIST)
  • • Ron Broersma, Chief Engineer, Defense Research and Engineering Network (DREN), Department of Defense

The workshop will take place in Washington, DC on September 28th, 9:00 a.m. – 12:30 p.m. (Agenda)

Other sources: (UPDATED Sep 28, 2010 12:00 PM PDT)
Will Feds Commit to Another IPv6 Mandate? Sep.28.2010, PCWorld

A Look at Nine Years of RIPE Database Objects: IPv6 Objects on the Rise

The RIPE Database is about to enter its fourth decade. It began humbly as a place to store network and contact information back when the RIPE community formed in 1989. When the RIPE Network Coordination Centre (NCC) was created three years later and started to assign and allocated IP address space, the database was expanded to include the registration of more detailed network and routing information.

The RIPE Database quickly became one of the cornerstones of the RIPE NCC and can provide us with a good indication of operational practices and trends.

Today the RIPE Database contains 22 different object types and 5.44 million entries in total.

We were interested to see how the various object types have been growing over time, especially in the last nine years. In Figure 1, the number of inet6num objects over time can be seen. These objects are used to register IPv6 addresses.

Figure 1: The number of inet6num objects in the RIPE Database since 2001

One can see that the number of inet6num objects is increasing steadily in the last few years. The small drops (for instance in 2005) are caused by clean-up activities in the RIPE Database.

Figure 2 shows the number of route6 objects (green) and those domain objects (orange) that refer to IPv6 addresses (reverse domain delegations made based on IPv6 addresses). Both plots show exponential growth. These numbers are a clear indication that IPv6 implementation is really taking off!

Figure 2: The number of route6 and IPv6 related domain objects in the RIPE Database since 2001

Please note that the drop in June 2006 was due to the removal of 180 domain objects after it was decided not to use .ip6.int for reverse IPv6 delegations anymore and to use ip6.arpa instead.

The peak in July 2010 has been caused by one ISP creating ~250 reverse domain objects.

For more information and other object types, please refer to the accompanying article on RIPE Labs: Nine Years of RIPE Database Objects

Written by Mirjam Kuehne

Native IPv6 @ Home

A new article from RIPE NCC Chief Scientist Daniel Karrenberg examines some interesting IPv6 observations, as seen from his private Internet connection. For the full story, see RIPE Labs…

IPv4 and IPv6 comparison and differences explained for beginners

Recently a friend was asking me about IPv4 and IPv6 which are very common terms for anyone in the networking field but even as a casual internet user, it is interesting to understand and compare these two protocols. The following is a generic differentiation written by one of our authors to brieftly compare the two protocols and should serve as an introductory guide to beginners.

Complete info at Island Crisis.