Significant traffic drop on 30th of november or where did my traffic go?

If you rely on European traffic, I guess you observed, too, that on 30th of November a great per cent of your traffic has gone… well, nowhere, but the users couldn’t reach your site hosted in the US.

Before we figured out what’s happening, we followed the golden route to debug the issue, nonetheless to say without any result, if you read the whole article you’ll understand why.

First, we checked the number of the incoming connections. It was significantly lower than the usual. A pattern could be observed, though, specifically that no user was coming from west European countries like France, United Kingdom, Germany, Spain and so on. This is weird because usually when there’s an issue with the webserver, the whole traffic is affected not only a certain part of the World. But it can still indicate a DNS issue — like records not propagating correctly — or, in certain cases problem with the firewall, for example the firewall deliberately started to ban some routers as an answer for an imaginary DOS attack.

So, after figuring out the above mentioned pattern, we checked the DNS records. There seemed to be no problem with them and were not even updated in the past few months. That can not be the problem, since records will get propagated again only if we make changes to the DNS records.

The firewall’s ban tables also seemed to be clean of issues. Only the bad robots and some aggressive spammers’ IP were in the table. Next we checked the Login Failure Daemon (LFD) to see if by any chance there are too many bans set due to login failures. But the table was also clean.

Vario traffic outage on 30th of November

The next step was to examine the uplink provider’s connection. We are linked to the Internet through 6 providers, including Level3, Savvis and Verio, but the reports showed all of them are working on optimal capacity. But! Here comes an interesting thing: there was a huge traffic drop on some of the provider’s traffic graphs (see graph on left). This isn’t normal, usually the traffic’s graph is a very nice sinus curve, so the outage is not normal, at all. But let’s move forward. So far we know that the issue is not on our side, and is likely that a 3rd party has issues. Now the only thing we have to figure out is that who’s the third party? This isn’t that easy as it sounds as the uplink providers don’t offer direct support for webmasters, they talk only with the datacenter the server is located in so contacting them is pointless because we either don’t receive an answer, or we will receive an answer like “Ask the DC, not us”. None of them is good for us at any time.

But we can do monitor remotely the global routers without contacting anyone. This is more than cool, because if there’s no reported issue with the routers, than the uplink providers have issues, else (some) of the routers.

To see what’s happening on the routers’ end, we can go to http://www.internettrafficreport.com/ and take a peek on the graphs which shows the global internet traffic’s details. Since we were interested only in the European traffic, we looked only on the European routers’ details. And we’ve found what we’ve been looking for: the average response time of the European router’s were in the skies, a bit below 195ms, which at the time of this post is way above 210ms. The value considered to be normal is around 150ms, 60 additional milliseconds can mean some really great issues for the users because they most likely will see only “Connection timed out” screens. Looking on the “Packet Loss” graph, we observed another uncool thing: more than 2% of the pockets were lost on the route. This, again leads to “Connection timed out” screens on the users end.

So, at the moment of this post there’s a great issue with the European traffic routers, which means for webmasters who rely on European traffic that they see a significant traffic drop. The bad thing is that we can do absolutely nothing about it, nothing at all, just to wait until someone fix the routers.

If we get permission, we’ll publish a graph from InternetTrafficReport too, just to show you how serious the problem is

Related posts brought to you by Yet Another Related Posts Plugin.