What is a recursive DNS query?

One of the most important part of the Internet is the DNS; I guess that’s clear for everybody. DNS is a distributed database, the largest in the world but in — at least — one aspect is extremely vulnerable: recursive queries. We distinguish two type of DNS queries:

1. Iterative or non-recursive
2. Recursive

Each 13 root server is configured to be non-recursive. When you query a root-server, it will return the answer for your query, or won’t return anything. The gTLD servers are also configured to be iterative.

Example: You query k.root-servers.net for records about google.com. In case it has information regarding your query, it will return data about which gTLD server should you query for further details about google.com. In case it has no information, it will NOT query another server for further information, but will return no data.

So what is a recursive DNS query?

Recursive DNS query is, when you query a DNS server, that is set to query other DNS servers until you get the requested information.

What are the risks of the recursive DNS?

When your DNS server is configured to support recursive queries, you should be aware of the following risks:

• DOS vulnerability: supporting recursive query may allow a client to flood one single IP with so many requests, that it can not be processed
• DNS-cache poisoning: since by default every result is cached, if a DNS server which supports recursive queries receives a fake/incorrect answer which is believed to be authoritative, it stores the answer in its cache, thus the cache is poisoned
• Resource hijacking: Imagine thousands of recursive queries in the same time. It will consume the server’s resources, thus will deny the service or, in better cases, performance degradation will occur
• Unnecessary load on root-servers: on DNS servers supporting recursive queries, if a query comes from a private address (see RFC 1918), the server is forced to query one of the root-servers

Who should support recursive DNS queries?

The Internet Service Providers has to provide recursive queries for its clients, but these DNS servers should never be accessible by those who are not the clients of the provider. Large corporations’ private networks’ DNS server also has to offer support for recursive queries.

How to end a server’s support for recursive DNS queries?

• Windows:
  Open a Remote Desktop connection to the server and login as Administrator. Open the Administrative Tools from the Start Menu/Control Panel, double-click the icon labeled DNS. Select from the tree on the left the DNS server you’d like to stop recursion on, then click Properties. Select the Advanced tab, then under the Server Options check the box which is labeled Disable Recursion.
  Save everything and as a final step clear the cache of the DNS.
• Unix-based (BIND 9):
  SSH to the server and login as root. Open for editing named.conf and in the Options section of the file insert the following line, then save/close the file:
  recursion no;
  Restart the named service with the following command:
  service named restart

To test whether your DNS supports recursive DNS queries or not, IANA (Internet Assigned Numbers Authority) has provided a Cross-Pollination Check tool which does a great job.

Google server running cPanel

While setting up a web-server is not a big deal, cPanel makes the whole process so much easier. Seemingly, Google takes the advantages of this software, too. We already knew that Google uses the open-source and so-popular Apache web-server, most likely because it can be adapted to various situations and has the lowest footprint yet amazing capabilities. But that they use cPanel is a new thing, here’s the proof:

cPanel is good for beginner server administrators, but for professionals it’s like a WYSWYG editor for the web-developers. It’s good cos it’s simple to use, bad, because it adds one more daemon process to the persistent processes list, which is less than ideal for servers which are bombarded with so many connections like the Google servers.

Another thing which is interesting: as I stated above cPanel makes setting up web-servers extremely easy. But why do Google need this help when the employment requirements for a server administrator job are the following :

• BA/BS in Computer Science or related field, or equivalent experience.
• 3 - 5 years experience with UNIX systems administration (5-15 years for Senior position).
• Solid scripting skills in Shell, PHP, Perl or Python.
• Proven technical troubleshooting and performance tuning experience.
• Experience in a high-volume or critical production service environment.
• Ability to handle periodic on-call duty as well as out-of-band requests.
• Tack-sharp analytical abilities.
• A strong sense of ownership, urgency, and drive.
• Fluent written communication and unusual verbal agility are strong assets.
• SQL experience a plus, MySQL a plus.
• Experience leading short projects involving outside teams is a plus.

The picture was taken when we typed “google.com” in the address-bar of Firefox and we resolved to that page. This can be caused by they putting the server too fast in the cluster, even before they could mirror the homepage’s software.

Significant traffic drop on 30th of november or where did my traffic go?

If you rely on European traffic, I guess you observed, too, that on 30th of November a great per cent of your traffic has gone… well, nowhere, but the users couldn’t reach your site hosted in the US.

Before we figured out what’s happening, we followed the golden route to debug the issue, nonetheless to say without any result, if you read the whole article you’ll understand why.

First, we checked the number of the incoming connections. It was significantly lower than the usual. A pattern could be observed, though, specifically that no user was coming from west European countries like France, United Kingdom, Germany, Spain and so on. This is weird because usually when there’s an issue with the webserver, the whole traffic is affected not only a certain part of the World. But it can still indicate a DNS issue — like records not propagating correctly — or, in certain cases problem with the firewall, for example the firewall deliberately started to ban some routers as an answer for an imaginary DOS attack.

So, after figuring out the above mentioned pattern, we checked the DNS records. There seemed to be no problem with them and were not even updated in the past few months. That can not be the problem, since records will get propagated again only if we make changes to the DNS records.

The firewall’s ban tables also seemed to be clean of issues. Only the bad robots and some aggressive spammers’ IP were in the table. Next we checked the Login Failure Daemon (LFD) to see if by any chance there are too many bans set due to login failures. But the table was also clean.

The next step was to examine the uplink provider’s connection. We are linked to the Internet through 6 providers, including Level3, Savvis and Verio, but the reports showed all of them are working on optimal capacity. But! Here comes an interesting thing: there was a huge traffic drop on some of the provider’s traffic graphs (see graph on left). This isn’t normal, usually the traffic’s graph is a very nice sinus curve, so the outage is not normal, at all. But let’s move forward. So far we know that the issue is not on our side, and is likely that a 3rd party has issues. Now the only thing we have to figure out is that who’s the third party? This isn’t that easy as it sounds as the uplink providers don’t offer direct support for webmasters, they talk only with the datacenter the server is located in so contacting them is pointless because we either don’t receive an answer, or we will receive an answer like “Ask the DC, not us”. None of them is good for us at any time.

But we can do monitor remotely the global routers without contacting anyone. This is more than cool, because if there’s no reported issue with the routers, than the uplink providers have issues, else (some) of the routers.

To see what’s happening on the routers’ end, we can go to http://www.internettrafficreport.com/ and take a peek on the graphs which shows the global internet traffic’s details. Since we were interested only in the European traffic, we looked only on the European routers’ details. And we’ve found what we’ve been looking for: the average response time of the European router’s were in the skies, a bit below 195ms, which at the time of this post is way above 210ms. The value considered to be normal is around 150ms, 60 additional milliseconds can mean some really great issues for the users because they most likely will see only “Connection timed out” screens. Looking on the “Packet Loss” graph, we observed another uncool thing: more than 2% of the pockets were lost on the route. This, again leads to “Connection timed out” screens on the users end.

So, at the moment of this post there’s a great issue with the European traffic routers, which means for webmasters who rely on European traffic that they see a significant traffic drop. The bad thing is that we can do absolutely nothing about it, nothing at all, just to wait until someone fix the routers.

If we get permission, we’ll publish a graph from InternetTrafficReport too, just to show you how serious the problem is

Ban IP on server level after a number of unsuccessful logins

This is a widely used function amongst the server managers. Depending on your system configuration, the server will ban the enforcer’s IP, either putting it in the firewall’s deny list or, on Linux servers with IPTables installed, will put the IP in the drop list.

To achieve this feature, the easiest way is to install a software firewall. My recommendation is CSF, Configserver Security & Firewall developed and maintained by Way To The Web Limited. It’s an extremely efficient software, and integrated into WHM is very easy to manage it, even a beginner can handle almost everything.

So, if configured correctly, CSF has a, say, extension: LFD or Login Failure Daemon.

This stuff is what we search for at the moment. You can configure which ports to listen on, so if the enforcer tries on SSH, POP3 or FTP and even HTTP authentication, it will get banned after a few tries.

You can also put IPs to its ignore list. This is very useful feature if you don’t want to get yourself banned.

Restrict access to directory or domain by IP, using .htaccess

I don’t blah too much on this subject.
Basically, you can restrict or allow who can connect to your site or who can access specific directories using .htaccess .

Here’s the code to block one specific IP, I use 192.168.0.1 to block, you replace it with the IP you want to deny.


order allow,deny
deny from 192.168.0.1
allow from all


That is. Placed in the root of your site, the user with the IP 192.168.0.1 will not be able to access your site at all. If you place it in a specific subdirectory of your site, the user won’t access the specific subdirectory. If you want to put more IPs in your deny list, just add one more deny line for each IP.

To block by domain, replace the IP with a domain. For example:


order allow,deny
deny from .comcast.net
deny from .google.com
allow from all


If you look hard, you will observe that I put a dot in front of the domains. It has only one meaning: if you put a dot in front of the domain, all the sub-domains will be blocked. For example, in the second deny rule i said to deny everybody from google.com, including www.google.com, googlebot.google.com, finance.google.com, you get it.

And as always, we saved the world again.

Hotlink Protection using .htaccess made easy

This is one of the most used tricks by the webmasters who care about their allocated bandwidth. The code which controls what are domains where your images can show up is very short, 4 line that is.
As always, I provide the full code, then below it I explain everything.
To use this code, you have to have an Apache web-server with mod_rewrite correctly installed.
So, let’s see the code for those who don’t want the explanations:


RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?example.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]


Now some explanations:
The first line,

RewriteEngine on

practically tells Apache we will do something with mod_rewrite so turn it on. This line is optional if you already turned it on before in the same .htaccess where you put the above code in.
The second line,

RewriteCond %{HTTP_REFERER} !^$

this is nastier. Basically, if there is no referrer, let the image to be displayed. I guess this needs a bit of explanation. When you navigate on the internet from one site to the other, the browser always sends a “referrer” header to the host you are accessing. So, for example if you are currently on http://www.Google.com and you navigate to http://yahoo.com, the browser will send yahoo the following : “Referrer: http://www.google.com”. This header is what we use in our .htaccess to prevent hotlinking, BUT! Some antiviruses, firewalls clears this header on the clients’ side so there is no referrer at all, thus we don’t know the user browses our site, or it’s hotlinking our image on another site. Thus we just let the image to be displayed if there is no referrer.

RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?example.com [NC]

If the referrer domain is our own domain, display the image. We set: http(s)?://(www\.)?yourdomain.com, so our condition will work on HTTP, HTTPS and also on our www and non-www hostname/domain.

And the last step is to tell Apache which files to protect:

RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

In the above case the jpg, jpeg, png and gif images will be protected. If you want to protect your Flash-files as well, put swf in the list and your movies will not display embedded in remote sites.
On our domain the php files are also protected because the Imagick examples are parsed by php codes.

I hope the above example was somewhat useful, if you need help with it, just say your problem below and will answer as soon as possible.

Filter your variables easily but like a pro!

How painful input validation is! Think about all the possible threats, combination of threats… think with the users’ mind. It’s a pain. And usually who can write scripts which filters effectively the user inputs is considered a pro, without hesitation. Just because it’s hard to do it.
Take the following scenario: you have a text-field which accepts text as user comment. You don’t want to let the user to use HTML in the comment box, and definitely not to allow the user to put javascript in the comment.
So how do you sanitize the string you get? It’s a long and hard way. You would use RegExp to exclude some entities then some php inbuilt functions to encode the remaining or even better to strip tags.
I show you an easier way:

filter_var(’<script>alert('Hello');</script>', FILTER_SANITIZE_STRING);

Done, the <script> tags will be stripped so the string will arrive in the database as alert(’Hello World’).
There are many available filters, just to mention the most interesting ones:

• FILTER_SANITIZE_EMAIL — it sanitizes email address, strips characters which are not in conformance with the applicable RFC (link)
• FILTER_SANITIZE_URL — whether the URL from the variable is in conformance with the applicable RFC (link)
• FILTER_VALIDATE_IP — whether if the input is an IP address or not

I recommend using the filter_var() function and its filters for two obvious reasons: it saves you a lot of headaches and saves you time. Even though the filter_var function was introduced only in php 5.2 the function is extremely useful and gives another reason for you of why to upgrade to php5

For a complete reference please check php.net.

Installing ImageMagick without headaches

This is simple: on windows systems you have the option to use the automated installer here, it’s neither complicated or hard to install. On Linux based systems I recommend to install it from source for obvious reasons like you know what’s installed, where, and if there was a problem, you can debug easily because the log is displayed.
So, login using SSH, you will need to login with a username which has compiling and install privileges.
After you logged in, create a directory where you place the gzipped package you will download from Imagemagick.org.
after you created the directory, type in the console:

wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick-6.4.2-1.tar.gz

Currently (13th of July, 2008) 6.4.2 is the latest version of ImageMagick, you may want to check what is the latest release when you install your ImageMagick.
So, the above command will download the imagemagick package to your server.
Let’s depack it using tar, tar, not untar as there’s no such command on Linux

tar xvfz ImageMagick-6.4.2-1.tar.gz

Depacked. Change the directory to the newly created ImageMagick directory, then let’s configure and compile it:

./configure
make


If everything was OK, no error, warning or any other weird stuff during compile, you are safe to install it:

make install


And theoretically you are done. To test your installation try Imagemagick’s convert program, if it works, you’re the best! Type the following in the command line:

/usr/local/bin/convert logo: logo.gif


ImageMagick is a dependency for Imagick PECL, so before you can use it in PHP for example, the first step is to install ImageMagick THEN Imagick.

Tightening PHP

OK, let’s see what’s my point of view on this subject. Please note that nothing I say is a must, probably others would do in another way what I did, many would do things better or less good.
First of all, I would like to say a “thank you” to all of those who I manage their servers and have cPanel/WHM installed. Since the folks from cPanel integrated EasyApache in their software, the job got much easier for us. Less work, more time to drink coffee.

So. My very first suggestion is to install PHP as a mod. For example, on Apache there is mod_php4 or mod_php5. Security holes are in both setup, whatever you do, but if you install PHP as a mod, at least you have less headaches. Maybe it’s just me, but when installed as CGI it was a nightmare. First of all, the developer will have to work in the cgi-bin directory (if the web-server is well configured), secondly, sometimes you have to place the php executable in the CGI directory which is not good… at all. Thirdly, why bring even more security holes in the server by involving a CGI wrapper if PHP already has enough? Probably the last is the best reason, and please, don’t tell me the wrappers don’t have security holes.

OK, let’s step forward. We still remain at Apache, two other mods: mod_suphp and mod_security. None of these is a must, it depends on your needs and on what will you do with your server. mod_security is “just” a software firewall, it can do many things, live-protection, monitoring, etc, there are a lot of features, better have a read here on mod_security official page. it’s pretty cool, but if you already have a firewall installed like CSF, probably is just a waste of resource. On the other hand, mod_suphp will execute php scripts as the script’s owner. OK, will try to explain: php is called by default “nobody”. This can vary on systems as it can be changed easily. So, by default, every PHP script is executed as “nobody”, say “nobody” will be the owner of all PHP scripts. Sounds weird. Let’s say, you run many virtual hosts on your server, example1.com is John’s, your best friend from school, Ran, your girlfriend’s ex has example2.com on your server. If suphp is not installed (or not working correctly), if Ran or John writes a script and runs on your server, it will be run as “nobody” as, remember, by default every PHP script is nobody’s. If you have suphp installed, then Ran’s PHP scripts will be run as Ran’s, John’s as John’s, and so on. Why is good to have it installed? Well, everyone will be able to execute its own script but not the others’. Again, an example: John will be able to execute his own PHP scripts but not Ran’s and vice-versa, and definitely John will not be able to execute nobody’s scripts. This tool is a must if the server will not be used only by you, else you can live without it, but you will have a security hole!

Install always the most up to date and STABLE release of PHP. Why everyone thinks that this is not good? The folks from PHP releases new versions cos the fixed or upgraded something, not cos they were bored. If you think your users’ PHP scripts won’t be compatible with the new PHP, first let them know, give ‘em a few days to fix their codes, then throw them in the deep water and upgrade PHP. Sincerely, I saw the other day a webhost which still used PHP 3. When I asked their live support why the php 3, they said, well, thy poor girl from the support desk that their users aren’t prepared for the upgrade yet. Yup, 2 or 3 years isn’t enough time. Too bad PHP 6 is knocking on the door.

Next step, let’s disable functions which might be dangerous on production servers! Let’s give a straight-forward example: you definitely don’t want your users who are all potential black-hat hackers to be able to run commands on your system like
format c:
Right? So let’s disable some inbuilt PHP functions, consider the following:

ini_set
system
show_source
shell_exec
passthru
exec
proc_open
popen

system, exec, passthru, shell_exec, proc_open, popen are related to command line, thus are good to have them disabled, remember, format c:. strong>ini_set, you already configured your PHP, why would you let others to reconfigure it? Finally show_source, would you like to show your PHP script’s source to the hackers?
You may also disable phpinfo, as you don’t really want to let others know what is your PHP configuration. If you don’t want to send e-mails from your script, you may want to consider disabling the mail function as well.

Ok, another one: fopen() . Did you know it can be used to open remote files? For example to open a YouTube page and view through your webpage. That’s the less severe case anyway. It could be used to place exploits on your server or for many other purposes. So
allow_url_fopen = Off

That’s for now, I will talk later about how and when to limit the server resources PHP can eat up.

Stop network flood in one step

Ok, two

First connect to your server’s private network using SSH and login as root. Your data center should provide you how can you do this, on Softlayer servers you have to create first a VPN tunnel to the Softlayer network then you can connect to the server’s private IP.

Usually a server has two network adapters: one for public traffic and one for the server’s private network. As You are smart and know the server’s configuration, must know how these adapters are named, anyway, usually eth0 for private network and eth1 for public.

If you are connected to the server’s private IP and only then, type in the command-line:


ifdown eth1


This command will shut your server’s public network down, thus closing all connections.
To fire the public network up:


ifup eth1


Why did I warn you about to not do this if you are not connected to the private network? Well, think a bit: if you close the public adapter and you manage the server through this network, how will you reconnect if the public network is shut down?

If you can connect only through the public network, you can try to just simply restart the public network’s adapter using the following command:


/etc/init.d/network restart


To check at any time your public network adapter’s status:


ifstatus eth1


Next Page »