HOWTO: Serving up default favicon.ico and robots.txt files with apache

As a webmaster, or system administrator of an Apache webserver, did you ever wonder how to stop those annoying 404 (page not found) errors for all the requests for favicon.ico and robots.txt? For your own site, you could of course just create a favicon.ico and a robots.txt file yourself. (See links for instructions. More info for favicon and robots.txt from Wikipedia, the free online encyclopedia.) But what if you administrate a server hosting sites for others? You can't just stick these files into every site. The webmasters of those sites will delete the files, or bother you with questions about them, etc. You also wouldn't want to blindly remap all requests for these files to your own copies. Again, the pesky webmasters would raise arms against you as soon as they found out they were unable to use their own versions of these files.

There is a way to have apache server up a default favicon.ico, or a default robots.txt file, only for sites that don't supply their own! You can do so with mod_rewrite. First, you need to have the module compiled in to your apache, or built as a dynamic module and set to load in the httpd.conf. See the docs in the prior link or this one for info on doing this. Note that I have only done this under Apache 1.3.x. Though, I believe mod_rewrite will work under 2.x.

Before setting up mod_rewrite you need some versions of these files to use as defaults. First, you will need a default favicon.ico. This could be a blank one like this (right-click Save As...), or anything you choose. But remember this is going to be served up for every site on your server that does not provide its own (and for which you include the rule in the VirtualHost section.) You will also need some kind of default robots.txt file like this.

Put the default robots.txt and favicon.ico files someplace where the server can get to them (usually under DocumentRoot someplace).

Once you have mod_rewrite setup, and the default files in place, you need to include some rules to handle our problem. For each VirtualHost you need to tell mod_rewrite to serve up the default if, and only if, there is not one already present.

Here are the rules to do this:

<VirtualHost xxx.xxx.xxx.xxx >
  ServerAdmin me@mydomain.com
  DocumentRoot /var/web/myhost
  ServerName www.mydomain.com

  ...

  RewriteEngine On
  RewriteCond  %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule  .*favicon\.ico$         /var/web/favicon.ico [L]   
  RewriteRule  .*robots\.txt$         /var/web/robots.txt [L]  

  ...

</VirtualHost>

Note that you will have other values for the VirtualHost ip address, and the ServerName, etc. The important parts here are the lines starting with "Rewrite". Those are the rules for mod_rewrite. You will need to set the filenames in those rules /var/web/favicon.ico and /var/web/robots.txt to point to the default files to be served (i.e. the ones you downloaded above). Make sure those files are readable by the webserver.

Note: If you are using mod_alias and have trouble with these rules please see the mod_rewrite docs for a discussion of the -f vs. the -U test. You may also consider the -s test instead of the -f test.

You may wish to edit the mime.types file used by your copy of Apache to be sure it is serving up the favicon.ico with the correct mime type. If the mime type is setup correctly then Firefox will show you the icon right in the browser when you view the file directly (by entering the full url ending in favicon.ico into the location field in the browser.) I have no idea if Internet Explorer will, but you wouldn't be so foolish as to be using IE, unless you want your computer security compromised... To edit the mime.types, search through your httpd.conf file to see which mime.types file Apache is using. Then add the following line to that file if it is not already there:

image/x-icon                    ico

Your Apache webserver logs should now be free of those annoying 404 Not Found errors for all the requests for favicon.ico and robots.txt!

This HOWTO Brought to you by LAFFEY.tv - Special Effects for Film and Television also producers of free Digital Fusion plugins.
Please do NOT reprint this HOWTO, or mirror it without expressed written permission. Instead, feel free to link to this page directly.