Jason Kottke raved about the new Whitehouse dot gov robots.txt file a few days ago, completely going around the bend and establishing a new low for fawning (kottke.org/09/01/the-countrys-new-robotstxt-file). Despite how stupid that was, I didn't note it here due to the fact that Kottke isn't important. Now I see that the BBC has joined in.

For those who don't know what those files are for, they site at the root level of a domain and they indicate which files or directories should or shouldn't be indexed by search engines, how often search engines should look for pages, and so on. A preliminary (the only one I'm going to do) search at google shows they've indexed 116,000 pages at whitehouse.gov. I didn't review the old file, but as someone points out here, the ones that were excluded were just copies of pages that were already indexed. For instance, the text-only version of HTML pages. There might have been some funny business going on, but that doesn't make any sense because the same spider used by a search engine could be used by someone who was looking for funny business, and if they'd found "hidden" files (which were linked by other pages and thus accessible by a search spider) we would have heard about it.

However, now I find that the BBC - the British Broadcasting Corporation - has tried the same tack in an unbylined article (link):

Searching for data about the Obama administration should get easier as the Whitehouse.gov website gets overhauled.

Barack Obama's new media team is letting search engines index almost everything on the site.

By contrast, after eight years of government the Bush administration was stopping huge swathes of data from being searchable.

The move is part of President Obama's larger push to make the US government more open and transparent.

If you're familiar with these topics, you know how incredibly stupid and KoolAid-besotted these people are. If not, trust me: they're really stupid and KoolAid-besotted.