[Rack] robots.txt

Andy Isaacson adi at hexapodia.org
Tue Apr 30 21:38:20 UTC 2013


On Tue, Apr 30, 2013 at 02:31:37PM -0700, Ben Kochie wrote:
> I added a robots.txt to https://noisebridge.net
> 
> User-agent: *
> Disallow: /wiki/Help
> Disallow: /wiki/MediaWiki
> Disallow: /wiki/Special:
> Disallow: /wiki/Template
> Disallow: /wiki/skins/
> 
> I noticed bingbot is uselessly crawling the entire contents of
> Special:RecentChanges.

Is robots.txt a prefix, or a directory based exclusion scheme?  Will
"Disallow: /wiki/Special:" cause bingbot to skip
"/wiki/Special:RecentChanges"?

-andy


More information about the Rack mailing list