User-agent:ia_archiver Disallow: / User-agent: * Allow: /All-Pdfs/TSB/ Disallow: /data/ Disallow: /All-Pdfs/ Disallow: /GIF/ Disallow: /gps/ Disallow: /jwplayer/ Disallow: /nfo-linux/ Disallow: /retired/ Disallow: /temp/ sitemap: http://www.fixkick.com/sitemap.xml # rules # allows must be first. # default, is allow all # http://www.robotstxt.org/norobots-rfc.txt # no empty lines # the first matching pattern always (most) wins #Google spec. is diffr. # At a group-member level, in particular for allow and disallow directives, # the most specific rule based on the length of the [path] entry , # will trump the less specific (shorter) rule. # so my TSB wins, can bet tested, easy. # http://www.freesitemapgenerator.com (no CR/LF on last line malarkey either)