User-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: /addcomment # Redundant content for non-humans Disallow: /polls # They're an experiment, not for real use Disallow: /rot # No movies for bots Disallow: /rand # Really old URL but still indexed # Reindex way too much and don't see point anyways User-agent: TurnitinBot Disallow: / # It doesn't like lots of links, it apparently needs the * duplicated too User-agent: googlebot Disallow: /fp-index.shtml Disallow: /addcomment # Redundant content for non-humans Disallow: /polls # They're an experiment, not for real use Disallow: /rot # No movies for bots Disallow: /rand # Really old URL but still indexed # This one is just annoying... User-Agent: sitecheck.internetseer.com Disallow: / # No archiving pictures User-agent: ia_archiver Disallow: /pics Disallow: /addcomment # Redundant content for non-humans Disallow: /polls # They're an experiment, not for real use Disallow: /rot # No movies for bots Disallow: /rand # Really old URL but still indexed