Sitemap: https://www.shipindex.org/sitemap.xml Sitemap: http://blog.shipindex.org/sitemap.xml # Default rules. Applies to every crawler not named in a group below, which # deliberately includes the search engines that send us readers: Googlebot, # Bingbot, Applebot, DuckDuckBot and Yahoo's Slurp all land here and are free # to crawl the whole public site. # # The Disallow patterns close the faceted-filter URL space. Ship, vessel and # resource pages accept filter parameters (resource_*, has_image, # has_passenger_list, has_main_entry, filter, showall, search_scope, hide_types) # that combine freely, so the number of distinct crawlable URLs is unbounded. # Those pages already emit back to the unfiltered page, # but a crawler has to fetch and render the page before it can read the # canonical tag — so canonicalisation stops the duplicates being indexed # without stopping the crawl. Only these rules stop the fetch. # # Pagination is deliberately NOT disallowed: /ships/:slug/page/N and # /ships/alpha_/:letter/:page are path segments, not query strings, and the # alpha pages are listed in the sitemap. User-agent: * Disallow: /admin/ Disallow: /ships? Disallow: /ships/*? Disallow: /vessels? Disallow: /vessels/*? Disallow: /resources? Disallow: /resources/*? # Sogou ignores Disallow often enough that it is also refused by IP in # config/initializers/rack_attack.rb, but it does respect Crawl-delay. User-agent: Sogou web spider Crawl-delay: 10 # --------------------------------------------------------------------------- # AI crawlers and answer engines: throttled, not refused. # # These operators publish verifiable egress IP ranges, declare themselves # honestly in the User-Agent, and honour robots.txt. They send few or no # readers back, so they get the same treatment as any well-behaved crawler we # simply do not want in the expensive part of the site: keep out of the # faceted URL space, and slow down. Rack::Attack additionally rate-limits # their published ranges to roughly a fifth of the normal per-IP allowance. # # Adding a "Disallow: /" line to this group would refuse them outright. That is # a data-rights decision rather than a traffic one, so it is left alone here. # --------------------------------------------------------------------------- User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: anthropic-ai User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Amazonbot User-agent: YouBot User-agent: CCBot Disallow: /admin/ Disallow: /ships? Disallow: /ships/*? Disallow: /vessels? Disallow: /vessels/*? Disallow: /resources? Disallow: /resources/*? Crawl-delay: 5 # --------------------------------------------------------------------------- # Refused outright. # # SEO backlink and keyword scrapers, and commercial content resellers. These # crawl heavily and return no readers. DotBot, SemrushBot, PetalBot and # Meta-ExternalAgent were all measured in production traffic; the rest are the # same class of backlink scraper and are listed pre-emptively. # # Bytespider is here because it has a long record of ignoring robots.txt # entirely, so this line is a formality — the IP rules are what stop it. # --------------------------------------------------------------------------- User-agent: dotbot User-agent: SemrushBot User-agent: PetalBot User-agent: meta-externalagent User-agent: Meta-ExternalAgent User-agent: AhrefsBot User-agent: MJ12bot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: serpstatbot User-agent: MegaIndex User-agent: SeekportBot User-agent: Bytespider User-agent: Diffbot User-agent: Omgilibot User-agent: ImagesiftBot User-agent: Timpibot Disallow: /