# Crawlers that send no search or answer traffic back. Training scrapers and SEO tool # indexers made up most of the ~120k requests/day CloudFront served in October 2026, # against ~500 real users/day. Search engines and AI answer engines are not listed. User-agent: ia_archiver User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: serpstatbot User-agent: Barkrowler User-agent: PetalBot User-agent: Bytespider User-agent: CCBot User-agent: meta-externalagent User-agent: ImagesiftBot User-agent: Amazonbot Disallow: / User-agent: * Disallow: /cms/ Disallow: /django-admin/ Disallow: /api/ Disallow: /analytics/ Disallow: /teams/personal/ Disallow: /team/*/edit Disallow: /team/*/view Disallow: /nav-fragment/ # Sort/filter/tab/search variants of the listing pages. Every one canonicalises to the # path without its query string, and the combinations (x 8 languages x every species' # teams tab) are an effectively unbounded crawl space. The longer Allow below wins for # tournament pagination, which is how crawlers reach older tournaments. Disallow: /*? Allow: /tournaments/?page= Allow: / Sitemap: https://pokepro.co/sitemap.xml