# babelfish.com
#
# AI crawlers are ALLOWED here, deliberately. Bing's index feeds ChatGPT's web
# search, and this site's Bing channel is stable brand traffic (~32 clicks/day,
# 5.6% CTR, no decline). Being cited by an assistant is the same opportunity that
# organic search used to be, so blocking these agents would forfeit it.
#
# Abuse is handled where it actually happens instead: crawlers GET HTML, scrapers
# POST to /api/translate. That endpoint is rate-limited per real client IP and
# sits behind a durable daily character budget that fails closed. Blocking
# user-agents in this file would not have stopped the ~$2,000 scraping incident
# and would cost us the citations.
#
# STRUCTURE: each named crawler gets its own group because a named group REPLACES
# the "*" group wholesale rather than adding to it -- so "Disallow: /api/" has to
# be repeated in every group, or that agent would inherit no restriction at all.
# The named groups are otherwise identical to "*"; they exist so that a future
# blanket block added to "*" cannot sweep these agents up by accident.

User-agent: *
Allow: /
Disallow: /api/

User-agent: bingbot
Allow: /
Disallow: /api/

User-agent: BraveBot
Allow: /
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /api/

User-agent: Claude-SearchBot
Allow: /
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: Google-Extended
Allow: /
Disallow: /api/

Sitemap: https://www.babelfish.com/sitemap.xml