# https://www.robotstxt.org/robotstxt.html # https://leadpoet.com User-agent: * Allow: / # Don't crawl authenticated app surfaces or API endpoints. Public auth and # recovery routes are intentionally not blocked here: their HTML responses # carry noindex, and crawlers must be able to fetch that directive to remove # stale results from the index. Disallow: /api/ Disallow: /admin Disallow: /admin/ Disallow: /home Disallow: /onboarding Disallow: /dashboard # Public marketing pages and well-known agent discovery endpoints are # explicitly allowed for crawlers and AI agents. Allow: /.well-known/ Sitemap: https://leadpoet.com/sitemap.xml