# Google Search Engine Crawlers - Full Access User-agent: Googlebot Allow: / Crawl-delay: 1 User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Video Allow: / User-agent: Googlebot-News Allow: / # Microsoft Bing Crawlers - Full Access User-agent: Bingbot Allow: / Crawl-delay: 1 User-agent: BingPreview Allow: / # Other Major Search Engines User-agent: Slurp Allow: / Crawl-delay: 2 User-agent: YandexBot Allow: / Crawl-delay: 2 User-agent: Baiduspider Allow: / Crawl-delay: 3 User-agent: DuckDuckBot Allow: / Crawl-delay: 1 # Social Media Crawlers User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: Slackbot Allow: / # SEO and Analytics Bots User-agent: AhrefsBot Allow: / Crawl-delay: 5 User-agent: SemrushBot Allow: / Crawl-delay: 5 User-agent: MJ12bot Allow: / Crawl-delay: 10 User-agent: DotBot Allow: / Crawl-delay: 5 # Website Monitoring Bots User-agent: UptimeRobot Allow: / User-agent: Pingdom Allow: / User-agent: Site24x7 Allow: / # AI Training Crawlers with restrictions User-agent: GPTBot Allow: / Crawl-delay: 10 User-agent: ChatGPT-User Allow: / Crawl-delay: 10 User-agent: ClaudeBot Allow: / Crawl-delay: 60 User-agent: PerplexityBot Allow: / Crawl-delay: 10 # Block known malicious or aggressive bots User-agent: SemrushBot-SA Disallow: / User-agent: AhrefsBot Disallow: / Crawl-delay: 86400 User-agent: MJ12bot Disallow: / Crawl-delay: 86400 # Content scrapers and suspicious bots to block completely User-agent: SiteSnagger Disallow: / User-agent: WebCopier Disallow: / User-agent: HTTrack Disallow: / User-agent: WebZIP Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EmailCollector Disallow: / User-agent: WebBandit Disallow: / User-agent: WebCapture Disallow: / User-agent: WebSauger Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebRipper Disallow: / User-agent: WebWhacker Disallow: / User-agent: WebDevil Disallow: / User-agent: WebReaper Disallow: / User-agent: Widow Disallow: / User-agent: WWWOFFLE Disallow: / User-agent: Xenu Disallow: / User-agent: Zeus Disallow: / User-agent: Indy Library Disallow: / User-agent: libwww-perl Disallow: / User-agent: Download Demon Disallow: / User-agent: GetRight Disallow: / User-agent: FlashGet Disallow: / User-agent: Go-Ahead-Got-It Disallow: / # Block vulnerability scanners and security tools that can overwhelm servers User-agent: Nmap Disallow: / User-agent: Nikto Disallow: / User-agent: Sqlmap Disallow: / User-agent: w3af Disallow: / User-agent: OpenVAS Disallow: / User-agent: Nessus Disallow: / User-agent: masscan Disallow: / User-agent: ZmEu Disallow: / User-agent: LieBaoFast Disallow: / User-agent: MQQBrowser Disallow: / # Block bots with empty or suspicious user agents User-agent: "" Disallow: / User-agent: - Disallow: / # XML Sitemaps - Include your sitemap URLs here Disallow: /emailer/demo/index.html Disallow: /emailer/after15days/index.html Disallow: /emailer/closedleads/index.html Disallow: /emailer/detailsplan/index.html Disallow: /emailer/followupemail/index.html Disallow: /emailer/phonenotpickedbylead/index.html Disallow: /emailer/specificcommercial/index.html Disallow: /emailer/thanks/index.html Disallow: /signature/signature.html Sitemap: https://www.eximpedia.app/sitemap.xml # End of robots.txt file