User-agent: admantx Disallow: / User-agent: grapeshot Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: BUbiNG Disallow: / User-agent: CompSpyBot Disallow: / User-agent: MaxPointCrawler Disallow: / User-agent: WikiDo Disallow: / User-agent: coccoc Disallow: / User-Agent: Flamingo_SearchEngine Disallow: / User-Agent: trendictionbot Disallow: / User-agent: WocBot Disallow: / User-agent: careerbot Disallow: / User-agent: Alexabot Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: rogerbot Disallow: / User-agent: Apexoo Spider 1.x Disallow: / User-agent: WBSearchBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-Agent: YoudaoBot Disallow: / User-Agent: yacy Disallow: / User-Agent: Backlink crawlers Disallow: / User-agent: attrakt Disallow: / User-agent: LucidWorks Disallow: / User-agent: Wotbox Disallow: / User-agent: seoengbot Disallow: / User-Agent: daumoa Disallow: / User-agent: MLBot Disallow: / User-agent: accelobot Disallow: / User-agent: Exabot Disallow: / User-agent: spbot Disallow: / User-agent: swebot Disallow: / User-agent: webcrawler Disallow: / User-agent: gigabot Disallow: / User-agent: AdBot Disallow: / User-agent: agbot Disallow: / User-agent: suggybot Disallow: / User-agent: TreEZy Disallow: / User-agent: AboutUsBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Netseer Disallow: / User-agent: proximic Disallow: / User-agent: Covario-IDS Disallow: / User-agent: Aboundexbot Disallow: / User-agent: Speedy Disallow: / User-agent: psbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Nutch Disallow: / User-agent: viglink/Nutch-1.0 Disallow: / User-agent: sistrix Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: GosoSpider Disallow: / User-agent: 008 Disallow: / User-agent: MJ12bot Disallow: / User-agent: YRSPider Disallow: / User-agent: CamontSpider Disallow: / User-agent: ccbot Disallow: / User-agent: purebot Disallow: / User-agent: pikimal Disallow: / User-agent: pik-a-part Disallow: / User-agent: dotbot Disallow: / # Wikipedia work bots: User-agent: IsraBot Disallow: / User-agent: Orthogaffe Disallow: / User-agent: UbiCrawler Disallow: / User-agent: Zao Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Xenu Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: wget Disallow: / User-agent: WebReaper Disallow: / User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: discobot Disallow: / User-agent: discoverybot Disallow: / User-agent: sitebot disallow: / User-agent: Bender disallow: / User-agent: Yandex Crawl-delay: 500 User-agent: Mediapartners-Google Allow: / User-agent: * # Crawl-delay: 10 # Directories Disallow: /follow_listing/ Disallow: /image_captcha Disallow: /image_captcha/ Disallow: /blkhol/* Disallow: /blkhol/ Disallow: /includes/ Disallow: /misc/ Disallow: /modules/ Disallow: /profiles/ Disallow: /profile/ Disallow: /profile/edit/ Disallow: /scripts/ Disallow: /site_search Allow: /sites/default/files/*.jpg Allow: /sites/default/files/*.jpeg Allow: /sites/default/files/*.png Allow: /sites/default/files/*/media Allow: /sites/default/files/*/media/ Allow: /sites/default/files/*/media/* Disallow: /sites/ Disallow: /themes/ # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /contact/ Disallow: /logout/ Disallow: /node/add/ Disallow: /search/ Disallow: /search/* Disallow: /user/* Disallow: /user/ Disallow: /user Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/login/edit_listing/ Disallow: /user/pending_listings/ Disallow: /sys_admin/ Disallow: /app_admin/ Disallow: /add_listing/ Disallow: /edit_listing/ Disallow: /edit_listing/* Disallow: /delete_listing/ Disallow: /delete_listing/* Disallow: /edit_assoc/ Disallow: /edit_assoc/* Disallow: /view_history/ Disallow: /view_history/* Disallow: /ajax_get_near_listings/* Disallow: /ajax_get_near_listings Disallow: /ajax_search/* Disallow: /ajax_search Disallow: /ajax_get_pricing_info/* Disallow: /ajax_get_pricing_info Disallow: /ajax_suggest_search/* Disallow: /ajax_review_helpful/* Disallow: /write_review Disallow: /write_review/* Disallow: /edit_approval/* Disallow: /b/edit/* Disallow: /publishers/* Disallow: /detail/* Disallow: /question/* Disallow: /directory/* Disallow: /w/ Disallow: /w/* Disallow: /ftb_embed.js/ Disallow: /ftb_embed.js/* Disallow: /slideshow/ Disallow: /slideshow/* Disallow: /search.aspx Disallow: /popups/* Disallow: /user_poll_ajax_submit Disallow: /rt* Disallow: /filters Disallow: /m/filters Disallow: /review Disallow: /theme_css/* # RPX Disallow: /rpx/ Disallow: /rpx/end_point/ Disallow: /rpx/end_point Disallow: /d/ Disallow: /m/d/ Disallow: /compares Disallow: /saved_compare/ Disallow: /compare/ Disallow: /l/ Disallow: /m/l/ Disallow: /q/ Disallow: /m/q/ Disallow: /app-questions/ Disallow: /m/app-questions/