# robots.txt User-agent: * Disallow: /login/ Disallow: /registration/ Disallow: /search/ Disallow: /account/ Disallow: /basket/ Sitemap: https://www.capublicnotice.com/sitemap.xml Crawl-Delay: 20 User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SemrushBot-CT Disallow: / User-agent: SemrushBot-BM Disallow: / User-agent: AhrefsBot Disallow: / User-Agent: MJ12bot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: MojeekBot Disallow: / User-agent: CCBot Disallow: / User-agent: LEXBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: GetHPinfo.com-Bot Disallow: / User-agent: YandexBot Disallow: / User-Agent: trendictionbot Disallow: / User-Agent: Veooz Disallow: / User-Agent: WikiDo Disallow: / User-Agent: Yeti Disallow: / User-Agent: SeznamBot Disallow: / User-agent: grapeshot Disallow: / User-agent: IstellaBot Disallow: / User-agent: AnyBot Disallow: / User-agent: AAABot Disallow: / User-agent: PetalBot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: GnowitNewsbot Disallow: / User-agent: Re-re Studio (+http://vip0.ru/) Disallow: / User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) Disallow: /