User-agent: 008 User-agent: ahrefsbot User-agent: baiduspider User-agent: bygglobot User-agent: comodospider User-agent: discobot User-agent: docomo User-agent: dotbot User-agent: exabot User-agent: ezooms User-agent: findlinks User-agent: garlikcrawler User-agent: hapax newscrawler User-agent: huaweisymantecspider User-agent: mj12bot User-agent: mlbot User-agent: nerdbynature.bot User-agent: nigma.ru User-agent: pflab User-agent: psbot User-agent: purebot User-agent: rankurbot User-agent: scoutjet User-agent: sistrix crawler User-agent: sitebot User-agent: speedy spider User-agent: testnutch User-agent: trends crawler User-agent: unchaos real time trends bot User-agent: yandexbot User-agent: yandeximages User-agent: web.idrc.ca/challenge User-agent: webvac User-agent: COMODOspider/Nutch-1.0 Disallow: / User-agent: bingbot Crawl-delay: 2 Disallow: /*?publicationPageNumber= Disallow: /*?print= Disallow: /*?researchersPageNumber= Disallow: /*?pageNumber= Disallow: /*?siteMap= User-agent: SiteSeekerCrawler/1.0 Disallow: /www.googletagmanager.com/*