User-agent: ia_archiver Disallow: / User-agent: Yandex Disallow: / User-agent: Baiduspider Disallow: / User-agent: Bytespider Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / # Disallow directory /cgi-bin/ User-agent: * Disallow: /cgi-bin/ # Disallow directory /bin/ User-agent: * Disallow: /bin/ # Disallow members zone access User-agent: * Disallow: /members/ # Disallow poland.html User-agent: * Disallow: /poland.html