The following allows all robots to visit all files because the wildcard "*" specifies all robots. User-agent: * Disallow: The next one bars all robots from the listed directories: User-agent: * Disallow: /catalogs/