public abstract class RobotsPolicy extends Object
Modifier and Type | Field and Description |
---|---|
static Map<String,RobotsPolicy> |
STANDARD_POLICIES |
Constructor and Description |
---|
RobotsPolicy() |
Modifier and Type | Method and Description |
---|---|
abstract boolean |
allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
String |
getPathQuery(CrawlURI curi) |
abstract boolean |
obeyMetaRobotsNofollow() |
public static Map<String,RobotsPolicy> STANDARD_POLICIES
Copyright © 2003–2019 Internet Archive. All rights reserved.