public class ObeyRobotsPolicy extends RobotsPolicy
Modifier and Type | Field and Description |
---|---|
static RobotsPolicy |
INSTANCE |
STANDARD_POLICIES
Constructor and Description |
---|
ObeyRobotsPolicy() |
Modifier and Type | Method and Description |
---|---|
boolean |
allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
boolean |
obeyMetaRobotsNofollow() |
getPathQuery
public static RobotsPolicy INSTANCE
public boolean allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
allows
in class RobotsPolicy
public boolean obeyMetaRobotsNofollow()
obeyMetaRobotsNofollow
in class RobotsPolicy
Copyright © 2003–2019 Internet Archive. All rights reserved.