public class FirstNamedRobotsPolicy extends RobotsPolicy
Modifier and Type | Field and Description |
---|---|
protected List<String> |
candidateUserAgents
list of user-agents to try; if any are allowed, a URI will be crawled
|
protected boolean |
obeyMetaRobotsNofollow
whether to obey the 'nofollow' directive in an HTML META ROBOTS element
|
protected boolean |
shouldMasquerade
whether to adopt the user-agent that is allowed for the fetch
|
STANDARD_POLICIES
Constructor and Description |
---|
FirstNamedRobotsPolicy() |
Modifier and Type | Method and Description |
---|---|
boolean |
allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
List<String> |
getCandidateUserAgents() |
boolean |
getShouldMasquerade() |
boolean |
isObeyMetaRobotsNofollow() |
boolean |
obeyMetaRobotsNofollow() |
void |
setCandidateUserAgents(List<String> candidateUserAgents) |
void |
setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow) |
void |
setShouldMasquerade(boolean shouldMasquerade) |
getPathQuery
protected List<String> candidateUserAgents
protected boolean shouldMasquerade
protected boolean obeyMetaRobotsNofollow
public boolean getShouldMasquerade()
public void setShouldMasquerade(boolean shouldMasquerade)
public boolean isObeyMetaRobotsNofollow()
public void setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow)
public boolean allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
allows
in class RobotsPolicy
public boolean obeyMetaRobotsNofollow()
obeyMetaRobotsNofollow
in class RobotsPolicy
Copyright © 2003–2019 Internet Archive. All rights reserved.