public class CustomRobotsPolicy extends RobotsPolicy
Modifier and Type | Field and Description |
---|---|
protected ReadSource |
customRobots
textual alternate robots.txt rules to follow
|
protected Robotstxt |
customRobotstxt |
protected boolean |
obeyMetaRobotsNofollow
whether to obey the 'nofollow' directive in an HTML META ROBOTS element
|
STANDARD_POLICIES
Constructor and Description |
---|
CustomRobotsPolicy() |
Modifier and Type | Method and Description |
---|---|
boolean |
allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
ReadSource |
getCustomRobots() |
boolean |
isObeyMetaRobotsNofollow() |
boolean |
obeyMetaRobotsNofollow() |
void |
setCustomRobots(ReadSource customRobots) |
void |
setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow) |
getPathQuery
protected Robotstxt customRobotstxt
protected ReadSource customRobots
protected boolean obeyMetaRobotsNofollow
public ReadSource getCustomRobots()
public void setCustomRobots(ReadSource customRobots)
public boolean isObeyMetaRobotsNofollow()
public void setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow)
public boolean allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
allows
in class RobotsPolicy
public boolean obeyMetaRobotsNofollow()
obeyMetaRobotsNofollow
in class RobotsPolicy
Copyright © 2003–2019 Internet Archive. All rights reserved.