Package com.yahoo.documentapi
Class ProgressToken
java.lang.Object
com.yahoo.documentapi.ProgressToken
Token to use to keep track of progress for visiting. Can be used to resume
visiting if visiting has been aborted for any reason.
- Author:
- Thomas Gundersen, vekterli
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
When a bucket has its state kept by the progress token, we need to discern between active buckets (i.e.static class
For consistent bucket key ordering, we need to ensure that reverse bucket IDs that have their MSB set actually are compared as being greater than those that don't.static enum
Any bucket kept track of by aProgressToken
instance may be in one of two states: pending or active. -
Field Summary
Modifier and TypeFieldDescriptionstatic final com.yahoo.document.BucketId
static final com.yahoo.document.BucketId
-
Constructor Summary
ConstructorDescriptionCreates a progress token.ProgressToken
(byte[] serialized) ProgressToken
(int distributionBits) ProgressToken
(String serialized) -
Method Summary
Modifier and TypeMethodDescriptionprotected void
addBucket
(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state) For use internally by DocumentAPI code only.void
addFailedBucket
(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg) static long
bucketToKey
(long id) protected void
boolean
static ProgressToken
fromSerializedString
(String serializedString) long
long
Get internal progress token bucket state map.int
Map<com.yahoo.document.BucketId,
com.yahoo.document.BucketId> Get all failed buckets and their progress.long
long
long
boolean
boolean
protected boolean
isBucketFinished
(com.yahoo.document.BucketId bucket) Checks whether or not a given bucket is certain to be finished.boolean
isEmpty()
boolean
boolean
static long
keyToBucketId
(long key) static long
makeNthBucketKey
(long n, int distributionBits) Directly generate a bucket Id key for then
th bucket in reverse sorted order.protected void
mergePendingBucket
(com.yahoo.document.BucketId bucket) double
Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets.double
progressFraction
(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress) byte[]
Returns a string (base64) encoding of the serial form of this tokenprotected void
setActiveBucketCount
(long activeBucketCount) protected void
protected void
setBucketCursor
(long bucketCursor) protected void
setDistributionBitCount
(int distributionBits) Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion.protected void
setFinishedBucketCount
(long finishedBucketCount) For use by bucket sources and unit tests only!void
setInconsistentState
(boolean inconsistentState) protected void
setPendingBucketCount
(long pendingBucketCount) protected void
setTotalBucketCount
(long totalBucketCount) For use by bucket sources and unit tests only!protected void
splitPendingBucket
(com.yahoo.document.BucketId bucket) toString()
The format of the bucket progress output is as follows:protected void
updateProgress
(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress) Updates internal progress state forbucket
, indicating it's currently atprogress
.
-
Field Details
-
NULL_BUCKET
public static final com.yahoo.document.BucketId NULL_BUCKET -
FINISHED_BUCKET
public static final com.yahoo.document.BucketId FINISHED_BUCKET
-
-
Constructor Details
-
ProgressToken
public ProgressToken()Creates a progress token. -
ProgressToken
public ProgressToken(int distributionBits) -
ProgressToken
-
ProgressToken
public ProgressToken(byte[] serialized)
-
-
Method Details
-
serialize
public byte[] serialize() -
serializeToString
Returns a string (base64) encoding of the serial form of this token -
fromSerializedString
-
addFailedBucket
public void addFailedBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg) -
getFailedBuckets
Get all failed buckets and their progress. Not thread safe.- Returns:
- Unmodifiable map of all failed buckets
-
updateProgress
protected void updateProgress(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress) Updates internal progress state forbucket
, indicating it's currently atprogress
. Assumes that given a completely finished bucket, this function will not be called again to further update its progress after the finished-update.- Parameters:
superbucket
- A valid superbucket ID that exists inbuckets
progress
- The sub-bucket progress that has been reached in the superbucket- See Also:
-
addBucket
protected void addBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state) For use internally by DocumentAPI code only. Using this method by itself will invariably lead to undefined ProgressToken state unless care is taken. Leave it to the VisitorIterator.- Parameters:
superbucket
- Superbucket that will be progress-trackedprogress
- Bucket progress thus farstate
- Initial bucket state. Only pending buckets may be returned
-
makeNthBucketKey
public static long makeNthBucketKey(long n, int distributionBits) Directly generate a bucket Id key for then
th bucket in reverse sorted order.- Parameters:
n
- a number in the range [0, 2**distributionBits
)distributionBits
- Distribution bit count for the generated key- Returns:
- A value where, if you had generated 2**
distributionBits
BucketId
s with incremental numerical IDs and then sorted them on their reverse bit-order keys, the returned key would be equal to the nth element in the resulting sorted sequence.
-
getDistributionBitCount
public int getDistributionBitCount() -
setDistributionBitCount
protected void setDistributionBitCount(int distributionBits) Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion. Note that simply invoking this method on the progress token does not actually change any of its bucket structures/counts! This is the bucket source's responsibility, since only it knows how such a change will affect the progress semantics.- Parameters:
distributionBits
- new distribution bit value
-
getActiveBucketCount
public long getActiveBucketCount() -
getBucketCursor
public long getBucketCursor() -
setBucketCursor
protected void setBucketCursor(long bucketCursor) -
getFinishedBucketCount
public long getFinishedBucketCount() -
setFinishedBucketCount
protected void setFinishedBucketCount(long finishedBucketCount) For use by bucket sources and unit tests only!- Parameters:
finishedBucketCount
- Number of buckets the token has finished
-
getTotalBucketCount
public long getTotalBucketCount() -
setTotalBucketCount
protected void setTotalBucketCount(long totalBucketCount) For use by bucket sources and unit tests only!- Parameters:
totalBucketCount
- Total number of buckets that the progress token spans
-
getPendingBucketCount
public long getPendingBucketCount() -
hasPending
public boolean hasPending() -
hasActive
public boolean hasActive() -
isFinished
public boolean isFinished() -
isEmpty
public boolean isEmpty() -
getFirstErrorMsg
-
containsFailedBuckets
public boolean containsFailedBuckets() -
isInconsistentState
public boolean isInconsistentState() -
setInconsistentState
public void setInconsistentState(boolean inconsistentState) -
getBuckets
Get internal progress token bucket state map. For internal use only!- Returns:
- Map of superbuckets → sub buckets
-
setActiveBucketCount
protected void setActiveBucketCount(long activeBucketCount) -
setPendingBucketCount
protected void setPendingBucketCount(long pendingBucketCount) -
toString
The format of the bucket progress output is as follows:VDS bucket progress file (n% completed)\n distribution bit count\n current bucket cursor\n number of finished buckets\n total number of buckets\n hex-of-superbucket:hex-of-progress\n ... repeat above line for each pending bucket ...
Note that unlike earlier versions of ProgressToken, the bucket IDs are not prefixed with '0x'. -
percentFinished
public double percentFinished()Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets. Runs in O(n+m) time, where n is the number of active buckets and m is the number of pending buckets. Both these values should be fairly small in practice, however. Method is synchronized, as legacy code treats this as an atomic read.- Returns:
- A value in the range [0, 100] estimating the progress.
-
bucketToKey
public static long bucketToKey(long id) -
keyToBucketId
public static long keyToBucketId(long key) -
progressFraction
public double progressFraction(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress) - Parameters:
superbucket
- The superbucket of whichprogress
is a sub-bucketprogress
- The sub-bucket for which a fractional progress should be calculated- Returns:
- a value in [0, 1] specifying how far the (sub-bucket) has reached in its superbucket. This is calculated by looking at the bucket's split factor.
-
isBucketFinished
protected boolean isBucketFinished(com.yahoo.document.BucketId bucket) Checks whether or not a given bucket is certain to be finished. Only looks at the super-bucket part of the given bucket ID, so it's possible that the bucket has in fact finished on a sub-bucket progress level. This does not affect the correctness of the result, however. During a distribution bit change, the token's buckets may be inconsistent. In this scenario, false is always returned since we can't tell for sure if the bucket is still active until the buckets have been made consistent.- Parameters:
bucket
- Bucket to check whether or not is finished.- Returns:
true
ifbucket
's super-bucket is finished,false
otherwise.
-
splitPendingBucket
protected void splitPendingBucket(com.yahoo.document.BucketId bucket) - Parameters:
bucket
- BucketId to be split into two buckets. Bucket's used-bits do not need to match the ProgressToken's current distribution bit count, as it is assumed the client knows what it's doing and will bring the token into a consistent state eventually.
-
mergePendingBucket
protected void mergePendingBucket(com.yahoo.document.BucketId bucket) -
setAllBucketsToState
-
clearAllBuckets
protected void clearAllBuckets()
-