Class ProgressToken


  • public class ProgressToken
    extends java.lang.Object
    Token to use to keep track of progress for visiting. Can be used to resume visiting if visiting has been aborted for any reason.
    Author:
    Thomas Gundersen, vekterli
    • Field Detail

      • NULL_BUCKET

        public static final com.yahoo.document.BucketId NULL_BUCKET
      • FINISHED_BUCKET

        public static final com.yahoo.document.BucketId FINISHED_BUCKET
    • Constructor Detail

      • ProgressToken

        public ProgressToken()
        Creates a progress token.
      • ProgressToken

        public ProgressToken​(int distributionBits)
      • ProgressToken

        public ProgressToken​(java.lang.String serialized)
      • ProgressToken

        public ProgressToken​(byte[] serialized)
    • Method Detail

      • serialize

        public byte[] serialize()
      • serializeToString

        public java.lang.String serializeToString()
        Returns a string (base64) encoding of the serial form of this token
      • fromSerializedString

        public static ProgressToken fromSerializedString​(java.lang.String serializedString)
      • addFailedBucket

        public void addFailedBucket​(com.yahoo.document.BucketId superbucket,
                                    com.yahoo.document.BucketId progress,
                                    java.lang.String errorMsg)
      • getFailedBuckets

        public java.util.Map<com.yahoo.document.BucketId,​com.yahoo.document.BucketId> getFailedBuckets()
        Get all failed buckets and their progress. Not thread safe.
        Returns:
        Unmodifiable map of all failed buckets
      • updateProgress

        protected void updateProgress​(com.yahoo.document.BucketId superbucket,
                                      com.yahoo.document.BucketId progress)
        Updates internal progress state for bucket, indicating it's currently at progress. Assumes that given a completely finished bucket, this function will not be called again to further update its progress after the finished-update.
        Parameters:
        superbucket - A valid superbucket ID that exists in buckets
        progress - The sub-bucket progress that has been reached in the superbucket
        See Also:
        VisitorIterator.update(com.yahoo.document.BucketId, com.yahoo.document.BucketId)
      • addBucket

        protected void addBucket​(com.yahoo.document.BucketId superbucket,
                                 com.yahoo.document.BucketId progress,
                                 ProgressToken.BucketState state)
        For use internally by DocumentAPI code only. Using this method by itself will invariably lead to undefined ProgressToken state unless care is taken. Leave it to the VisitorIterator.
        Parameters:
        superbucket - Superbucket that will be progress-tracked
        progress - Bucket progress thus far
        state - Initial bucket state. Only pending buckets may be returned
      • makeNthBucketKey

        public static long makeNthBucketKey​(long n,
                                            int distributionBits)
        Directly generate a bucket Id key for the nth bucket in reverse sorted order.
        Parameters:
        n - a number in the range [0, 2**distributionBits)
        distributionBits - Distribution bit count for the generated key
        Returns:
        A value where, if you had generated 2**distributionBits BucketIds with incremental numerical IDs and then sorted them on their reverse bit-order keys, the returned key would be equal to the nth element in the resulting sorted sequence.
      • getDistributionBitCount

        public int getDistributionBitCount()
      • setDistributionBitCount

        protected void setDistributionBitCount​(int distributionBits)
        Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion. Note that simply invoking this method on the progress token does not actually change any of its bucket structures/counts! This is the bucket source's responsibility, since only it knows how such a change will affect the progress semantics.
        Parameters:
        distributionBits - new distribution bit value
      • getActiveBucketCount

        public long getActiveBucketCount()
      • getBucketCursor

        public long getBucketCursor()
      • setBucketCursor

        protected void setBucketCursor​(long bucketCursor)
      • getFinishedBucketCount

        public long getFinishedBucketCount()
      • setFinishedBucketCount

        protected void setFinishedBucketCount​(long finishedBucketCount)
        For use by bucket sources and unit tests only!
        Parameters:
        finishedBucketCount - Number of buckets the token has finished
      • getTotalBucketCount

        public long getTotalBucketCount()
      • setTotalBucketCount

        protected void setTotalBucketCount​(long totalBucketCount)
        For use by bucket sources and unit tests only!
        Parameters:
        totalBucketCount - Total number of buckets that the progress token spans
      • getPendingBucketCount

        public long getPendingBucketCount()
      • hasPending

        public boolean hasPending()
      • hasActive

        public boolean hasActive()
      • isFinished

        public boolean isFinished()
      • isEmpty

        public boolean isEmpty()
      • getFirstErrorMsg

        public java.lang.String getFirstErrorMsg()
      • containsFailedBuckets

        public boolean containsFailedBuckets()
      • isInconsistentState

        public boolean isInconsistentState()
      • setInconsistentState

        public void setInconsistentState​(boolean inconsistentState)
      • setActiveBucketCount

        protected void setActiveBucketCount​(long activeBucketCount)
      • setPendingBucketCount

        protected void setPendingBucketCount​(long pendingBucketCount)
      • toString

        public java.lang.String toString()
        The format of the bucket progress output is as follows:
           VDS bucket progress file (n% completed)\n
           distribution bit count\n
           current bucket cursor\n
           number of finished buckets\n
           total number of buckets\n
           hex-of-superbucket:hex-of-progress\n
           ... repeat above line for each pending bucket ...
         
        Note that unlike earlier versions of ProgressToken, the bucket IDs are not prefixed with '0x'.
        Overrides:
        toString in class java.lang.Object
      • percentFinished

        public double percentFinished()
        Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets. Runs in O(n+m) time, where n is the number of active buckets and m is the number of pending buckets. Both these values should be fairly small in practice, however. Method is synchronized, as legacy code treats this as an atomic read.
        Returns:
        A value in the range [0, 100] estimating the progress.
      • bucketToKey

        public static long bucketToKey​(long id)
      • keyToBucketId

        public static long keyToBucketId​(long key)
      • progressFraction

        public double progressFraction​(com.yahoo.document.BucketId superbucket,
                                       com.yahoo.document.BucketId progress)
        Parameters:
        superbucket - The superbucket of which progress is a sub-bucket
        progress - The sub-bucket for which a fractional progress should be calculated
        Returns:
        a value in [0, 1] specifying how far the (sub-bucket) has reached in its superbucket. This is calculated by looking at the bucket's split factor.
      • isBucketFinished

        protected boolean isBucketFinished​(com.yahoo.document.BucketId bucket)
        Checks whether or not a given bucket is certain to be finished. Only looks at the super-bucket part of the given bucket ID, so it's possible that the bucket has in fact finished on a sub-bucket progress level. This does not affect the correctness of the result, however. During a distribution bit change, the token's buckets may be inconsistent. In this scenario, false is always returned since we can't tell for sure if the bucket is still active until the buckets have been made consistent.
        Parameters:
        bucket - Bucket to check whether or not is finished.
        Returns:
        true if bucket's super-bucket is finished, false otherwise.
      • splitPendingBucket

        protected void splitPendingBucket​(com.yahoo.document.BucketId bucket)
        Parameters:
        bucket - BucketId to be split into two buckets. Bucket's used-bits do not need to match the ProgressToken's current distribution bit count, as it is assumed the client knows what it's doing and will bring the token into a consistent state eventually.
      • mergePendingBucket

        protected void mergePendingBucket​(com.yahoo.document.BucketId bucket)
      • clearAllBuckets

        protected void clearAllBuckets()