Package com.yahoo.documentapi

Class ProgressToken

java.lang.Object
com.yahoo.documentapi.ProgressToken
public class ProgressToken extends Object
Token to use to keep track of progress for visiting. Can be used to resume visiting if visiting has been aborted for any reason.
Author:
Thomas Gundersen, vekterli

  • Field Details

    • NULL_BUCKET

      public static final com.yahoo.document.BucketId NULL_BUCKET

    • FINISHED_BUCKET

      public static final com.yahoo.document.BucketId FINISHED_BUCKET

  • Constructor Details

    • ProgressToken

      public ProgressToken()
      Creates a progress token.

    • ProgressToken

      public ProgressToken(int distributionBits)

    • ProgressToken

      public ProgressToken(String serialized)

    • ProgressToken

      public ProgressToken(byte[] serialized)

  • Method Details

    • serialize

      public byte[] serialize()

    • serializeToString

      public String serializeToString()
      Returns a string (base64) encoding of the serial form of this token

    • fromSerializedString

      public static ProgressToken fromSerializedString(String serializedString)

    • addFailedBucket

      public void addFailedBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg)

    • getFailedBuckets

      public Map<com.yahoo.document.BucketId,com.yahoo.document.BucketId> getFailedBuckets()
      Get all failed buckets and their progress. Not thread safe.
      Returns:
      Unmodifiable map of all failed buckets

    • updateProgress

      protected void updateProgress(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
      Updates internal progress state for bucket, indicating it's currently at progress. Assumes that given a completely finished bucket, this function will not be called again to further update its progress after the finished-update.
      Parameters:
      superbucket - A valid superbucket ID that exists in buckets
      progress - The sub-bucket progress that has been reached in the superbucket
      See Also:

    • addBucket

      protected void addBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state)
      For use internally by DocumentAPI code only. Using this method by itself will invariably lead to undefined ProgressToken state unless care is taken. Leave it to the VisitorIterator.
      Parameters:
      superbucket - Superbucket that will be progress-tracked
      progress - Bucket progress thus far
      state - Initial bucket state. Only pending buckets may be returned

    • makeNthBucketKey

      public static long makeNthBucketKey(long n, int distributionBits)
      Directly generate a bucket Id key for the nth bucket in reverse sorted order.
      Parameters:
      n - a number in the range [0, 2**distributionBits)
      distributionBits - Distribution bit count for the generated key
      Returns:
      A value where, if you had generated 2**distributionBits BucketIds with incremental numerical IDs and then sorted them on their reverse bit-order keys, the returned key would be equal to the nth element in the resulting sorted sequence.

    • getDistributionBitCount

      public int getDistributionBitCount()

    • setDistributionBitCount

      protected void setDistributionBitCount(int distributionBits)
      Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion. Note that simply invoking this method on the progress token does not actually change any of its bucket structures/counts! This is the bucket source's responsibility, since only it knows how such a change will affect the progress semantics.
      Parameters:
      distributionBits - new distribution bit value

    • getActiveBucketCount

      public long getActiveBucketCount()

    • getBucketCursor

      public long getBucketCursor()

    • setBucketCursor

      protected void setBucketCursor(long bucketCursor)

    • getFinishedBucketCount

      public long getFinishedBucketCount()

    • setFinishedBucketCount

      protected void setFinishedBucketCount(long finishedBucketCount)
      For use by bucket sources and unit tests only!
      Parameters:
      finishedBucketCount - Number of buckets the token has finished

    • getTotalBucketCount

      public long getTotalBucketCount()

    • setTotalBucketCount

      protected void setTotalBucketCount(long totalBucketCount)
      For use by bucket sources and unit tests only!
      Parameters:
      totalBucketCount - Total number of buckets that the progress token spans

    • getPendingBucketCount

      public long getPendingBucketCount()

    • hasPending

      public boolean hasPending()

    • hasActive

      public boolean hasActive()

    • isFinished

      public boolean isFinished()

    • isEmpty

      public boolean isEmpty()

    • getFirstErrorMsg

      public String getFirstErrorMsg()

    • containsFailedBuckets

      public boolean containsFailedBuckets()

    • isInconsistentState

      public boolean isInconsistentState()

    • setInconsistentState

      public void setInconsistentState(boolean inconsistentState)

    • getBuckets

      protected TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> getBuckets()
      Get internal progress token bucket state map. For internal use only!
      Returns:
      Map of superbuckets → sub buckets

    • setActiveBucketCount

      protected void setActiveBucketCount(long activeBucketCount)

    • setPendingBucketCount

      protected void setPendingBucketCount(long pendingBucketCount)

    • toString

      public String toString()
      The format of the bucket progress output is as follows: 
         VDS bucket progress file (n% completed)\n
   distribution bit count\n
   current bucket cursor\n
   number of finished buckets\n
   total number of buckets\n
   hex-of-superbucket:hex-of-progress\n
   ... repeat above line for each pending bucket ...
      Note that unlike earlier versions of ProgressToken, the bucket IDs are not prefixed with '0x'.
      Overrides:
      toString in class Object

    • percentFinished

      public double percentFinished()
      Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets. Runs in O(n+m) time, where n is the number of active buckets and m is the number of pending buckets. Both these values should be fairly small in practice, however. Method is synchronized, as legacy code treats this as an atomic read.
      Returns:
      A value in the range [0, 100] estimating the progress.

    • bucketToKey

      public static long bucketToKey(long id)

    • keyToBucketId

      public static long keyToBucketId(long key)

    • progressFraction

      public double progressFraction(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
      Parameters:
      superbucket - The superbucket of which progress is a sub-bucket
      progress - The sub-bucket for which a fractional progress should be calculated
      Returns:
      a value in [0, 1] specifying how far the (sub-bucket) has reached in its superbucket. This is calculated by looking at the bucket's split factor.

    • isBucketFinished

      protected boolean isBucketFinished(com.yahoo.document.BucketId bucket)
      Checks whether or not a given bucket is certain to be finished. Only looks at the super-bucket part of the given bucket ID, so it's possible that the bucket has in fact finished on a sub-bucket progress level. This does not affect the correctness of the result, however. During a distribution bit change, the token's buckets may be inconsistent. In this scenario, false is always returned since we can't tell for sure if the bucket is still active until the buckets have been made consistent.
      Parameters:
      bucket - Bucket to check whether or not is finished.
      Returns:
      true if bucket's super-bucket is finished, false otherwise.

    • splitPendingBucket

      protected void splitPendingBucket(com.yahoo.document.BucketId bucket)
      Parameters:
      bucket - BucketId to be split into two buckets. Bucket's used-bits do not need to match the ProgressToken's current distribution bit count, as it is assumed the client knows what it's doing and will bring the token into a consistent state eventually.

    • mergePendingBucket

      protected void mergePendingBucket(com.yahoo.document.BucketId bucket)

    • setAllBucketsToState

      protected void setAllBucketsToState(ProgressToken.BucketState state)

    • clearAllBuckets

      protected void clearAllBuckets()