Package net.snowflake.ingest.streaming

Interface OffsetTokenVerificationFunction

public interface OffsetTokenVerificationFunction

Interface to provide a custom offset verification logic. If specified, verification failures will be logged as warnings and reported to Snowflake. This interface could be used when there are certain assumption about the offset token behavior. Please reach out to Snowflake if you notice any unexpected behaviors.

Below is an example that verifies that all offset tokens are monotonically increasing numbers:

     
      private static final OffsetTokenVerificationFunction offsetTokenVerificationFunction =
       (prevBatchEndOffset, curBatchStartOffset, curBatchEndOffset, rowCount) -> {
         boolean isMatch = true;

         if (curBatchStartOffset != null) {
           try {
             long curStart = Long.parseLong(curBatchStartOffset);
             long curEnd = Long.parseLong(curBatchEndOffset);

             // We verify that the end_offset - start_offset + 1 = row_count
             if (curEnd - curStart + 1 != rowCount) {
               isMatch = false;
             }

             // We verify that start_offset_of_current_batch = end_offset_of_previous_batch+1
             if (prevBatchEndOffset != null) {
               long prevEnd = Long.parseLong(prevBatchEndOffset);
               if (curStart != prevEnd + 1) {
                 isMatch = false;
               }
             }
           } catch (NumberFormatException ignored) {
             // Do nothing since we can't compare the offset, or report a mismatch if number is expected
           }
         }

         return isMatch;
       };
     
 

Method Summary

Modifier and Type	Method	Description
boolean	verify(String prevBatchEndOffset, String curBatchStartOffset, String curBatchEndOffset, long rowCount)

Method Detail

verify

boolean verify(String prevBatchEndOffset,
               String curBatchStartOffset,
               String curBatchEndOffset,
               long rowCount)

Parameters:

prevBatchEndOffset - end offset token of the previous batch

curBatchStartOffset - start offset token of the current batch

curBatchEndOffset - end offset token of the current batch

rowCount - number of rows in the current batch

Returns:

a boolean indicates whether the verification passed or not, if not, we will log a warning and report it to SF