Class BaseTokenFilter<T extends GenericToken>

    • Constructor Detail

      • BaseTokenFilter

        public BaseTokenFilter​(TokenManager tokenManager)
        Creates a new BaseTokenFilter
        Parameters:
        tokenManager - The token manager from which to retrieve tokens to be filtered
    • Method Detail

      • getNextToken

        public final T getNextToken()
        Description copied from interface: TokenFilter
        Retrieves the next token to pass the filter
        Specified by:
        getNextToken in interface TokenFilter
        Returns:
        The next token to pass the filter, or null if the end of the stream was reached
      • analyzeToken

        protected void analyzeToken​(T currentToken)
        Extension point for subclasses to analyze all tokens (before filtering) and update internal status to decide on custom discard rules.
        Parameters:
        currentToken - The token to be analyzed
        See Also:
        isLanguageSpecificDiscarding()
      • analyzeTokens

        protected void analyzeTokens​(T currentToken,
                                     Iterable<T> remainingTokens)
        Extension point for subclasses to analyze all tokens (before filtering) and update internal status to decide on custom discard rules.
        Parameters:
        currentToken - The token to be analyzed
        remainingTokens - All upcoming tokens
        See Also:
        isLanguageSpecificDiscarding()
      • isLanguageSpecificDiscarding

        protected boolean isLanguageSpecificDiscarding()
        Extension point for subclasses to indicate tokens are to be filtered.
        Returns:
        True if tokens should be filtered, false otherwise
      • shouldStopProcessing

        protected abstract boolean shouldStopProcessing​(T currentToken)
        Extension point for subclasses to indicate when to stop filtering tokens.
        Parameters:
        currentToken - The token to be analyzed
        Returns:
        True if the token filter has finished consuming all tokens, false otherwise