Package org.apache.camel.component.file

Class GenericFileConsumer<T>

  • All Implemented Interfaces:
    AutoCloseable, Runnable, org.apache.camel.BatchConsumer, org.apache.camel.Consumer, org.apache.camel.EndpointAware, org.apache.camel.health.HealthCheckAware, org.apache.camel.PollingConsumerPollingStrategy, org.apache.camel.RouteAware, org.apache.camel.Service, org.apache.camel.ShutdownableService, org.apache.camel.spi.RouteIdAware, org.apache.camel.spi.ShutdownAware, org.apache.camel.spi.ShutdownPrepared, org.apache.camel.StatefulService, org.apache.camel.Suspendable, org.apache.camel.SuspendableService
    Direct Known Subclasses:
    FileConsumer
    public abstract class GenericFileConsumer<T>
extends org.apache.camel.support.ScheduledBatchPollingConsumer
    Base class for file consumers.

    • Method Summary

      All Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      void afterPoll()  
      long beforePoll​(long timeout)  
      boolean canPollMoreFiles​(List<?> fileList)
      Whether or not we can continue polling for more files
      protected abstract org.apache.camel.Exchange createExchange​(GenericFile<T> file)
      Creates the exchange from the polled file
      protected boolean customProcessExchange​(org.apache.camel.Exchange exchange, org.apache.camel.Processor processor)
      Processes the exchange using a custom processor.
      protected void doInit()  
      protected void doStart()  
      protected void doStop()  
      protected String evaluateFileExpression​(org.apache.camel.Exchange exchange)  
      org.apache.camel.Processor getCustomProcessor()  
      protected boolean ignoreCannotRetrieveFile​(String name, org.apache.camel.Exchange exchange, Exception cause)
      Whether to ignore if the file cannot be retrieved.
      boolean isEagerLimitMaxMessagesPerPoll()  
      protected boolean isMatched​(GenericFile<T> file, boolean isDirectory, T[] files)
      Strategy to perform file matching based on endpoint configuration.
      protected abstract boolean isMatched​(GenericFile<T> file, String doneFileName, T[] files)
      Strategy to perform file matching based on endpoint configuration in terms of done file name.
      protected boolean isRetrieveFile()
      Override if required.
      protected boolean isValidFile​(GenericFile<T> file, boolean isDirectory, T[] files)
      Strategy for validating if the given remote file should be included or not
      void onInit()  
      int poll()
      Poll for files
      protected abstract boolean pollDirectory​(String fileName, List<GenericFile<T>> fileList, int depth)
      Polls the given directory for files to process
      protected void postPollCheck​(int polledMessages)
      Override if required.
      protected boolean prePollCheck()
      Override if required.
      int processBatch​(Queue<Object> exchanges)  
      protected boolean processExchange​(org.apache.camel.Exchange exchange)
      Processes the exchange
      protected void removeExcessiveInProgressFiles​(Deque<org.apache.camel.Exchange> exchanges, int limit)
      Drain any in progress files as we are done with this batch
      protected void removeExcessiveInProgressFiles​(List<GenericFile<T>> files)
      Drain any in progress files as we are done with the files
      void setCustomProcessor​(org.apache.camel.Processor processor)
      Use a custom processor to process the exchange.
      void setEagerLimitMaxMessagesPerPoll​(boolean eagerLimitMaxMessagesPerPoll)  
      void setOperations​(GenericFileOperations<T> operations)
      Sets the operations to be used.
      protected abstract void updateFileHeaders​(GenericFile<T> file, org.apache.camel.Message message)
      Updates the information on Message after we have acquired read-lock and can begin process the file.

      • Methods inherited from class org.apache.camel.support.ScheduledBatchPollingConsumer

        deferShutdown, getMaxMessagesPerPoll, getPendingExchangesSize, isBatchAllowed, prepareShutdown, processEmptyMessage, setMaxMessagesPerPoll

      • Methods inherited from class org.apache.camel.support.ScheduledPollConsumer

        addLastErrorDetail, afterConfigureScheduler, doBuild, doShutdown, doSuspend, getBackoffCounter, getBackoffErrorThreshold, getBackoffIdleThreshold, getBackoffMultiplier, getCounter, getDelay, getErrorCounter, getInitialDelay, getLastError, getLastErrorDetails, getPollStrategy, getRepeatCount, getRunLoggingLevel, getScheduledExecutorService, getScheduler, getSchedulerProperties, getSuccessCounter, getTimeUnit, initialHealthCheckState, isFirstPollDone, isGreedy, isPollAllowed, isPolling, isSchedulerStarted, isSendEmptyMessageWhenIdle, isStartScheduler, isUseFixedDelay, run, setBackoffErrorThreshold, setBackoffIdleThreshold, setBackoffMultiplier, setDelay, setGreedy, setInitialDelay, setPollStrategy, setRepeatCount, setRunLoggingLevel, setScheduledExecutorService, setScheduler, setSchedulerProperties, setSendEmptyMessageWhenIdle, setStartScheduler, setTimeUnit, setUseFixedDelay, startScheduler

      • Methods inherited from class org.apache.camel.support.DefaultConsumer

        createExchange, createUoW, defaultConsumerCallback, doneUoW, getAsyncProcessor, getEndpoint, getExceptionHandler, getHealthCheck, getProcessor, getRoute, getRouteId, handleException, handleException, releaseExchange, setExceptionHandler, setHealthCheck, setRoute, setRouteId, toString

      • Methods inherited from class org.apache.camel.support.service.BaseService

        build, doFail, doLifecycleChange, doResume, fail, getStatus, init, isBuild, isInit, isNew, isRunAllowed, isShutdown, isStarted, isStarting, isStartingOrStarted, isStopped, isStopping, isStoppingOrStopped, isSuspended, isSuspending, isSuspendingOrSuspended, resume, shutdown, start, stop, suspend

      • Methods inherited from interface org.apache.camel.Consumer

        createExchange, defaultConsumerCallback, getProcessor, releaseExchange

      • Methods inherited from interface org.apache.camel.EndpointAware

        getEndpoint

      • Methods inherited from interface org.apache.camel.health.HealthCheckAware

        getHealthCheck, setHealthCheck

      • Methods inherited from interface org.apache.camel.Service

        build, close, init, start, stop

      • Methods inherited from interface org.apache.camel.ShutdownableService

        shutdown

      • Methods inherited from interface org.apache.camel.StatefulService

        getStatus, isRunAllowed, isStarted, isStarting, isStopped, isStopping, isSuspending

      • Methods inherited from interface org.apache.camel.SuspendableService

        isSuspended, resume, suspend

    • Field Detail

      • shutdownRunningTask

        protected volatile org.apache.camel.ShutdownRunningTask shutdownRunningTask

      • pendingExchanges

        protected volatile int pendingExchanges

      • customProcessor

        protected org.apache.camel.Processor customProcessor

      • eagerLimitMaxMessagesPerPoll

        protected boolean eagerLimitMaxMessagesPerPoll

      • prepareOnStartup

        protected volatile boolean prepareOnStartup

    • Method Detail

      • getCustomProcessor

        public org.apache.camel.Processor getCustomProcessor()

      • setCustomProcessor

        public void setCustomProcessor​(org.apache.camel.Processor processor)
        Use a custom processor to process the exchange.

        Only set this if you need to do custom processing, instead of the regular processing.

        This is for example used to browse file endpoints by leveraging the file consumer to poll the directory to gather the list of exchanges. But to avoid processing the files regularly we can use a custom processor.

        Parameters:
        processor - a custom processor

      • isEagerLimitMaxMessagesPerPoll

        public boolean isEagerLimitMaxMessagesPerPoll()

      • setEagerLimitMaxMessagesPerPoll

        public void setEagerLimitMaxMessagesPerPoll​(boolean eagerLimitMaxMessagesPerPoll)

      • createExchange

        protected abstract org.apache.camel.Exchange createExchange​(GenericFile<T> file)
        Creates the exchange from the polled file

      • poll

        public int poll()
         throws Exception
        Poll for files
        Specified by:
        poll in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • processBatch

        public int processBatch​(Queue<Object> exchanges)

      • removeExcessiveInProgressFiles

        protected void removeExcessiveInProgressFiles​(Deque<org.apache.camel.Exchange> exchanges,
                                              int limit)
        Drain any in progress files as we are done with this batch
        Parameters:
        exchanges - the exchanges
        limit - the limit

      • removeExcessiveInProgressFiles

        protected void removeExcessiveInProgressFiles​(List<GenericFile<T>> files)
        Drain any in progress files as we are done with the files
        Parameters:
        files - the files

      • canPollMoreFiles

        public boolean canPollMoreFiles​(List<?> fileList)
        Whether or not we can continue polling for more files
        Parameters:
        fileList - the current list of gathered files
        Returns:
        true to continue, false to stop due hitting maxMessagesPerPoll limit

      • prePollCheck

        protected boolean prePollCheck()
                        throws Exception
        Override if required. Perform some checks (and perhaps actions) before we poll.
        Returns:
        true to poll, false to skip this poll.
        Throws:
        Exception

      • postPollCheck

        protected void postPollCheck​(int polledMessages)
        Override if required. Perform some checks (and perhaps actions) after we have polled.
        Parameters:
        polledMessages - number of polled messages

      • pollDirectory

        protected abstract boolean pollDirectory​(String fileName,
                                         List<GenericFile<T>> fileList,
                                         int depth)
        Polls the given directory for files to process
        Parameters:
        fileName - current directory or file
        fileList - current list of files gathered
        depth - the current depth of the directory (will start from 0)
        Returns:
        whether or not to continue polling, false means the maxMessagesPerPoll limit has been hit

      • setOperations

        public void setOperations​(GenericFileOperations<T> operations)
        Sets the operations to be used.

        Can be used to set a fresh operations in case of recovery attempts

        Parameters:
        operations - the operations

      • ignoreCannotRetrieveFile

        protected boolean ignoreCannotRetrieveFile​(String name,
                                           org.apache.camel.Exchange exchange,
                                           Exception cause)
        Whether to ignore if the file cannot be retrieved.

        By default an GenericFileOperationFailedException is thrown if the file cannot be retrieved.

        This method allows to suppress this and just ignore that.

        Parameters:
        name - the file name
        exchange - the exchange
        cause - optional exception occurred during retrieving file
        Returns:
        true to ignore, false is the default.

      • processExchange

        protected boolean processExchange​(org.apache.camel.Exchange exchange)
        Processes the exchange
        Parameters:
        exchange - the exchange
        Returns:
        true if the file was started to be processed, false if the file was not started to be processed, for some reason (not found, or aborted etc)

      • updateFileHeaders

        protected abstract void updateFileHeaders​(GenericFile<T> file,
                                          org.apache.camel.Message message)
        Updates the information on Message after we have acquired read-lock and can begin process the file.
        Parameters:
        file - the file
        message - the Camel message to update its headers

      • isRetrieveFile

        protected boolean isRetrieveFile()
        Override if required. Files are retrieved / returns true by default
        Returns:
        true to retrieve files, false to skip retrieval of files.

      • customProcessExchange

        protected boolean customProcessExchange​(org.apache.camel.Exchange exchange,
                                        org.apache.camel.Processor processor)
        Processes the exchange using a custom processor.
        Parameters:
        exchange - the exchange
        processor - the custom processor

      • isValidFile

        protected boolean isValidFile​(GenericFile<T> file,
                              boolean isDirectory,
                              T[] files)
        Strategy for validating if the given remote file should be included or not
        Parameters:
        file - the file
        isDirectory - whether the file is a directory or a file
        files - files in the directory
        Returns:
        true to include the file, false to skip it

      • isMatched

        protected boolean isMatched​(GenericFile<T> file,
                            boolean isDirectory,
                            T[] files)
        Strategy to perform file matching based on endpoint configuration.

        Will always return false for certain files/folders:

        • Starting with a dot
        • lock files
        And then true for directories.
        Parameters:
        file - the file
        isDirectory - whether the file is a directory or a file
        files - files in the directory
        Returns:
        true if the file is matched, false if not

      • isMatched

        protected abstract boolean isMatched​(GenericFile<T> file,
                                     String doneFileName,
                                     T[] files)
        Strategy to perform file matching based on endpoint configuration in terms of done file name.
        Parameters:
        file - the file
        doneFileName - the done file name (without any paths)
        files - files in the directory
        Returns:
        true if the file is matched, false if not

      • evaluateFileExpression

        protected String evaluateFileExpression​(org.apache.camel.Exchange exchange)

      • doInit

        protected void doInit()
               throws Exception
        Overrides:
        doInit in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • doStart

        protected void doStart()
                throws Exception
        Overrides:
        doStart in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • doStop

        protected void doStop()
               throws Exception
        Overrides:
        doStop in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • onInit

        public void onInit()
            throws Exception
        Specified by:
        onInit in interface org.apache.camel.PollingConsumerPollingStrategy
        Overrides:
        onInit in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • beforePoll

        public long beforePoll​(long timeout)
                throws Exception
        Specified by:
        beforePoll in interface org.apache.camel.PollingConsumerPollingStrategy
        Overrides:
        beforePoll in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception

      • afterPoll

        public void afterPoll()
               throws Exception
        Specified by:
        afterPoll in interface org.apache.camel.PollingConsumerPollingStrategy
        Overrides:
        afterPoll in class org.apache.camel.support.ScheduledPollConsumer
        Throws:
        Exception