Package org.dataloader

Class DataLoader<K,​V>

  • Type Parameters:
    K - type parameter indicating the type of the data load keys
    V - type parameter indicating the type of the data that is returned
    @PublicApi
public class DataLoader<K,​V>
extends java.lang.Object
    Data loader is a utility class that allows batch loading of data that is identified by a set of unique keys. For each key that is loaded a separate CompletableFuture is returned, that completes as the batch function completes.

    With batching enabled the execution will start after calling dispatch(), causing the queue of loaded keys to be sent to the batch function, clears the queue, and returns a promise to the values.

    As BatchLoader batch functions are executed the resulting futures are cached using a cache implementation of choice, so they will only execute once. Individual cache keys can be cleared, so they will be re-fetched when referred to again.

    It is also possible to clear the cache entirely, and prime it with values before they are used.

    Both caching and batching can be disabled. Configuration of the data loader is done by providing a DataLoaderOptions instance on creation.

    A call to the batch loader might result in individual exception failures for item with the returned list. if you want to capture these specific item failures then use Try as a return value and create the data loader with newDataLoaderWithTry(BatchLoader) form. The Try values will be interpreted as either success values or cause the load(Object) promise to complete exceptionally.

    • Constructor Detail

      • DataLoader

        @Deprecated
public DataLoader​(BatchLoader<K,​V> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates a new data loader with the provided batch load function, and default options.
        Parameters:
        batchLoadFunction - the batch load function to use

      • DataLoader

        @Deprecated
public DataLoader​(BatchLoader<K,​V> batchLoadFunction,
                  DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates a new data loader with the provided batch load function and options.
        Parameters:
        batchLoadFunction - the batch load function to use
        options - the batch load options

    • Method Detail

      • newDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoader​(BatchLoader<K,​V> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size).
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        Returns:
        a new DataLoader

      • newDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoader​(BatchLoader<K,​V> batchLoadFunction,
                                                              DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function with the provided options
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        options - the options to use
        Returns:
        a new DataLoader

      • newDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoaderWithTry​(BatchLoader<K,​Try<V>> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size) where the batch loader function returns a list of Try objects.

        If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then you can use this form to create the data loader.

        Using Try objects allows you to capture a value returned or an exception that might have occurred trying to get a value. .

        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        Returns:
        a new DataLoader

      • newDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoaderWithTry​(BatchLoader<K,​Try<V>> batchLoadFunction,
                                                                     DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and with the provided options where the batch loader function returns a list of Try objects.
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        options - the options to use
        Returns:
        a new DataLoader
        See Also:
        DataLoaderFactory.newDataLoaderWithTry(BatchLoader)

      • newDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoader​(BatchLoaderWithContext<K,​V> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size).
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        Returns:
        a new DataLoader

      • newDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoader​(BatchLoaderWithContext<K,​V> batchLoadFunction,
                                                              DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function with the provided options
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        options - the options to use
        Returns:
        a new DataLoader

      • newDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoaderWithTry​(BatchLoaderWithContext<K,​Try<V>> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size) where the batch loader function returns a list of Try objects.

        If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then you can use this form to create the data loader.

        Using Try objects allows you to capture a value returned or an exception that might have occurred trying to get a value. .

        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        Returns:
        a new DataLoader

      • newDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newDataLoaderWithTry​(BatchLoaderWithContext<K,​Try<V>> batchLoadFunction,
                                                                     DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and with the provided options where the batch loader function returns a list of Try objects.
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        options - the options to use
        Returns:
        a new DataLoader
        See Also:
        DataLoaderFactory.newDataLoaderWithTry(BatchLoader)

      • newMappedDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoader​(MappedBatchLoader<K,​V> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size).
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        Returns:
        a new DataLoader

      • newMappedDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoader​(MappedBatchLoader<K,​V> batchLoadFunction,
                                                                    DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function with the provided options
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        options - the options to use
        Returns:
        a new DataLoader

      • newMappedDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoaderWithTry​(MappedBatchLoader<K,​Try<V>> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size) where the batch loader function returns a list of Try objects.

        If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then you can use this form to create the data loader.

        Using Try objects allows you to capture a value returned or an exception that might have occurred trying to get a value. .

        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        Returns:
        a new DataLoader

      • newMappedDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoaderWithTry​(MappedBatchLoader<K,​Try<V>> batchLoadFunction,
                                                                           DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and with the provided options where the batch loader function returns a list of Try objects.
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        options - the options to use
        Returns:
        a new DataLoader
        See Also:
        DataLoaderFactory.newDataLoaderWithTry(BatchLoader)

      • newMappedDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoader​(MappedBatchLoaderWithContext<K,​V> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified mapped batch loader function and default options (batching, caching and unlimited batch size).
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        Returns:
        a new DataLoader

      • newMappedDataLoader

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoader​(MappedBatchLoaderWithContext<K,​V> batchLoadFunction,
                                                                    DataLoaderOptions options)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function with the provided options
        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use
        options - the options to use
        Returns:
        a new DataLoader

      • newMappedDataLoaderWithTry

        @Deprecated
public static <K,​V> DataLoader<K,​V> newMappedDataLoaderWithTry​(MappedBatchLoaderWithContext<K,​Try<V>> batchLoadFunction)
        Deprecated.
        use DataLoaderFactory instead
        Creates new DataLoader with the specified batch loader function and default options (batching, caching and unlimited batch size) where the batch loader function returns a list of Try objects.

        If it's important you to know the exact status of each item in a batch call and whether it threw exceptions then you can use this form to create the data loader.

        Using Try objects allows you to capture a value returned or an exception that might have occurred trying to get a value. .

        Type Parameters:
        K - the key type
        V - the value type
        Parameters:
        batchLoadFunction - the batch load function to use that uses Try objects
        Returns:
        a new DataLoader

      • getLastDispatchTime

        public java.time.Instant getLastDispatchTime()
        This returns the last instant the data loader was dispatched. When the data loader is created this value is set to now.
        Returns:
        the instant since the last dispatch

      • getTimeSinceDispatch

        public java.time.Duration getTimeSinceDispatch()
        This returns the Duration since the data loader was dispatched. When the data loader is created this is zero.
        Returns:
        the time duration since the last dispatch

      • load

        public java.util.concurrent.CompletableFuture<V> load​(K key)
        Requests to load the data with the specified key asynchronously, and returns a future of the resulting value.

        If batching is enabled (the default), you'll have to call dispatch() at a later stage to start batch execution. If you forget this call the future will never be completed (unless already completed, and returned from cache).

        Parameters:
        key - the key to load
        Returns:
        the future of the value

      • getIfPresent

        public java.util.Optional<java.util.concurrent.CompletableFuture<V>> getIfPresent​(K key)
        This will return an optional promise to a value previously loaded via a load(Object) call or empty if not call has been made for that key.

        If you do get a present CompletableFuture it does not mean it has been dispatched and completed yet. It just means it's at least pending and in cache.

        If caching is disabled there will never be a present Optional returned.

        NOTE : This will NOT cause a data load to happen. You must call load(Object) for that to happen.

        Parameters:
        key - the key to check
        Returns:
        an Optional to the future of the value

      • getIfCompleted

        public java.util.Optional<java.util.concurrent.CompletableFuture<V>> getIfCompleted​(K key)
        This will return an optional promise to a value previously loaded via a load(Object) call that has in fact been completed or empty if no call has been made for that key or the promise has not completed yet.

        If you do get a present CompletableFuture it means it has been dispatched and completed. Completed is defined as CompletableFuture.isDone() returning true.

        If caching is disabled there will never be a present Optional returned.

        NOTE : This will NOT cause a data load to happen. You must call load(Object) for that to happen.

        Parameters:
        key - the key to check
        Returns:
        an Optional to the future of the value

      • load

        public java.util.concurrent.CompletableFuture<V> load​(K key,
                                                      java.lang.Object keyContext)
        Requests to load the data with the specified key asynchronously, and returns a future of the resulting value.

        If batching is enabled (the default), you'll have to call dispatch() at a later stage to start batch execution. If you forget this call the future will never be completed (unless already completed, and returned from cache).

        The key context object may be useful in the batch loader interfaces such as BatchLoaderWithContext or MappedBatchLoaderWithContext to help retrieve data.

        Parameters:
        key - the key to load
        keyContext - a context object that is specific to this key
        Returns:
        the future of the value

      • loadMany

        public java.util.concurrent.CompletableFuture<java.util.List<V>> loadMany​(java.util.List<K> keys)
        Requests to load the list of data provided by the specified keys asynchronously, and returns a composite future of the resulting values.

        If batching is enabled (the default), you'll have to call dispatch() at a later stage to start batch execution. If you forget this call the future will never be completed (unless already completed, and returned from cache).

        Parameters:
        keys - the list of keys to load
        Returns:
        the composite future of the list of values

      • loadMany

        public java.util.concurrent.CompletableFuture<java.util.List<V>> loadMany​(java.util.List<K> keys,
                                                                          java.util.List<java.lang.Object> keyContexts)
        Requests to load the list of data provided by the specified keys asynchronously, and returns a composite future of the resulting values.

        If batching is enabled (the default), you'll have to call dispatch() at a later stage to start batch execution. If you forget this call the future will never be completed (unless already completed, and returned from cache).

        The key context object may be useful in the batch loader interfaces such as BatchLoaderWithContext or MappedBatchLoaderWithContext to help retrieve data.

        Parameters:
        keys - the list of keys to load
        keyContexts - the list of key calling context objects
        Returns:
        the composite future of the list of values

      • loadMany

        public java.util.concurrent.CompletableFuture<java.util.Map<K,​V>> loadMany​(java.util.Map<K,​?> keysAndContexts)
        Requests to load the map of data provided by the specified keys asynchronously, and returns a composite future of the resulting values.

        If batching is enabled (the default), you'll have to call dispatch() at a later stage to start batch execution. If you forget this call the future will never be completed (unless already completed, and returned from cache).

        The key context object may be useful in the batch loader interfaces such as BatchLoaderWithContext or MappedBatchLoaderWithContext to help retrieve data.

        Parameters:
        keysAndContexts - the map of keys to their respective contexts
        Returns:
        the composite future of the map of keys and values

      • dispatch

        public java.util.concurrent.CompletableFuture<java.util.List<V>> dispatch()
        Dispatches the queued load requests to the batch execution function and returns a promise of the result.

        If batching is disabled, or there are no queued requests, then a succeeded promise is returned.

        Returns:
        the promise of the queued load requests

      • dispatchWithCounts

        public DispatchResult<V> dispatchWithCounts()
        Dispatches the queued load requests to the batch execution function and returns both the promise of the result and the number of entries that were dispatched.

        If batching is disabled, or there are no queued requests, then a succeeded promise with no entries dispatched is returned.

        Returns:
        the promise of the queued load requests and the number of keys dispatched.

      • dispatchAndJoin

        public java.util.List<V> dispatchAndJoin()
        Normally dispatch() is an asynchronous operation but this version will 'join' on the results if dispatch and wait for them to complete. If the CompletableFuture callbacks make more calls to this data loader then the dispatchDepth() will be > 0 and this method will loop around and wait for any other extra batch loads to occur.
        Returns:
        the list of all results when the dispatchDepth() reached 0

      • dispatchDepth

        public int dispatchDepth()
        Returns:
        the depth of the batched key loads that need to be dispatched

      • clear

        public DataLoader<K,​V> clear​(K key)
        Clears the future with the specified key from the cache, if caching is enabled, so it will be re-fetched on the next load request.
        Parameters:
        key - the key to remove
        Returns:
        the data loader for fluent coding

      • clear

        public DataLoader<K,​V> clear​(K key,
                                   java.util.function.BiConsumer<java.lang.Void,​java.lang.Throwable> handler)
        Clears the future with the specified key from the cache remote value store, if caching is enabled and a remote store is set, so it will be re-fetched and stored on the next load request.
        Parameters:
        key - the key to remove
        handler - a handler that will be called after the async remote clear completes
        Returns:
        the data loader for fluent coding

      • clearAll

        public DataLoader<K,​V> clearAll()
        Clears the entire cache map of the loader.
        Returns:
        the data loader for fluent coding

      • clearAll

        public DataLoader<K,​V> clearAll​(java.util.function.BiConsumer<java.lang.Void,​java.lang.Throwable> handler)
        Clears the entire cache map of the loader, and of the cached value store.
        Parameters:
        handler - a handler that will be called after the async remote clear all completes
        Returns:
        the data loader for fluent coding

      • prime

        public DataLoader<K,​V> prime​(K key,
                                   V value)
        Primes the cache with the given key and value. Note this will only prime the future cache and not the value store. Use ValueCache.set(Object, Object) if you want to prime it with values before use
        Parameters:
        key - the key
        value - the value
        Returns:
        the data loader for fluent coding

      • prime

        public DataLoader<K,​V> prime​(K key,
                                   java.lang.Exception error)
        Primes the cache with the given key and error.
        Parameters:
        key - the key
        error - the exception to prime instead of a value
        Returns:
        the data loader for fluent coding

      • prime

        public DataLoader<K,​V> prime​(K key,
                                   java.util.concurrent.CompletableFuture<V> value)
        Primes the cache with the given key and value. Note this will only prime the future cache and not the value store. Use ValueCache.set(Object, Object) if you want to prime it with values before use
        Parameters:
        key - the key
        value - the value
        Returns:
        the data loader for fluent coding

      • getCacheKey

        public java.lang.Object getCacheKey​(K key)
        Gets the object that is used in the internal cache map as key, by applying the cache key function to the provided key.

        If no cache key function is present in DataLoaderOptions, then the returned value equals the input key.

        Parameters:
        key - the input key
        Returns:
        the cache key after the input is transformed with the cache key function

      • getCacheMap

        public CacheMap<java.lang.Object,​V> getCacheMap()
        Gets the cacheMap associated with this data loader passed in via DataLoaderOptions.cacheMap()
        Returns:
        the cacheMap of this data loader