Module org.elasticsearch.server
Package org.elasticsearch.ingest

Class IngestService

java.lang.Object
org.elasticsearch.ingest.IngestService
All Implemented Interfaces:
ClusterStateApplier, ReportingService<IngestInfo>
Direct Known Subclasses:
SimulateIngestService
public class IngestService extends Object implements ClusterStateApplier, ReportingService<IngestInfo>
Holder class for several ingest related services.

  • Field Details

  • Constructor Details

  • Method Details

    • createGrokThreadWatchdog

      public static MatcherWatchdog createGrokThreadWatchdog(Environment env, ThreadPool threadPool)

    • resolvePipelinesAndUpdateIndexRequest

      public static void resolvePipelinesAndUpdateIndexRequest(DocWriteRequest<?> originalRequest, IndexRequest indexRequest, Metadata metadata)
      Resolves the potential pipelines (default and final) from the requests or templates associated to the index and then **mutates** the IndexRequest passed object with the pipeline information.

      Also, this method marks the request as `isPipelinesResolved = true`: Due to the request could be rerouted from a coordinating node to an ingest node, we have to be able to avoid double resolving the pipelines and also able to distinguish that either the pipeline comes as part of the request or resolved from this method. All this is made to later be able to reject the request in case the pipeline was set by a required pipeline **and** the request also has a pipeline request too.

      Parameters:
      originalRequest - Original write request received.
      indexRequest - The IndexRequest object to update.
      metadata - Cluster metadata from where the pipeline information could be derived.

    • getClusterService

      public ClusterService getClusterService()

    • getScriptService

      public ScriptService getScriptService()

    • delete

      public void delete(DeletePipelineRequest request, ActionListener<AcknowledgedResponse> listener)
      Deletes the pipeline specified by id in the request.

    • getPipelines

      public static List<PipelineConfiguration> getPipelines(ClusterState clusterState, String... ids)
      Returns:
      pipeline configuration specified by id. If multiple ids or wildcards are specified multiple pipelines may be returned

    • putPipeline

      public void putPipeline(PutPipelineRequest request, ActionListener<AcknowledgedResponse> listener, Consumer<ActionListener<NodesInfoResponse>> nodeInfoListener) throws Exception
      Stores the specified pipeline definition in the request.
      Throws:
      Exception

    • validatePipelineRequest

      public void validatePipelineRequest(PutPipelineRequest request, NodesInfoResponse nodeInfos) throws Exception
      Throws:
      Exception

    • isNoOpPipelineUpdate

      public static boolean isNoOpPipelineUpdate(ClusterState state, PutPipelineRequest request)

    • getPipeline

      public Pipeline getPipeline(String id)
      Returns the pipeline by the specified id

    • getProcessorFactories

      public Map<String,Processor.Factory> getProcessorFactories()

    • info

      public IngestInfo info()
      Specified by:
      info in interface ReportingService<IngestInfo>

    • executeBulkRequest

      public void executeBulkRequest(int numberOfActionRequests, Iterable<DocWriteRequest<?>> actionRequests, IntConsumer onDropped, Predicate<String> shouldStoreFailure, TriConsumer<Integer,String,Exception> onStoreFailure, BiConsumer<Integer,Exception> onFailure, BiConsumer<Thread,Exception> onCompletion, Executor executor)
      Executes all applicable pipelines for a collection of documents.
      Parameters:
      numberOfActionRequests - The total number of requests to process.
      actionRequests - The collection of requests to be processed.
      onDropped - A callback executed when a document is dropped by a pipeline. Accepts the slot in the collection of requests that the document occupies.
      shouldStoreFailure - A predicate executed on each ingest failure to determine if the failure should be stored somewhere.
      onStoreFailure - A callback executed when a document fails ingest but the failure should be persisted elsewhere. Accepts the slot in the collection of requests that the document occupies, the index name that the request was targeting at the time of failure, and the exception that the document encountered.
      onFailure - A callback executed when a document fails ingestion and does not need to be persisted. Accepts the slot in the collection of requests that the document occupies, and the exception that the document encountered.
      onCompletion - A callback executed once all documents have been processed. Accepts the thread that ingestion completed on or an exception in the event that the entire operation has failed.
      executor - Which executor the bulk request should be executed on.

    • stats

      public IngestStats stats()

    • addIngestClusterStateListener

      public void addIngestClusterStateListener(Consumer<ClusterState> listener)
      Adds a listener that gets invoked with the current cluster state before processor factories get invoked.

      This is useful for components that are used by ingest processors, so that they have the opportunity to update before these components get used by the ingest processor factory.

    • applyClusterState

      public void applyClusterState(ClusterChangedEvent event)
      Description copied from interface: ClusterStateApplier
      Called when a new cluster state (ClusterChangedEvent.state() needs to be applied. The cluster state to be applied is already committed when this method is called, so an applier must therefore be prepared to deal with any state it receives without throwing an exception. Throwing an exception from an applier is very bad because it will stop the application of this state before it has reached all the other appliers, and will likely result in another attempt to apply the same (or very similar) cluster state which might continue until this node is removed from the cluster.

      Cluster states are applied one-by-one which means they can be a performance bottleneck. Implementations of this method should therefore be fast, so please consider forking work into the background rather than doing everything inline.

      Specified by:
      applyClusterState in interface ClusterStateApplier

    • getProcessorsInPipeline

      public <P extends Processor> List<P> getProcessorsInPipeline(String pipelineId, Class<P> clazz)
      Gets all the Processors of the given type from within a Pipeline.
      Parameters:
      pipelineId - the pipeline to inspect
      clazz - the Processor class to look for
      Returns:
      True if the pipeline contains an instance of the Processor class passed in

    • getPipelineWithProcessorType

      public <P extends Processor> Collection<String> getPipelineWithProcessorType(Class<P> clazz, Predicate<P> predicate)

    • reloadPipeline

      public void reloadPipeline(String id) throws Exception
      Throws:
      Exception

    • hasPipeline

      public static boolean hasPipeline(IndexRequest indexRequest)
      Checks whether an IndexRequest has at least one pipeline defined.

      This method assumes that the pipelines are beforehand resolved.