Module org.elasticsearch.server
Package org.elasticsearch.ingest

Class IngestDocument

java.lang.Object
org.elasticsearch.ingest.IngestDocument
public final class IngestDocument extends Object
Represents a single document being captured before indexing and holds the source and metadata (like id, type and index).

  • Field Details

  • Constructor Details

  • Method Details

    • getFieldValue

      public <T> T getFieldValue(String path, Class<T> clazz)
      Returns the value contained in the document for the provided path
      Parameters:
      path - The path within the document in dot-notation
      clazz - The expected class of the field value
      Returns:
      the value for the provided path if existing
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid, if the field doesn't exist or if the field that is found at the provided path is not of the expected type.

    • getFieldValue

      public <T> T getFieldValue(String path, Class<T> clazz, boolean ignoreMissing)
      Returns the value contained in the document for the provided path
      Parameters:
      path - The path within the document in dot-notation
      clazz - The expected class of the field value
      ignoreMissing - The flag to determine whether to throw an exception when `path` is not found in the document.
      Returns:
      the value for the provided path if existing, null otherwise.
      Throws:
      IllegalArgumentException - only if ignoreMissing is false and the path is null, empty, invalid, if the field doesn't exist or if the field that is found at the provided path is not of the expected type.

    • getFieldValueAsBytes

      public byte[] getFieldValueAsBytes(String path)
      Returns the value contained in the document for the provided path as a byte array. If the path value is a string, a base64 decode operation will happen. If the path value is a byte array, it is just returned
      Parameters:
      path - The path within the document in dot-notation
      Returns:
      the byte array for the provided path if existing
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid, if the field doesn't exist or if the field that is found at the provided path is not of the expected type.

    • getFieldValueAsBytes

      public byte[] getFieldValueAsBytes(String path, boolean ignoreMissing)
      Returns the value contained in the document for the provided path as a byte array. If the path value is a string, a base64 decode operation will happen. If the path value is a byte array, it is just returned
      Parameters:
      path - The path within the document in dot-notation
      ignoreMissing - The flag to determine whether to throw an exception when `path` is not found in the document.
      Returns:
      the byte array for the provided path if existing
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid, if the field doesn't exist or if the field that is found at the provided path is not of the expected type.

    • hasField

      public boolean hasField(String path)
      Checks whether the document contains a value for the provided path
      Parameters:
      path - The path within the document in dot-notation
      Returns:
      true if the document contains a value for the field, false otherwise
      Throws:
      IllegalArgumentException - if the path is null, empty or invalid.

    • hasField

      public boolean hasField(String path, boolean failOutOfRange)
      Checks whether the document contains a value for the provided path
      Parameters:
      path - The path within the document in dot-notation
      failOutOfRange - Whether to throw an IllegalArgumentException if array is accessed outside of its range
      Returns:
      true if the document contains a value for the field, false otherwise
      Throws:
      IllegalArgumentException - if the path is null, empty or invalid.

    • removeField

      public void removeField(String path)
      Removes the field identified by the provided path.
      Parameters:
      path - the path of the field to be removed
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid or if the field doesn't exist.

    • appendFieldValue

      public void appendFieldValue(String path, Object value)
      Appends the provided value to the provided path in the document. Any non existing path element will be created. If the path identifies a list, the value will be appended to the existing list. If the path identifies a scalar, the scalar will be converted to a list and the provided value will be added to the newly created list. Supports multiple values too provided in forms of list, in that case all the values will be appended to the existing (or newly created) list.
      Parameters:
      path - The path within the document in dot-notation
      value - The value or values to append to the existing ones
      Throws:
      IllegalArgumentException - if the path is null, empty or invalid.

    • appendFieldValue

      public void appendFieldValue(String path, Object value, boolean allowDuplicates)
      Appends the provided value to the provided path in the document. Any non existing path element will be created. If the path identifies a list, the value will be appended to the existing list. If the path identifies a scalar, the scalar will be converted to a list and the provided value will be added to the newly created list. Supports multiple values too provided in forms of list, in that case all the values will be appended to the existing (or newly created) list.
      Parameters:
      path - The path within the document in dot-notation
      value - The value or values to append to the existing ones
      allowDuplicates - When false, any values that already exist in the field will not be added
      Throws:
      IllegalArgumentException - if the path is null, empty or invalid.

    • appendFieldValue

      public void appendFieldValue(String path, ValueSource valueSource, boolean allowDuplicates)
      Appends the provided value to the provided path in the document. Any non existing path element will be created. If the path identifies a list, the value will be appended to the existing list. If the path identifies a scalar, the scalar will be converted to a list and the provided value will be added to the newly created list. Supports multiple values too provided in forms of list, in that case all the values will be appended to the existing (or newly created) list.
      Parameters:
      path - The path within the document in dot-notation
      valueSource - The value source that will produce the value or values to append to the existing ones
      allowDuplicates - When false, any values that already exist in the field will not be added
      Throws:
      IllegalArgumentException - if the path is null, empty or invalid.

    • setFieldValue

      public void setFieldValue(String path, Object value)
      Sets the provided value to the provided path in the document. Any non existing path element will be created. If the last item in the path is a list, the value will replace the existing list as a whole. Use appendFieldValue(String, Object) to append values to lists instead.
      Parameters:
      path - The path within the document in dot-notation
      value - The value to put in for the path key
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid or if the value cannot be set to the item identified by the provided path.

    • setFieldValue

      public void setFieldValue(String path, ValueSource valueSource)
      Sets the provided value to the provided path in the document. Any non existing path element will be created. If the last element is a list, the value will replace the existing list.
      Parameters:
      path - The path within the document in dot-notation
      valueSource - The value source that will produce the value to put in for the path key
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid or if the value cannot be set to the item identified by the provided path.

    • setFieldValue

      public void setFieldValue(String path, ValueSource valueSource, boolean ignoreEmptyValue)
      Sets the provided value to the provided path in the document. Any non existing path element will be created. If the last element is a list, the value will replace the existing list.
      Parameters:
      path - The path within the document in dot-notation
      valueSource - The value source that will produce the value to put in for the path key
      ignoreEmptyValue - The flag to determine whether to exit quietly when the value produced by TemplatedValue is null or empty
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid or if the value cannot be set to the item identified by the provided path.

    • setFieldValue

      public void setFieldValue(String path, Object value, boolean ignoreEmptyValue)
      Sets the provided value to the provided path in the document. Any non existing path element will be created. If the last element is a list, the value will replace the existing list.
      Parameters:
      path - The path within the document in dot-notation
      value - The value to put in for the path key
      ignoreEmptyValue - The flag to determine whether to exit quietly when the value produced by TemplatedValue is null or empty
      Throws:
      IllegalArgumentException - if the path is null, empty, invalid or if the value cannot be set to the item identified by the provided path.

    • renderTemplate

      public String renderTemplate(TemplateScript.Factory template)
      Renders a template into a string. This allows field access via both literal fields like "foo.bar.baz" and dynamic fields like "{{other_field}}" (that is, look up the value of the 'other_field' in the document and then use the resulting string as the field to operate on).

      See ConfigurationUtils.compileTemplate(String, String, String, String, ScriptService) and associated methods, which create these TemplateScript.Factory instances.

      Note: for clarity and efficiency reasons, it is advisable to invoke this method outside IngestDocument itself -- fields should be rendered by a caller (once), and then passed to an ingest document repeatedly. There are enough methods on IngestDocument that operate on String paths already, we don't want to mirror all of them with twin methods that accept a template.

      Parameters:
      template - the template or literal string to evaluate
      Returns:
      a literal string field path

    • getSourceAndMetadata

      public Map<String,Object> getSourceAndMetadata()
      Get source and metadata map

    • getCtxMap

      public CtxMap<?> getCtxMap()
      Get the CtxMap

    • getMetadata

      public Metadata getMetadata()
      Get the strongly typed metadata

    • getSource

      public Map<String,Object> getSource()
      Get all source values in a Map

    • getIngestMetadata

      public Map<String,Object> getIngestMetadata()
      Returns the available ingest metadata fields, by default only timestamp, but it is possible to set additional ones. Use only for reading values, modify them instead using setFieldValue(String, Object) and removeField(String)

    • deepCopyMap

      public static <K, V> Map<K,V> deepCopyMap(Map<K,V> source)

    • deepCopy

      public static Object deepCopy(Object value)

    • getAllFields

      public static Set<String> getAllFields(Map<String,Object> input)

    • executePipeline

      public void executePipeline(Pipeline pipeline, BiConsumer<IngestDocument,Exception> handler)
      Executes the given pipeline with for this document unless the pipeline has already been executed for this document.
      Parameters:
      pipeline - the pipeline to execute
      handler - handles the result or failure

    • updateIndexHistory

      public boolean updateIndexHistory(String index)
      Adds an index to the index history for this document, returning true if the index was added to the index history (i.e. if it wasn't already in the index history).
      Parameters:
      index - the index to potentially add to the index history
      Returns:
      true if the index history did not already contain the index in question

    • getIndexHistory

      public Set<String> getIndexHistory()
      Returns:
      an unmodifiable view of the document's index history

    • doNoSelfReferencesCheck

      public boolean doNoSelfReferencesCheck()
      Returns:
      Whether a self referencing check should be performed

    • doNoSelfReferencesCheck

      public void doNoSelfReferencesCheck(boolean doNoSelfReferencesCheck)
      Whether the ingest framework should perform a self referencing check after this ingest document has been processed by all pipelines. Doing this check adds an extra tax to ingest and should only be performed when really needed. Only if a processor is executed that could add self referencing maps or lists then this check must be performed. Most processors will not be able to do this, hence the default is false.
      Parameters:
      doNoSelfReferencesCheck - Whether a self referencing check should be performed

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • reroute

      public void reroute(String destIndex)