MongoInputConfig
Related Doc:
package config
The type of the MongoConfig
The type of the MongoConfig
Create a configuration from the values in the Map, using the optional default configuration for any default values.
Create a configuration from the values in the Map, using the optional default configuration for any default values.
Note: Values in the map do not need to be prefixed with the configPrefix.
a map of properties and their string values
the optional default configuration, used for determining the default values for the properties
the configuration
Create a configuration easily from the Java API using the values in the Map, using the optional default configuration for any
default values.
Create a configuration easily from the Java API using the values in the Map, using the optional default configuration for any
default values.
Note: Values in the map do not need to be prefixed with the configPrefix.
a map of properties and their string values
the optional default configuration, used for determining the default values for the properties
the configuration
Create a configuration easily from the Java API using the values in the Map
Create a configuration easily from the Java API using the values in the Map
Note: Values in the map do not need to be prefixed with the configPrefix.
a map of properties and their string values
the configuration
Create a configuration from the sparkConf
Create a configuration from the sparkConf
Uses the prefixed properties that are set in the Spark configuration to create the config.
the spark configuration
overloaded parameters
the configuration
Create a configuration easily from the Java API using the sparkConf
Create a configuration easily from the Java API using the sparkConf
Uses the prefixed properties that are set in the Spark configuration to create the config.
the spark configuration
the configuration
Create a configuration easily from the Java API using the JavaSparkContext
Create a configuration easily from the Java API using the JavaSparkContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the SparkSession
the configuration
Create a configuration easily from the Java API using the JavaSparkContext
Create a configuration easily from the Java API using the JavaSparkContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the java spark context
the configuration
Create a configuration easily from the Java API using the JavaSparkContext
Create a configuration easily from the Java API using the JavaSparkContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the SQL context
the configuration
(Since version 2.0.0) As of Spark 2.0 SQLContext was replaced by SparkSession. Use the SparkSession method instead
The allow disk use property
The allow disk use property
Enables writing to temporary files
2.3.1
Create a configuration from the values in the Map
Create a configuration from the values in the Map
Note: Values in the map do not need to be prefixed with the configPrefix.
a map of properties and their string values
the configuration
Create a configuration from the sparkConf
Create a configuration from the sparkConf
Uses the prefixed properties that are set in the Spark configuration to create the config.
the spark configuration
overloaded parameters
the configuration
Create a configuration from the sparkConf
Create a configuration from the sparkConf
Uses the prefixed properties that are set in the Spark configuration to create the config.
the spark configuration
the configuration
Create a configuration from the sqlContext
Create a configuration from the sqlContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the SparkSession
the configuration
Create a configuration from the sparkContext
Create a configuration from the sparkContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the spark context
the configuration
The batch size property
The batch size property
The size of batches used by the underlying cursor. Smaller batches will result in more round trips to MongoDB.
Default: The servers default
2.4.1
The collation property
The collation property
The json representation of a Collation. Created via Collation.asDocument.toJson.
2.3
The collection name property
The configuration prefix string for the current configuration scope
The configuration prefix string for the current configuration scope
The database name property
Gets an options map from the SparkConf
Gets an options map from the SparkConf
the SparkConf
the options
The hint property
The hint property
The json representation of a hint document
2.3
The infer schema MapType enabled property
The infer schema MapType enabled property
A boolean flag to enable or disable MapType infer. If this flag is enabled, large compatible struct types will be inferred to a MapType instead.
Default: true
2.3
The infer schema MapType minimum keys property
The infer schema MapType minimum keys property
The minimum keys property controls how large a struct must be before a MapType should be inferred.
Default: 250
2.3
The localThreshold property
The localThreshold property
The local threshold in milliseconds is used when choosing among multiple MongoDB servers to send a request. Only servers whose ping time is less than or equal to the server with the fastest ping time *plus* the local threshold will be chosen.
For example when choosing which MongoS to send a request through a localThreshold of 0 would pick the MongoS with the fastest ping time.
Default: 15 ms
The mongo URI property
The mongo URI property
Represents a connection string.
Any values set in the connection string will override any default values for the configuration.
The partitioner options property
The partitioner options property
Represents a map of options for customising the configuration of a partitioner.
Default: Map.empty[String, String]
The partition property
The partition property
Represents the name of the partitioner to use when partitioning the data in the collection.
Default: MongoDefaultPartitioner
The sql include pipeline filters and projections property
The sql include pipeline filters and projections property
A boolean flag to enable or disable pushing down filters and projections into MongoDB when using spark sql.
A false value will be expensive as all data will be sent to spark and filtered in Spark.
Default: true
2.3
The sql include null filters in the pipeline property
The sql include null filters in the pipeline property
A boolean flag to enable or disable pushing null value checks into MongoDB when using spark sql. These ensure that the value exists and is not null for each not nullable field.
Default: true
2.3
The pipeline property
The pipeline property
Enables custom aggregation pipelines to applied to the collection before sending to Spark. When configuring this should either be an extended json representation of a list of documents:
"""[{"$match": {"closed": false}}, {"$project": {"status": 1, "name": 1, "description": 1}}]"""
Or the extended json syntax of a single document:
"""{"$match": {"closed": false}}"""
Note: Custom aggregation pipelines must work with the partitioner strategy. Some stages such as $group may not work as expected.
2.3.1
The ReadConcern level property
The ReadConcern level property
Default: DEFAULT
The ReadPreference name property
The ReadPreference name property
Default: primary
The ReadPreference tags property
The ReadPreference tags property
Register SQL Helper functions
Register SQL Helper functions
The SQL helper functions allow easy querying of Bson types inside SQL queries
1.1
The sample pool size property
The sample pool size property
The size of the pool to take a sample from, used when there is no $sample support or if there is a pushed down aggregation. Can be
used to significantly reduce the costs of inferring the schema. A negative value disables limiting when using $sample and
will sample from the whole collection.
Default: 10000
2.3.1
The sample size property
The sample size property
Used when sampling data from MongoDB to determine the Schema. Should be equal or less than the sample pool size.
Default: 1000
Strip the prefix from options
Strip the prefix from options
options that may contain the prefix
prefixLess options
Create a configuration from the sqlContext
Create a configuration from the sqlContext
Uses the prefixed properties that are set in the Spark configuration to create the config.
the SQL context
the configuration
(Since version 2.0.0) As of Spark 2.0 SQLContext was replaced by SparkSession. Use the SparkSession method instead
Mongo input configurations
Configurations used when reading from MongoDB
Configuration Properties
The prefix when using
sparkConfis:spark.mongodb.input.followed by the property name:ReadPreferenceto use.ReadPreferenceTagSets to use.ReadConcernlevel to use.1.0
com.mongodb.spark.config.ReadConfig$