The aggregation configuration
Mongo Spark Configurations
Mongo Spark Configurations
Defines helper methods for transforming or updating configurations.
1.0
Configurations for connecting to a specific collection in a database
Configurations for connecting to a specific collection in a database
1.0
The Mongo configuration base trait
The Mongo configuration base trait
Defines companion object helper methods for creating MongoConfig
instances
1.0
Mongo input configurations
Mongo input configurations
Configurations used when reading from MongoDB
The prefix when using sparkConf
is: spark.mongodb.input.
followed by the property name:
ReadPreference
to use.ReadPreference
TagSets to use.ReadConcern
level to use.1.0
Mongo output configurations
Mongo output configurations
Configurations used when writing data from Spark into MongoDB
outputProperties
1.0
The ReadConcern
configuration used by the ReadConfig.
The ReadConcern
configuration used by the ReadConfig.
the optional read concern level. If None the servers default level will be used.
1.0
Read Configuration used when reading data from MongoDB
Read Configuration used when reading data from MongoDB
the database name
the collection name
the optional connection string used in the creation of this configuration
a positive integer sample size to draw from the collection when inferring the schema
the class name of the partitioner to use to create partitions
the configuration options for the partitioner
the local threshold in milliseconds used when choosing among multiple MongoDB servers to send a request. Only servers whose ping time is less than or equal to the server with the fastest ping time plus the local threshold will be chosen.
the readPreference configuration
the readConcern configuration
the aggregation configuration
true to register sql helper functions
true to detect MapTypes when inferring Schema.
the minimum number of keys before a document can be inferred as a MapType.
true to include and push down null and exists filters into the pipeline when using sql.
true to push down filters and projections into the pipeline when using sql.
the size of the pool to take a sample from, used when there is no $sample
support or if there is a pushed down aggregation
the optional size for the internal batches used within the cursor
1.0
The ReadPreference
configuration used by the ReadConfig
The ReadPreference
configuration used by the ReadConfig
the read preference name
optional string of tagSets
1.0
The WriteConcern
configuration used by the WriteConfig
The WriteConcern
configuration used by the WriteConfig
the optional w integer value
the optional w string value
the optional journal value
the optional timeout value
1.0
Write Configuration for writes to MongoDB
Write Configuration for writes to MongoDB
the database name
the collection name
the optional connection string used in the creation of this configuration.
replaces the whole document, when saving a Dataset that contains an _id
field.
If false only updates / sets the fields declared in the Dataset.
the maxBatchSize when performing a bulk update/insert. Defaults to 512.
the local threshold in milliseconds used when choosing among multiple MongoDB servers to send a request. Only servers whose ping time is less than or equal to the server with the fastest ping time plus the local threshold will be chosen.
the write concern configuration
an optional shardKey in extended json form: "{key: 1, key2: 1}"
. Used when upserting DataSets in sharded clusters.
if true forces the writes to be inserts, even if a Dataset contains an _id
field. Default false
.
configures the bulk operation ordered property. Defaults to true.
the data contains extended bson types and any datasets that contain structs that follow the extended bson
types will automatically be converted into native bson types. For example the following _id
field would be
converted into an ObjectId: {_id: {oid: "000000000000000000000000"}}
1.0
The AggregationConfig
companion object.
The AggregationConfig
companion object.
2.3
This object was generated by sbt-buildinfo.
The ReadConcernConfig
companion object
The ReadConcernConfig
companion object
1.0
The ReadConfig
companion object
The ReadConfig
companion object
The prefix when using sparkConf
is: spark.mongodb.input.
followed by the property name:
ReadPreference
to use.ReadPreference
TagSets to use.ReadConcern
level to use.1.0
The ReadPreferenceConfig
companion object
The ReadPreferenceConfig
companion object
1.0
The WriteConcernConfig
companion object
The WriteConcernConfig
companion object
1.0
The WriteConfig
companion object
The WriteConfig
companion object
1.0
The aggregation configuration
the optional collation config
the optional hint document in extended json format
the optional aggregation pipeline, either a list of documents in json syntax or a single document in json syntax
enables writing to temporary files