com.datastax.spark.connector.rdd
This is slightly different than Scala this.
This is slightly different than Scala this.type.
this.type is the unique singleton type of an object which is not compatible with other
instances of the same type, so returning anything other than this is not really possible
without lying to the compiler by explicit casts.
Here SelfType is used to return a copy of the object - a different instance of the same type
Use a specific CassandraPartitioner to use with this PairRDD.
Use a specific CassandraPartitioner to use with this PairRDD.
Use the CassandraPartitioner from another CassandraTableScanRDD which shares the same key type.
Use the CassandraPartitioner from another CassandraTableScanRDD which shares the same key type. All Partition Keys columns must also be present in the keys of the target RDD.
Maps each row into object of a different type using provided function taking column value(s) as argument(s).
Maps each row into object of a different type using provided function taking column value(s) as argument(s). Can be used to convert each row to a tuple or a case class object:
sc.cassandraTable("ks", "table") .select("column1") .as((s: String) => s) // yields CassandraRDD[String] sc.cassandraTable("ks", "table") .select("column1", "column2") .as((_: String, _: Long)) // yields CassandraRDD[(String, Long)] case class MyRow(key: String, value: Long) sc.cassandraTable("ks", "table") .select("column1", "column2") .as(MyRow) // yields CassandraRDD[MyRow]
Counts the number of items in this RDD by selecting count(*) on Cassandra table
Counts the number of items in this RDD by selecting count(*) on Cassandra table
Adds a CQL ORDER BY clause to the query.
Adds a CQL ORDER BY clause to the query.
It can be applied only in case there are clustering columns and primary key predicate is
pushed down in where.
It is useful when the default direction of ordering rows within a single Cassandra partition
needs to be changed.
This method overrides the default spark behavior and will not create a CoalesceRDD.
This method overrides the default spark behavior and will not create a CoalesceRDD. Instead it will reduce the number of partitions by adjusting the partitioning of C* data on read. Using this method will override spark.cassandra.input.split.size. The method is useful with where() method call, when actual size of data is smaller then the table size. It has no effect if a partition key is used in where clause.
number of partitions
whether to call shuffle after
is ignored if no shuffle, or just passed to shuffled CoalesceRDD
new CassandraTableScanRDD with predefined number of partitions
Allows to copy this RDD with changing some of the properties
Allows to copy this RDD with changing some of the properties
Delete data from Cassandra table, using data from the RDD as primary keys.
Delete data from Cassandra table, using data from the RDD as primary keys. Uses the specified column names.
the name of the Keyspace to use
the name of the Table to use
The list of column names to delete, empty ColumnSelector means full row.
Primary key columns selector, Optional. All RDD primary columns columns will be checked by default
additional configuration object allowing to set consistency level, batch size, etc.
Uses the data from RDD to join with a Cassandra table without retrieving the entire table.
Uses the data from RDD to join with a Cassandra table without retrieving the entire table. Any RDD which can be used to saveToCassandra can be used to joinWithCassandra as well as any RDD which only specifies the partition Key of a Cassandra Table. This method executes single partition requests against the Cassandra Table and accepts the functional modifiers that a normal com.datastax.spark.connector.rdd.CassandraTableScanRDD takes.
By default this method only uses the Partition Key for joining but any combination of columns which are acceptable to C* can be used in the join. Specify columns using joinColumns as a parameter or the on() method.
Example With Prior Repartitioning:
val source = sc.parallelize(keys).map(x => new KVRow(x)) val repart = source.repartitionByCassandraReplica(keyspace, tableName, 10) val someCass = repart.joinWithCassandraTable(keyspace, tableName)
Example Joining on Clustering Columns:
val source = sc.parallelize(keys).map(x => (x, x * 100)) val someCass = source.joinWithCassandraTable(keyspace, wideTable).on(SomeColumns("key", "group"))
Shortcut for rdd.keyBy[K].applyPartitionerFrom(thatRDD[K, V]) where K is the key
type of the target RDD.
Shortcut for rdd.keyBy[K].applyPartitionerFrom(thatRDD[K, V]) where K is the key
type of the target RDD. This guarentees that the partitioner applied to this rdd
will match the key type.
Extracts a key of the given class from all the available columns.
Extracts a key of the given class from all the available columns.
keyBy(ColumnSelector)
Extracts a key of the given class from the given columns.
Extracts a key of the given class from the given columns.
keyBy(ColumnSelector)
Selects a subset of columns mapped to the key and returns an RDD of pairs.
Selects a subset of columns mapped to the key and returns an RDD of pairs. Similar to the builtin Spark keyBy method, but this one uses implicit RowReaderFactory to construct the key objects. The selected columns must be available in the CassandraRDD.
If the selected columns contain the complete partition key a
CassandraPartitioner will also be created.
column selector passed to the rrf to create the row reader, useful when the key is mapped to a tuple or a single value
Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row.
Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row. The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.
Uses the data from RDD to left join with a Cassandra table without retrieving the entire table.
Uses the data from RDD to left join with a Cassandra table without retrieving the entire table. Any RDD which can be used to saveToCassandra can be used to leftJoinWithCassandra as well as any RDD which only specifies the partition Key of a Cassandra Table. This method executes single partition requests against the Cassandra Table and accepts the functional modifiers that a normal com.datastax.spark.connector.rdd.CassandraTableScanRDD takes.
By default this method only uses the Partition Key for joining but any combination of columns which are acceptable to C* can be used in the join. Specify columns using joinColumns as a parameter or the on() method.
Example With Prior Repartitioning:
val source = sc.parallelize(keys).map(x => new KVRow(x)) val repart = source.repartitionByCassandraReplica(keyspace, tableName, 10) val someCass = repart.leftJoinWithCassandraTable(keyspace, tableName)
Example Joining on Clustering Columns:
val source = sc.parallelize(keys).map(x => (x, x * 100)) val someCass = source.leftJoinWithCassandraTable(keyspace, wideTable).on(SomeColumns("key", "group"))
Adds the limit clause to CQL select statement.
Adds the limit clause to CQL select statement. The limit will be applied for each created Spark partition. In other words, unless the data are fetched from a single Cassandra partition the number of results is unpredictable.
The main purpose of passing limit clause is to fetch top n rows from a single Cassandra partition when the table is designed so that it uses clustering keys and a partition key predicate is passed to the where clause.
Filters currently selected set of columns with a new set of columns
Filters currently selected set of columns with a new set of columns
Adds the PER PARTITION LIMIT clause to CQL select statement.
Adds the PER PARTITION LIMIT clause to CQL select statement. The limit will be applied for every Cassandra Partition. Only Valid For Cassandra 3.6+
Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName.
Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName.
Calling this method before using joinWithCassandraTable will ensure that requests will be coordinator
local. partitionsPerHost Controls the number of Spark Partitions that will be created in this repartitioning
event.
The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.
RowReaderFactory and ClassTag should be provided from implicit parameters in the constructor of the class implementing this trait
RowReaderFactory and ClassTag should be provided from implicit parameters in the constructor of the class implementing this trait
CassandraTableScanRDD
Saves the data from RDD to a new table with definition taken from the
ColumnMapper for this class.
Saves the data from RDD to a new table with definition taken from the
ColumnMapper for this class.
keyspace where to create a new table
name of the table to create; the table must not exist
Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.
additional configuration object allowing to set consistency level, batch size, etc.
optional, implicit connector to Cassandra
factory for obtaining the row writer to be used to extract column values from items of the RDD
a column mapper determining the definition of the table
Saves the data from RDD to a new table defined by the given TableDef.
Saves the data from RDD to a new table defined by the given TableDef.
First it creates a new table with all columns from the TableDef
and then it saves RDD content in the same way as saveToCassandra.
The table must not exist prior to this call.
table definition used to create a new table
Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.
additional configuration object allowing to set consistency level, batch size, etc.
optional, implicit connector to Cassandra
factory for obtaining the row writer to be used to extract column values from items of the RDD
Saves the data from RDD to a Cassandra table.
Saves the data from RDD to a Cassandra table. Uses the specified column names.
the name of the Keyspace to use
the name of the Table to use
additional configuration object allowing to set consistency level, batch size, etc.
Narrows down the selected set of columns.
Narrows down the selected set of columns.
Use this for better performance, when you don't need all the columns in the result RDD.
When called multiple times, it selects the subset of the already selected columns, so
after a column was removed by the previous select call, it is not possible to
add it back.
The selected columns are ColumnRef instances. This type allows to specify columns for
straightforward retrieval and to read TTL or write time of regular columns as well. Implicit
conversions included in com.datastax.spark.connector package make it possible to provide
just column names (which is also backward compatible) and optional add .ttl or .writeTime
suffix in order to create an appropriate ColumnRef instance.
Returns the columns to be selected from the table.
Returns the columns to be selected from the table.
Applies a function to each item, and groups consecutive items having the same value together.
Applies a function to each item, and groups consecutive items having the same value together.
Contrary to groupBy, items from the same group must be already next to each other in the
original collection. Works locally on each partition, so items from different
partitions will never be placed in the same group.
Groups items with the same key, assuming the items with the same key are next to each other in the collection.
Groups items with the same key, assuming the items with the same key are next to each other
in the collection. It does not perform shuffle, therefore it is much faster than using
much more universal Spark RDD groupByKey. For this method to be useful with Cassandra tables,
the key must represent a prefix of the primary key, containing at least the partition key of the
Cassandra table.
Checks for existence of keyspace and table.
Checks for existence of keyspace and table.
Adds a CQL WHERE predicate(s) to the query.
Adds a CQL WHERE predicate(s) to the query.
Useful for leveraging secondary indexes in Cassandra.
Implicitly adds an ALLOW FILTERING clause to the WHERE clause,
however beware that some predicates might be rejected by Cassandra,
particularly in cases when they filter on an unindexed, non-clustering column.
Returns a copy of this Cassandra RDD with specified connector
Returns a copy of this Cassandra RDD with specified connector
Allows to set custom read configuration, e.
Allows to set custom read configuration, e.g. consistency level or fetch size.
(cassandraTableScanRDD: DoubleRDDFunctions).histogram(buckets, evenBuckets)
(cassandraTableScanRDD: DoubleRDDFunctions).histogram(bucketCount)
(cassandraTableScanRDD: DoubleRDDFunctions).histogram(buckets, evenBuckets)
(cassandraTableScanRDD: DoubleRDDFunctions).histogram(bucketCount)
(cassandraTableScanRDD: DoubleRDDFunctions).mean()
(cassandraTableScanRDD: DoubleRDDFunctions).mean()
(cassandraTableScanRDD: DoubleRDDFunctions).meanApprox(timeout, confidence)
(cassandraTableScanRDD: DoubleRDDFunctions).meanApprox(timeout, confidence)
(cassandraTableScanRDD: DoubleRDDFunctions).sampleStdev()
(cassandraTableScanRDD: DoubleRDDFunctions).sampleStdev()
(cassandraTableScanRDD: DoubleRDDFunctions).sampleVariance()
(cassandraTableScanRDD: DoubleRDDFunctions).sampleVariance()
(cassandraTableScanRDD: StringAdd).self
(cassandraTableScanRDD: StringFormat).self
(cassandraTableScanRDD: RDDFunctions[R]).sparkContext
(cassandraTableScanRDD: DoubleRDDFunctions).stats()
(cassandraTableScanRDD: DoubleRDDFunctions).stats()
(cassandraTableScanRDD: DoubleRDDFunctions).stdev()
(cassandraTableScanRDD: DoubleRDDFunctions).stdev()
(cassandraTableScanRDD: DoubleRDDFunctions).sum()
(cassandraTableScanRDD: DoubleRDDFunctions).sum()
(cassandraTableScanRDD: DoubleRDDFunctions).sumApprox(timeout, confidence)
(cassandraTableScanRDD: DoubleRDDFunctions).sumApprox(timeout, confidence)
(cassandraTableScanRDD: DoubleRDDFunctions).variance()
(cassandraTableScanRDD: DoubleRDDFunctions).variance()
(cassandraTableScanRDD: ArrowAssoc[CassandraTableScanRDD[R]]).x
(Since version 2.10.0) Use leftOfArrow instead
(cassandraTableScanRDD: Ensuring[CassandraTableScanRDD[R]]).x
(Since version 2.10.0) Use resultOfEnsuring instead
RDD representing a Table Scan of A Cassandra table.
This class is the main entry point for analyzing data in Cassandra database with Spark. Obtain objects of this class by calling com.datastax.spark.connector.SparkContextFunctions.cassandraTable.
Configuration properties should be passed in the SparkConf configuration of SparkContext.
CassandraRDDneeds to open connection to Cassandra, therefore it requires appropriate connection property values to be present in SparkConf. For the list of required and available properties, see CassandraConnector.CassandraRDDdivides the data set into smaller partitions, processed locally on every cluster node. A data partition consists of one or more contiguous token ranges. To reduce the number of roundtrips to Cassandra, every partition is fetched in batches.The following properties control the number of partitions and the fetch size: - spark.cassandra.input.split.size_in_mb: approx amount of data to be fetched into a single Spark partition, default 64 MB - spark.cassandra.input.fetch.size_in_rows: number of CQL rows fetched per roundtrip, default 1000
A
CassandraRDDobject gets serialized and sent to every Spark Executor, which then calls thecomputemethod to fetch the data on every node. ThegetPreferredLocationsmethod tells Spark the preferred nodes to fetch a partition from, so that the data for the partition are at the same node the task was sent to. If Cassandra nodes are collocated with Spark nodes, the queries are always sent to the Cassandra process running on the same node as the Spark Executor process, hence data are not transferred between nodes. If a Cassandra node fails or gets overloaded during read, the queries are retried to a different node.By default, reads are performed at ConsistencyLevel.LOCAL_ONE in order to leverage data-locality and minimize network traffic. This read consistency level is controlled by the spark.cassandra.input.consistency.level property.