SnappyContext
Related Docs:
object SnappyContext
| package sql
alter table adds/drops provided column, only supprted for row tables.
alter table adds/drops provided column, only supprted for row tables. For adding a column isAddColumn should be true, else it will be drop column
Append dataframe to cache table in Spark.
Append dataframe to cache table in Spark.
default storage level is MEMORY_AND_DISK
@todo -> return type?
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support. Java friendly api.
the qualified name of the top-K structure
the base table of the top-K structure, if any, or null
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support.
the qualified name of the top-K structure
the base table of the top-K structure, if any
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support. Java friendly api.
the qualified name of the top-K structure
the base table of the top-K structure, if any, or null
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support.
the qualified name of the top-K structure
the base table of the top-K structure, if any
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g.
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples). This method handles generic array datatype like Array[Decimal]
Create an index on a table.
Create an index on a table.
Index name which goes in the catalog
Fully qualified name of table on which the index is created.
Columns on which the index has to be created with the direction of sorting. Direction can be specified as None.
Options for indexes. For e.g. column table index - ("COLOCATE_WITH"->"CUSTOMER"). row table index - ("INDEX_TYPE"->"GLOBAL HASH") or ("INDEX_TYPE"->"UNIQUE")
Create an index on a table.
Create an index on a table.
Index name which goes in the catalog
Fully qualified name of table on which the index is created.
Columns on which the index has to be created along with the sorting direction.The direction of index will be ascending if value is true and descending when value is false. Direction can be specified as null
Options for indexes. For e.g. column table index - ("COLOCATE_WITH"->"CUSTOMER"). row table index - ("INDEX_TYPE"->"GLOBAL HASH") or ("INDEX_TYPE"->"UNIQUE")
Create a stratified sample table.
Create a stratified sample table. Java friendly version.
the qualified name of the table
the base table of the sample table, if any, or null
schema of the table
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table.
the qualified name of the table
the base table of the sample table, if any
schema of the table
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table. Java friendly version.
the qualified name of the table
the base table of the sample table, if any, or null
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table.
the qualified name of the table
the base table of the sample table, if any
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Creates a SnappyData managed JDBC table which takes a free format ddl string.
Creates a SnappyData managed JDBC table which takes a free format ddl string. The ddl string should adhere to syntax of underlying JDBC store. SnappyData ships with inbuilt JDBC store, which can be accessed by Row format data store. The option parameter can take connection details.
val props = Map( "url" -> s"jdbc:derby:$path", "driver" -> "org.apache.derby.jdbc.EmbeddedDriver", "poolImpl" -> "tomcat", "user" -> "app", "password" -> "app" ) val schemaDDL = "(OrderId INT NOT NULL PRIMARY KEY,ItemId INT, ITEMREF INT)" snappyContext.createTable("jdbcTable", "jdbc", schemaDDL, props)
Any DataFrame of the same schema can be inserted into the JDBC table using DataFrameWriter API.
e.g.
case class Data(col1: Int, col2: Int, col3: Int) val data = Seq(Seq(1, 2, 3), Seq(7, 8, 9), Seq(9, 2, 3), Seq(4, 2, 3), Seq(5, 6, 7)) val rdd = sc.parallelize(data, data.length).map(s => new Data(s(0), s(1), s(2))) val dataDF = snc.createDataFrame(rdd) dataDF.write.insertInto("jdbcTable")
Name of the table
Provider name 'ROW' or 'JDBC'.
Table schema as a string interpreted by provider
Properties for table creation. See options list for different tables. https://github.com/SnappyDataInc/snappydata/blob/master/docs/rowAndColumnTables.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed JDBC table which takes a free format ddl string.
Creates a SnappyData managed JDBC table which takes a free format ddl string. The ddl string should adhere to syntax of underlying JDBC store. SnappyData ships with inbuilt JDBC store, which can be accessed by Row format data store. The option parameter can take connection details.
val props = Map( "url" -> s"jdbc:derby:$path", "driver" -> "org.apache.derby.jdbc.EmbeddedDriver", "poolImpl" -> "tomcat", "user" -> "app", "password" -> "app" ) val schemaDDL = "(OrderId INT NOT NULL PRIMARY KEY,ItemId INT, ITEMREF INT)" snappyContext.createTable("jdbcTable", "jdbc", schemaDDL, props)
Any DataFrame of the same schema can be inserted into the JDBC table using DataFrameWriter API.
e.g.
case class Data(col1: Int, col2: Int, col3: Int) val data = Seq(Seq(1, 2, 3), Seq(7, 8, 9), Seq(9, 2, 3), Seq(4, 2, 3), Seq(5, 6, 7)) val rdd = sc.parallelize(data, data.length).map(s => new Data(s(0), s(1), s(2))) val dataDF = snc.createDataFrame(rdd) dataDF.write.insertInto("jdbcTable")
Name of the table
Provider name 'ROW' or 'JDBC'.
Table schema as a string interpreted by provider
Properties for table creation. See options list for different tables. https://github.com/SnappyDataInc/snappydata/blob/master/docs/rowAndColumnTables.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
case class Data(col1: Int, col2: Int, col3: Int) val props = Map.empty[String, String] val data = Seq(Seq(1, 2, 3), Seq(7, 8, 9), Seq(9, 2, 3), Seq(4, 2, 3), Seq(5, 6, 7)) val rdd = sc.parallelize(data, data.length).map(s => new Data(s(0), s(1), s(2))) val dataDF = snc.createDataFrame(rdd) snappyContext.createTable(tableName, "column", dataDF.schema, props)
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW', 'JDBC' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/SnappyDataInc/snappydata/blob/master/docs/rowAndColumnTables.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
case class Data(col1: Int, col2: Int, col3: Int) val props = Map.empty[String, String] val data = Seq(Seq(1, 2, 3), Seq(7, 8, 9), Seq(9, 2, 3), Seq(4, 2, 3), Seq(5, 6, 7)) val rdd = sc.parallelize(data, data.length).map(s => new Data(s(0), s(1), s(2))) val dataDF = snc.createDataFrame(rdd) snappyContext.createTable(tableName, "column", dataDF.schema, props)
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW', 'JDBC' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/SnappyDataInc/snappydata/blob/master/docs/rowAndColumnTables.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
val airlineDF = snappyContext.createTable(stagingAirline, "column", Map("buckets" -> "29"))
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW', 'JDBC', 'PARQUET' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
val airlineDF = snappyContext.createTable(stagingAirline, "column", Map("buckets" -> "29"))
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW', 'JDBC', 'PARQUET' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Delete all rows in table that match passed filter expression
Delete all rows in table that match passed filter expression
table name
SQL WHERE criteria to select rows that will be updated
number of rows deleted
Drops an index on a table
Drops an index on a table
Index name which goes in catalog
Drop if exists, else exit gracefully
Drop a SnappyData table created by a call to SnappyContext.createTable, createExternalTable or registerTempTable.
Drop a SnappyData table created by a call to SnappyContext.createTable, createExternalTable or registerTempTable.
table to be dropped
attempt drop only if the table exists
Insert one or more org.apache.spark.sql.Row into an existing table
Insert one or more org.apache.spark.sql.Row into an existing table
java.util.ArrayList[java.util.ArrayList[_] rows = ... * snc.insert(tableName, rows)
number of rows inserted
Insert one or more org.apache.spark.sql.Row into an existing table
Insert one or more org.apache.spark.sql.Row into an existing table
snc.insert(tableName, dataDF.collect(): _*)
number of rows inserted
Upsert one or more org.apache.spark.sql.Row into an existing table
Upsert one or more org.apache.spark.sql.Row into an existing table
java.util.ArrayList[java.util.ArrayList[_] rows = ... * snSession.put(tableName, rows)
Upsert one or more org.apache.spark.sql.Row into an existing table
Upsert one or more org.apache.spark.sql.Row into an existing table
snSession.put(tableName, dataDF.collect(): _*)
why do we need this method? K is optional in the above method
Fetch the topK entries in the Approx TopK synopsis for the specified time interval.
Fetch the topK entries in the Approx TopK synopsis for the specified time interval. See _createTopK_ for how to create this data structure and associate this to a base table (i.e. the full data set). The time interval specified here should not be less than the minimum time interval used when creating the TopK synopsis.
- The topK structure that is to be queried.
start time as string of the format "yyyy-mm-dd hh:mm:ss". If passed as null, oldest interval is considered as the start interval.
end time as string of the format "yyyy-mm-dd hh:mm:ss". If passed as null, newest interval is considered as the last interval.
Optional. Number of elements to be queried. This is to be passed only for stream summary
returns the top K elements with their respective frequencies between two time
provide an example and explain the returned DataFrame. Key is the attribute stored but the value is a struct containing count_estimate, and lower, upper bounds? How many elements are returned if K is not specified?
:: DeveloperApi ::
:: DeveloperApi ::
do we need this anymore? If useful functionality, make this private to sql package ... SchemaDStream should use the data source API? Tagging as developer API, for now
Set current database/schema.
Set current database/schema.
schema name which goes in the catalog
Run SQL string without any plan caching.
Empties the contents of the table without deleting the catalog entry.
Empties the contents of the table without deleting the catalog entry.
full table name to be truncated
attempt truncate only if the table exists
Update all rows in table that match passed filter expression
Update all rows in table that match passed filter expression
snappyContext.update("jdbcTable", "ITEMREF = 3" , Row(99) , "ITEMREF" )
table name which needs to be updated
SQL WHERE criteria to select rows that will be updated
A list containing all the updated column values. They MUST match the updateColumn list passed
List of all column names being updated
Update all rows in table that match passed filter expression
Update all rows in table that match passed filter expression
snappyContext.update("jdbcTable", "ITEMREF = 3" , Row(99) , "ITEMREF" )
table name which needs to be updated
SQL WHERE criteria to select rows that will be updated
A single Row containing all updated column values. They MUST match the updateColumn list passed
List of all column names being updated
(Since version 1.3.0) Use createDataFrame instead.
(Since version 1.3.0) Use createDataFrame instead.
(Since version 1.3.0) Use createDataFrame instead.
(Since version 1.3.0) Use createDataFrame instead.
(Since version 1.4.0) Use read.jdbc() instead.
(Since version 1.4.0) Use read.jdbc() instead.
(Since version 1.4.0) Use read.jdbc() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.json() instead.
(Since version 1.4.0) Use read.format(source).schema(schema).options(options).load() instead.
(Since version 1.4.0) Use read.format(source).schema(schema).options(options).load() instead.
(Since version 1.4.0) Use read.format(source).options(options).load() instead.
(Since version 1.4.0) Use read.format(source).options(options).load() instead.
(Since version 1.4.0) Use read.format(source).load(path) instead.
(Since version 1.4.0) Use read.load(path) instead.
(Since version 1.4.0) Use read.parquet() instead.
Main entry point for SnappyData extensions to Spark. A SnappyContext extends Spark's org.apache.spark.sql.SQLContext to work with Row and Column tables. Any DataFrame can be managed as SnappyData tables and any table can be accessed as a DataFrame. This integrates the SQLContext functionality with the Snappy store.
When running in the embedded mode (i.e. Spark executor collocated with Snappy data store), Applications typically submit Jobs to the Snappy-JobServer (provide link) and do not explicitly create a SnappyContext. A single shared context managed by SnappyData makes it possible to re-use Executors across client connections or applications.
SnappyContext uses a HiveMetaStore for catalog , which is persistent. This enables table metadata info recreated on driver restart.
User should use obtain reference to a SnappyContext instance as below val snc: SnappyContext = SnappyContext.getOrCreate(sparkContext)
Provide links to above descriptions
,document describing the Job server API
https://github.com/SnappyDataInc/snappydata#interacting-with-snappydata
https://github.com/SnappyDataInc/snappydata#step-1---start-the-snappydata-cluster