org.apache.cassandra.hadoop.pig

Class CqlStorage

java.lang.Object

org.apache.pig.LoadFunc

org.apache.cassandra.hadoop.pig.AbstractCassandraStorage

org.apache.cassandra.hadoop.pig.CqlStorage

All Implemented Interfaces:

org.apache.pig.LoadMetadata, org.apache.pig.StoreFuncInterface

Direct Known Subclasses:

CqlNativeStorage

public class CqlStorage
extends AbstractCassandraStorage

A LoadStoreFunc for retrieving data from and storing data to Cassandra A row from a standard CF will be returned as nested tuples: (((key1, value1), (key2, value2)), ((name1, val1), (name2, val2))).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.cassandra.hadoop.pig.AbstractCassandraStorage

AbstractCassandraStorage.CfInfo, AbstractCassandraStorage.MarshallerType

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	columns
protected java.lang.String	outputQuery
protected int	pageSize
protected java.lang.String	whereClause
protected org.apache.hadoop.mapreduce.RecordWriter<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.List<java.nio.ByteBuffer>>	writer

Fields inherited from class org.apache.cassandra.hadoop.pig.AbstractCassandraStorage

column_family, conf, DEFAULT_INPUT_FORMAT, DEFAULT_OUTPUT_FORMAT, initHostAddress, inputFormatClass, keyspace, loadSignature, outputFormatClass, PARTITION_FILTER_SIGNATURE, partitionerClass, password, PIG_INITIAL_ADDRESS, PIG_INPUT_FORMAT, PIG_INPUT_INITIAL_ADDRESS, PIG_INPUT_PARTITIONER, PIG_INPUT_RPC_PORT, PIG_INPUT_SPLIT_SIZE, PIG_OUTPUT_FORMAT, PIG_OUTPUT_INITIAL_ADDRESS, PIG_OUTPUT_PARTITIONER, PIG_OUTPUT_RPC_PORT, PIG_PARTITIONER, PIG_RPC_PORT, rpcPort, splitSize, storeSignature, usePartitionFilter, username

Constructor Summary

Constructors

Constructor and Description
CqlStorage()
CqlStorage(int pageSize)

Method Summary

Methods

Modifier and Type	Method and Description
protected java.lang.Object	cqlColumnToObj(Cell col, org.apache.cassandra.thrift.CfDef cfDef) convert a cql column to an object
protected java.util.List<org.apache.cassandra.thrift.ColumnDef>	getColumnMetadata(org.apache.cassandra.thrift.Cassandra.Client client) include key columns
protected java.util.List<org.apache.cassandra.thrift.ColumnDef>	getKeysMeta(org.apache.cassandra.thrift.Cassandra.Client client) get keys meta data
org.apache.pig.data.Tuple	getNext() get next row
org.apache.pig.ResourceSchema	getSchema(java.lang.String location, org.apache.hadoop.mapreduce.Job job) schema: (value, value, value) where keys are in the front.
java.nio.ByteBuffer	nullToBB() Thrift API can't handle null, so use empty byte array
void	prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader, org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit split)
void	prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)
void	putNext(org.apache.pig.data.Tuple t) output: (((name, value), (name, value)), (value ...
void	setLocation(java.lang.String location, org.apache.hadoop.mapreduce.Job job) set read configuration settings
void	setPartitionFilter(org.apache.pig.Expression partitionFilter)
void	setStoreLocation(java.lang.String location, org.apache.hadoop.mapreduce.Job job) set store configuration settings
protected void	setTupleValue(org.apache.pig.data.Tuple tuple, int position, java.lang.Object value, AbstractType<?> validator) set the value to the position of the tuple

Methods inherited from class org.apache.cassandra.hadoop.pig.AbstractCassandraStorage

cassandraToObj, cfdefFromString, cfdefToString, checkSchema, cleanupOnFailure, cleanupOnSuccess, columnToTuple, composeComposite, getCfInfo, getCfInfo, getCFMetaData, getColumnMeta, getDefaultMarshallers, getFullyQualifiedClassName, getIndexes, getIndexType, getInputFormat, getOutputFormat, getPartitionKeys, getPigType, getQueryMap, getStatistics, getValidatorMap, initSchema, objToBB, parseType, relativeToAbsolutePath, relToAbsPathForStoreLocation, setConnectionInformation, setStoreFuncUDFContextSignature, setTupleValue, setUDFContextSignature

Methods inherited from class org.apache.pig.LoadFunc

getAbsolutePath, getLoadCaster, getPathStrings, join, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

writer

protected org.apache.hadoop.mapreduce.RecordWriter<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.List<java.nio.ByteBuffer>> writer

pageSize

protected int pageSize

columns

protected java.lang.String columns

outputQuery

protected java.lang.String outputQuery

whereClause

protected java.lang.String whereClause

Constructor Detail

CqlStorage

public CqlStorage()

CqlStorage

public CqlStorage(int pageSize)

Parameters:

pageSize - limit number of CQL rows to fetch in a thrift request

Method Detail

prepareToRead

public void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
                 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit split)

Specified by:

prepareToRead in class org.apache.pig.LoadFunc

getNext

public org.apache.pig.data.Tuple getNext()
                                  throws java.io.IOException

get next row

Specified by:

getNext in class org.apache.pig.LoadFunc

Throws:

java.io.IOException

setTupleValue

protected void setTupleValue(org.apache.pig.data.Tuple tuple,
                 int position,
                 java.lang.Object value,
                 AbstractType<?> validator)
                      throws org.apache.pig.backend.executionengine.ExecException

set the value to the position of the tuple

Throws:

org.apache.pig.backend.executionengine.ExecException

cqlColumnToObj

protected java.lang.Object cqlColumnToObj(Cell col,
                              org.apache.cassandra.thrift.CfDef cfDef)
                                   throws java.io.IOException

convert a cql column to an object

Throws:

java.io.IOException

setLocation

public void setLocation(java.lang.String location,
               org.apache.hadoop.mapreduce.Job job)
                 throws java.io.IOException

set read configuration settings

Specified by:

setLocation in class org.apache.pig.LoadFunc

Throws:

java.io.IOException

setStoreLocation

public void setStoreLocation(java.lang.String location,
                    org.apache.hadoop.mapreduce.Job job)
                      throws java.io.IOException

set store configuration settings

Throws:

java.io.IOException

getSchema

public org.apache.pig.ResourceSchema getSchema(java.lang.String location,
                                      org.apache.hadoop.mapreduce.Job job)
                                        throws java.io.IOException

schema: (value, value, value) where keys are in the front.

Throws:

java.io.IOException

setPartitionFilter

public void setPartitionFilter(org.apache.pig.Expression partitionFilter)
                        throws java.io.IOException

Throws:

java.io.IOException

prepareToWrite

public void prepareToWrite(org.apache.hadoop.mapreduce.RecordWriter writer)

putNext

public void putNext(org.apache.pig.data.Tuple t)
             throws java.io.IOException

output: (((name, value), (name, value)), (value ... value), (value...value))

Throws:

java.io.IOException

getColumnMetadata

protected java.util.List<org.apache.cassandra.thrift.ColumnDef> getColumnMetadata(org.apache.cassandra.thrift.Cassandra.Client client)
                                                                           throws org.apache.cassandra.thrift.InvalidRequestException,
                                                                                  org.apache.cassandra.thrift.UnavailableException,
                                                                                  org.apache.cassandra.thrift.TimedOutException,
                                                                                  org.apache.cassandra.thrift.SchemaDisagreementException,
                                                                                  org.apache.thrift.TException,
                                                                                  java.nio.charset.CharacterCodingException,
                                                                                  InvalidRequestException,
                                                                                  ConfigurationException,
                                                                                  org.apache.cassandra.thrift.NotFoundException

include key columns

Specified by:

getColumnMetadata in class AbstractCassandraStorage

Throws:

org.apache.cassandra.thrift.InvalidRequestException

org.apache.cassandra.thrift.UnavailableException

org.apache.cassandra.thrift.TimedOutException

org.apache.cassandra.thrift.SchemaDisagreementException

org.apache.thrift.TException

java.nio.charset.CharacterCodingException

ConfigurationException

org.apache.cassandra.thrift.NotFoundException

getKeysMeta

protected java.util.List<org.apache.cassandra.thrift.ColumnDef> getKeysMeta(org.apache.cassandra.thrift.Cassandra.Client client)
                                                                     throws java.lang.Exception

get keys meta data

Throws:

java.lang.Exception

nullToBB

public java.nio.ByteBuffer nullToBB()

Thrift API can't handle null, so use empty byte array

Specified by:

nullToBB in class AbstractCassandraStorage