io.cdap.cdap.api.dataset.lib

Interface PartitionedFileSet

All Superinterfaces:

AutoCloseable, Closeable, Dataset, InputFormatProvider, OutputFormatProvider

All Known Subinterfaces:

TimePartitionedFileSet

@Beta
public interface PartitionedFileSet
extends Dataset, InputFormatProvider, OutputFormatProvider

Represents a dataset that is split into partitions that can be uniquely addressed by partition keys along multiple dimensions. Each partition is a path in a file set, the partition key attached as meta data. Note that the partitioning of the dataset is fixed, that is, all operations that accept a partition key as a parameter require that that key has exactly the same schema as the partitioning. This dataset can be made available for querying with SQL (explore). This is enabled through dataset properties when the dataset is created. See FileSetProperties for details. If it is enabled for explore, a Hive external table will be created when the dataset is created. The Hive table is partitioned by the same keys as this dataset.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TYPE Type name

Method Summary

Modifier and Type	Method and Description
void	addMetadata(PartitionKey key, Map<String,String> metadata) Adds a set of new metadata entries for a particular partition.
void	addMetadata(PartitionKey key, String metadataKey, String metadataValue) Adds a new metadata entry for a particular partition.
void	addPartition(PartitionKey key, String path) Add a partition for a given partition key, stored at a given path (relative to the file set's base path).
void	addPartition(PartitionKey key, String path, Map<String,String> metadata) Add a partition for a given partition key, stored at a given path (relative to the file set's base path), with the given metadata.
Future<Void>	concatenatePartition(PartitionKey key) Asynchronous operation to concatenate the partition in Hive.
PartitionConsumerResult	consumePartitions(PartitionConsumerState partitionConsumerState) Incrementally consumes partitions.
PartitionConsumerResult	consumePartitions(PartitionConsumerState partitionConsumerState, int limit, Predicate<PartitionDetail> predicate) Incrementally consumes partitions.
void	dropPartition(PartitionKey key) Remove a partition for a given partition key, silently ignoring if the key is not found.
FileSet	getEmbeddedFileSet()
PartitionDetail	getPartition(PartitionKey key) Return the partition for a specific partition key, or null if key is not found.
Partitioning	getPartitioning() Get the partitioning declared for the file set.
PartitionOutput	getPartitionOutput(PartitionKey key) Return a partition output for a specific partition key, in preparation for creating a new partition.
Set<PartitionDetail>	getPartitions(PartitionFilter filter) Return all partitions matching the partition filter.
Map<String,String>	getRuntimeArguments() Allow direct access to the runtime arguments of this partitioned file set.
void	removeMetadata(PartitionKey key, Set<String> metadataKeys) Removes a set of metadata entries for a particular partition.
void	removeMetadata(PartitionKey key, String metadataKey) Removes a metadata entry for a particular partition.
void	setMetadata(PartitionKey key, Map<String,String> metadata) Sets metadata entries for a particular partition.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface io.cdap.cdap.api.data.batch.InputFormatProvider

getInputFormatClassName, getInputFormatConfiguration

Methods inherited from interface io.cdap.cdap.api.data.batch.OutputFormatProvider

getOutputFormatClassName, getOutputFormatConfiguration

Field Detail

TYPE

static final String TYPE

Type name

See Also:

Constant Field Values

Method Detail

getPartitioning

Partitioning getPartitioning()

Get the partitioning declared for the file set.

addPartition

void addPartition(PartitionKey key,
                  String path)

Add a partition for a given partition key, stored at a given path (relative to the file set's base path).

Throws:

PartitionAlreadyExistsException - if a partition for the same key already exists

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

addPartition

void addPartition(PartitionKey key,
                  String path,
                  Map<String,String> metadata)

Add a partition for a given partition key, stored at a given path (relative to the file set's base path), with the given metadata.

Throws:

PartitionAlreadyExistsException - if a partition for the same key already exists

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

addMetadata

void addMetadata(PartitionKey key,
                 String metadataKey,
                 String metadataValue)

Adds a new metadata entry for a particular partition. Note that existing entries cannot be updated.

Throws:

DataSetException - when an attempt is made to update an existing entry

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

addMetadata

void addMetadata(PartitionKey key,
                 Map<String,String> metadata)

Adds a set of new metadata entries for a particular partition. Note that existing entries cannot be updated.

Throws:

DataSetException - when an attempt is made to update existing entries

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

setMetadata

void setMetadata(PartitionKey key,
                 Map<String,String> metadata)

Sets metadata entries for a particular partition. If the metadata entry key does not already exist, it will be created; otherwise, it will be overwritten. Other existing keys remain unchanged.

Throws:

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

removeMetadata

void removeMetadata(PartitionKey key,
                    String metadataKey)

Removes a metadata entry for a particular partition. If the metadata key does not exist, no error is thrown.

Throws:

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

removeMetadata

void removeMetadata(PartitionKey key,
                    Set<String> metadataKeys)

Removes a set of metadata entries for a particular partition. If any metadata key does not exist, no error is thrown.

Throws:

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

dropPartition

void dropPartition(PartitionKey key)

Remove a partition for a given partition key, silently ignoring if the key is not found.

Throws:

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

concatenatePartition

Future<Void> concatenatePartition(PartitionKey key)

Asynchronous operation to concatenate the partition in Hive. Note that Hive only supports certain formats. Concatenation merges the files within one partition into fewer, larger files. If this Dataset is not enabled for Explore, this operation is a no-op.

Returns:

a Future which returns null, but may be used to await completion of the concatenation operation.

Throws:

PartitionNotFoundException - when a partition for the given key is not found

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

getPartition

@Nullable
PartitionDetail getPartition(PartitionKey key)

Return the partition for a specific partition key, or null if key is not found.

Throws:

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

getPartitions

Set<PartitionDetail> getPartitions(@Nullable
                                   PartitionFilter filter)

Return all partitions matching the partition filter.

Parameters:

filter - If non null, only partitions that match this filter are returned. If null, all partitions are returned.

consumePartitions

PartitionConsumerResult consumePartitions(PartitionConsumerState partitionConsumerState)

Incrementally consumes partitions. This method can be used to retrieve partitions that have been created since the last call to this method. Note that it is the client's responsibility to maintain state of the partitions processed in the iterator returned in the PartitionConsumerResult.

Parameters:

partitionConsumerState - the state from which to start consuming from

Returns:

PartitionConsumerResult which holds the state of consumption as well as an iterator to the consumed Partitions

consumePartitions

PartitionConsumerResult consumePartitions(PartitionConsumerState partitionConsumerState,
                                          int limit,
                                          Predicate<PartitionDetail> predicate)

Incrementally consumes partitions. This method can be used to retrieve partitions that have been created since the last call to this method. Note that it is the client's responsibility to maintain state of the partitions processed in the iterator returned in the PartitionConsumerResult.

Parameters:

partitionConsumerState - the state from which to start consuming from

limit - number of partitions, which once reached, will not add add more partitions committed by other transactions; the limit is checked after adding consuming all partitions of a transaction, so the total number of consumed partitions may be greater than this limit

predicate - a predicate which determines the partitions to be consumed

Returns:

PartitionConsumerResult which holds the state of consumption as well as an iterator to the consumed Partitions

getPartitionOutput

PartitionOutput getPartitionOutput(PartitionKey key)

Return a partition output for a specific partition key, in preparation for creating a new partition. Obtain the location to write from the PartitionOutput, then call the PartitionOutput.addPartition() to add the partition to this dataset.

Throws:

PartitionAlreadyExistsException - if the partition already exists

IllegalArgumentException - if the partition key does not match the partitioning of the dataset

getEmbeddedFileSet

FileSet getEmbeddedFileSet()

Returns:

the underlying (embedded) file set.

getRuntimeArguments

Map<String,String> getRuntimeArguments()

Allow direct access to the runtime arguments of this partitioned file set.

Returns:

the runtime arguments specified for this dataset.