@Beta public interface TimePartitionedFileSet extends PartitionedFileSet
PartitionedFileSet
with five integer partitioning fields: the year, month, day, hour and minute. Partitions can
be retrieved using time range or using a PartitionFilter
.
The granularity of time is in minutes, that is, any seconds or milliseconds after the
full minute is ignored for the partition keys. That means, there can not be be two partitions
in the same minute. Also, when retrieving partitions via time or time range using
getPartitionByTime(long)
, getPartitionsByTime(long, long)
, or when writing a partition using
getPartitionOutput(long)
, the seconds and milliseconds on the
time or time range are ignored.
This dataset can be made available for querying with SQL (explore). This is enabled through dataset
properties when the dataset is created. See FileSetProperties
for details. If it is enabled for explore, a Hive external table will be created when the dataset is
created. The Hive table is partitioned by year, month, day, hour and minute.
Modifier and Type | Method and Description |
---|---|
void |
addMetadata(long time,
Map<String,String> metadata)
Adds a set of new metadata entries for a particular partition
Note that existing entries can not be updated.
|
void |
addMetadata(long time,
String metadataKey,
String metadataValue)
Adds a new metadata entry for a particular partition.
|
void |
addPartition(long time,
String path)
Add a partition for a given time, stored at a given path (relative to the file set's base path).
|
void |
addPartition(long time,
String path,
Map<String,String> metadata)
Add a partition for a given time, stored at a given path (relative to the file set's base path),
with given metadata.
|
void |
dropPartition(long time)
Remove a partition for a given time.
|
TimePartitionDetail |
getPartitionByTime(long time)
Return the partition associated with the given time, rounded to the minute;
or null if no such partition exists.
|
TimePartitionOutput |
getPartitionOutput(long time)
Return a partition output for a specific time, rounded to the minute, in preparation for creating a new partition.
|
Set<TimePartitionDetail> |
getPartitionsByTime(long startTime,
long endTime)
Return all partitions within the time range given by startTime (inclusive) and endTime (exclusive),
both rounded to the full minute.
|
void |
removeMetadata(long time,
Set<String> metadataKeys)
Removes a set of metadata entries for a particular time.
|
void |
removeMetadata(long time,
String metadataKey)
Removes a metadata entry for a particular time.
|
void |
setMetadata(long time,
Map<String,String> metadata)
Sets metadata entries for a particular partition.
|
addMetadata, addMetadata, addPartition, addPartition, concatenatePartition, consumePartitions, consumePartitions, dropPartition, getEmbeddedFileSet, getPartition, getPartitioning, getPartitionOutput, getPartitions, getRuntimeArguments, removeMetadata, removeMetadata, setMetadata
getInputFormatClassName, getInputFormatConfiguration
getOutputFormatClassName, getOutputFormatConfiguration
static final String TYPE
void addPartition(long time, String path)
time
- the partition time in milliseconds since the EpochPartitionAlreadyExistsException
- if the partition for the given time already existsvoid addPartition(long time, String path, Map<String,String> metadata)
time
- the partition time in milliseconds since the EpochPartitionAlreadyExistsException
- if the partition for the given time already existsvoid addMetadata(long time, String metadataKey, String metadataValue)
time
- the partition time in milliseconds since the EpochDataSetException
- in case an attempt is made to update existing entries.void addMetadata(long time, Map<String,String> metadata)
time
- the partition time in milliseconds since the EpochDataSetException
- in case an attempt is made to update existing entries.void setMetadata(long time, Map<String,String> metadata)
PartitionNotFoundException
- when a partition for the given key is not foundIllegalArgumentException
- if the partition key does not match the partitioning of the datasetvoid removeMetadata(long time, String metadataKey)
PartitionNotFoundException
- when a partition for the given time is not foundvoid removeMetadata(long time, Set<String> metadataKeys)
PartitionNotFoundException
- when a partition for the given time is not foundvoid dropPartition(long time)
time
- the partition time in milliseconds since the Epoch@Nullable TimePartitionDetail getPartitionByTime(long time)
time
- the partition time in milliseconds since the EpochSet<TimePartitionDetail> getPartitionsByTime(long startTime, long endTime)
startTime
- the inclusive lower bound for the partition time in milliseconds since the EpochendTime
- the exclusive upper bound for the partition time in milliseconds since the EpochTimePartitionOutput getPartitionOutput(long time)
PartitionOutput.addPartition()
to add the partition to this dataset.time
- the partition time in milliseconds since the EpochPartitionAlreadyExistsException
- if the partition for the given time already existsCopyright © 2022 Cask Data, Inc. Licensed under the Apache License, Version 2.0.