org.apache.accumulo.core.file.map
Class MySequenceFile

java.lang.Object
  org.apache.accumulo.core.file.map.MySequenceFile

Deprecated. since 1.4, replaced by RFile

public class MySequenceFile
extends Object

MySequenceFiles are flat files consisting of binary key/value pairs.

MySequenceFile provides MySequenceFile.Writer, MySequenceFile.Reader and MySequenceFile.Sorter classes for writing, reading and sorting respectively.

There are three MySequenceFile Writers based on the MySequenceFile.CompressionType used to compress key/value pairs:

1. Writer : Uncompressed records.
2. RecordCompressWriter : Record-compressed files, only compress values.
3. BlockCompressWriter : Block-compressed files, both keys & values are collected in 'blocks' separately and compressed. The size of the 'block' is configurable.

The actual compression algorithm used to compress key and/or values can be specified by using the appropriate CompressionCodec.

The recommended way is to use the static createWriter methods provided by the MySequenceFile to chose the preferred format.

The MySequenceFile.Reader acts as the bridge and can read any of the above MySequenceFile formats.

MySequenceFile Formats

Essentially there are 3 different formats for MySequenceFiles depending on the CompressionType specified. All of them share a common header described below.

MySequenceFile Header

• version - 3 bytes of magic header SEQ, followed by 1 byte of actual version number (e.g. SEQ4 or SEQ6)
• keyClassName -key class
• valueClassName - value class
• compression - A boolean which specifies if compression is turned on for keys/values in this file.
• blockCompression - A boolean which specifies if block-compression is turned on for keys/values in this file.
• compression codec - CompressionCodec class which is used for compression of keys and/or values (if compression is enabled).
• metadata - MySequenceFile.Metadata for this file.
• sync - A sync marker to denote end of the header.

Uncompressed MySequenceFile Format

• Header
• Record
  • Record length
  • Key length
  • Key
  • Value
• A sync-marker every few 100 bytes or so.

Record-Compressed MySequenceFile Format

• Header
• Record
  • Record length
  • Key length
  • Key
  • Compressed Value
• A sync-marker every few 100 bytes or so.

Block-Compressed MySequenceFile Format

• Header
• Record Block
  • Compressed key-lengths block-size
  • Compressed key-lengths block
  • Compressed keys block-size
  • Compressed keys block
  • Compressed value-lengths block-size
  • Compressed value-lengths block
  • Compressed values block-size
  • Compressed values block
• A sync-marker every few 100 bytes or so.

The compressed blocks of key lengths and value lengths consist of the actual lengths of individual keys/values encoded in ZeroCompressedInteger format.

See Also:

CompressionCodec

Nested Class Summary

static class	MySequenceFile.CompressionType Deprecated. The compression type used to compress key/value pairs in the MySequenceFile.
static class	MySequenceFile.Metadata Deprecated. The class encapsulating with the metadata of a file.
static class	MySequenceFile.Reader Deprecated. Reads key/value pairs from a sequence-format file.
static class	MySequenceFile.Sorter Deprecated. Sorts key/value pairs in a sequence-format file.
static interface	MySequenceFile.ValueBytes Deprecated. The interface to 'raw' values of SequenceFiles.
static class	MySequenceFile.Writer Deprecated. Write key/value pairs to a sequence-format file.

Field Summary

static int

SYNC_INTERVAL Deprecated. The number of bytes between sync points.

Method Summary

static MySequenceFile.Writer	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FSDataOutputStream out, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec) Deprecated. Construct the preferred type of 'raw' MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FSDataOutputStream out, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec, MySequenceFile.Metadata metadata) Deprecated. Construct the preferred type of 'raw' MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, int bufferSize, short replication, long blockSize, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.hadoop.util.Progressable progress, MySequenceFile.Metadata metadata) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.hadoop.util.Progressable progress) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.hadoop.util.Progressable progress, MySequenceFile.Metadata metadata) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.Writer	createWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, Class keyClass, Class valClass, MySequenceFile.CompressionType compressionType, org.apache.hadoop.util.Progressable progress) Deprecated. Construct the preferred type of MySequenceFile Writer.
static MySequenceFile.CompressionType	getCompressionType(org.apache.hadoop.conf.Configuration job) Deprecated. Use org.apache.hadoop.mapred.SequenceFileOutputFormat#getOutputCompressionType(org.apache.hadoop.mapred.JobConf) to get MySequenceFile.CompressionType for job-outputs.
static void	setCompressionType(org.apache.hadoop.conf.Configuration job, MySequenceFile.CompressionType val) Deprecated. Use the one of the many MySequenceFile.createWriter methods to specify the MySequenceFile.CompressionType while creating the MySequenceFile to specify the MySequenceFile.CompressionType for job-outputs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SYNC_INTERVAL

public static final int SYNC_INTERVAL

Deprecated.

The number of bytes between sync points.

See Also:

Constant Field Values

Method Detail

getCompressionType

@Deprecated
public static MySequenceFile.CompressionType getCompressionType(org.apache.hadoop.conf.Configuration job)

Deprecated. Use org.apache.hadoop.mapred.SequenceFileOutputFormat#getOutputCompressionType(org.apache.hadoop.mapred.JobConf) to get MySequenceFile.CompressionType for job-outputs.

Get the compression type for the reduce outputs

Parameters:

job - the job config to look in

Returns:

the kind of compression to use

setCompressionType

@Deprecated
public static void setCompressionType(org.apache.hadoop.conf.Configuration job,
                                                 MySequenceFile.CompressionType val)

Deprecated. Use the one of the many MySequenceFile.createWriter methods to specify the MySequenceFile.CompressionType while creating the MySequenceFile to specify the MySequenceFile.CompressionType for job-outputs.

Set the compression type for sequence files.

Parameters:

job - the configuration to modify

val - the new compression type (none, block, record)

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.util.Progressable progress)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

progress - The Progressable object to track progress.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec,
                                                 org.apache.hadoop.util.Progressable progress,
                                                 MySequenceFile.Metadata metadata)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

progress - The Progressable object to track progress.

metadata - The metadata of the file.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 int bufferSize,
                                                 short replication,
                                                 long blockSize,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec,
                                                 org.apache.hadoop.util.Progressable progress,
                                                 MySequenceFile.Metadata metadata)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

bufferSize - buffer size for the underlaying outputstream.

replication - replication factor for the file.

blockSize - block size for the file.

compressionType - The compression type.

codec - The compression codec.

progress - The Progressable object to track progress.

metadata - The metadata of the file.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.fs.FileSystem fs,
                                                 org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path name,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec,
                                                 org.apache.hadoop.util.Progressable progress)
                                          throws IOException

Deprecated.

Construct the preferred type of MySequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

progress - The Progressable object to track progress.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.FSDataOutputStream out,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec,
                                                 MySequenceFile.Metadata metadata)
                                          throws IOException

Deprecated.

Construct the preferred type of 'raw' MySequenceFile Writer.

Parameters:

conf - The configuration.

out - The stream on top which the writer is to be constructed.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

metadata - The metadata of the file.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException

createWriter

public static MySequenceFile.Writer createWriter(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.FSDataOutputStream out,
                                                 Class keyClass,
                                                 Class valClass,
                                                 MySequenceFile.CompressionType compressionType,
                                                 org.apache.hadoop.io.compress.CompressionCodec codec)
                                          throws IOException

Deprecated.

Construct the preferred type of 'raw' MySequenceFile Writer.

Parameters:

conf - The configuration.

out - The stream on top which the writer is to be constructed.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

Returns:

Returns the handle to the constructed MySequenceFile Writer.

Throws:

IOException