Base path of the partitioned directory
Template for the partitioned path
Text encoding of the file content
Transform this Mappable into another by mapping after.
Transform this Mappable into another by mapping after. We don't call this map because of conflicts with Mappable, unfortunately
Transform this sink into another type by applying a function first
Transform this sink into another type by applying a function first
Combine both the partition and value converter to extract the data from a flat cascading tuple
into a pair of P
and (offset, line)
.
Combine both the partition and value converter to extract the data from a flat cascading tuple
into a pair of P
and (offset, line)
.
Creates the taps for local and hdfs mode.
Creates the taps for local and hdfs mode.
Text encoding of the file content
If you want to filter, you should use this and output a 0 or 1 length Iterable.
If you want to filter, you should use this and output a 0 or 1 length Iterable. Filter does not change column names, and we generally expect to change columns here
The scheme to use if the source is on hdfs.
The scheme to use if the source is on hdfs.
The scheme to use if the source is local.
The scheme to use if the source is local.
Base path of the partitioned directory
Flatten a pair of P
and line
into a cascading tuple.
Flatten a pair of P
and line
into a cascading tuple.
This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other
This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other
Template for the partitioned path
Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS.
Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS. Typical use might be to read in Job.next to determine if another job is needed
The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source.
The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.
write the pipe but return the input so it can be chained into the next operation
write the pipe but return the input so it can be chained into the next operation
(Since version 0.9.0) replace with Mappable.toIterator
Scalding source to read or write partitioned text.
For writing it expects a pair of
(P, String)
, whereP
is the data used for partitioning andString
is the output to write out. Below is an example.For reading it produces a pair
(P, (Long, String))
whereP
is the partition data,Long
is the offset into the file andString
is a line from the file. Below is an example.Base path of the partitioned directory
Template for the partitioned path
Text encoding of the file content