com.twitter.scalding.source

DailySuffixTypedTsv

class DailySuffixTypedTsv[T] extends DailySuffixSource with TypedDelimited[T]

Linear Supertypes
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. DailySuffixTypedTsv
  2. TypedDelimited
  3. TypedSink
  4. Mappable
  5. TypedSource
  6. DelimitedScheme
  7. DailySuffixSource
  8. TimePathedSource
  9. TimeSeqPathedSource
  10. FileSource
  11. LocalSourceOverride
  12. SchemedSource
  13. Source
  14. Serializable
  15. AnyRef
  16. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DailySuffixTypedTsv(prefix: String)(implicit dateRange: DateRange, mf: Manifest[T], conv: TupleConverter[T], tset: TupleSetter[T])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def addTypes(sel: Array[Comparable[_]]): Fields

    Definition Classes
    TypedDelimited
  7. def allPaths: Iterable[String]

    These are all the paths we will read for this data completely enumerated

    These are all the paths we will read for this data completely enumerated

    Definition Classes
    TimeSeqPathedSource
  8. def allPathsFor(pattern: String): Iterable[String]

    Attributes
    protected
    Definition Classes
    TimeSeqPathedSource
  9. def andThen[U](fn: (T) ⇒ U): typed.TypedSource[U]

    Transform this TypedSource into another by mapping after.

    Transform this TypedSource into another by mapping after. We don't call this map because of conflicts with Mappable, unfortunately

    Definition Classes
    TypedSource
  10. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  11. def checkFlowDefNotNull(implicit flowDef: FlowDef, mode: Mode): Unit

    Attributes
    protected
    Definition Classes
    Source
  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def contraMap[U](fn: (U) ⇒ T): typed.TypedSink[U]

    Transform this sink into another type by applying a function first

    Transform this sink into another type by applying a function first

    Definition Classes
    TypedSink
  14. implicit val conv: TupleConverter[T]

    Definition Classes
    DailySuffixTypedTsvTypedDelimited
  15. def converter[U >: T]: TupleConverter[U]

    Because TupleConverter cannot be covariant, we need to jump through this hoop.

    Because TupleConverter cannot be covariant, we need to jump through this hoop. A typical implementation might be: (implicit conv: TupleConverter[T]) and then:

    override def converter[U >: T] = TupleConverter.asSuperConverter[T, U](conv)

    Definition Classes
    TypedDelimitedTypedSource
  16. def createHdfsReadTap(hdfsMode: Hdfs): Tap[JobConf, _, _]

    Attributes
    protected
    Definition Classes
    FileSource
  17. def createLocalTap(sinkMode: SinkMode): Tap[_, _, _]

    Creates a local tap.

    Creates a local tap.

    sinkMode

    The mode for handling output conflicts.

    Definition Classes
    LocalSourceOverride
  18. def createTap(readOrWrite: AccessMode)(implicit mode: Mode): Tap[_, _, _]

    Subclasses of Source MUST override this method.

    Subclasses of Source MUST override this method. They may call out to TestTapFactory for making Taps suitable for testing.

    Definition Classes
    FileSourceSource
  19. implicit val dateRange: DateRange

  20. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  21. def equals(that: Any): Boolean

    Definition Classes
    TimeSeqPathedSource → AnyRef → Any
  22. val fields: Fields

    Definition Classes
    TypedDelimitedDelimitedScheme
  23. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  24. final def flatMapTo[U](out: Fields)(mf: (T) ⇒ TraversableOnce[U])(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

    If you want to filter, you should use this and output a 0 or 1 length Iterable.

    If you want to filter, you should use this and output a 0 or 1 length Iterable. Filter does not change column names, and we generally expect to change columns here

    Definition Classes
    Mappable
  25. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  26. def getPathStatuses(conf: Configuration): Iterable[(String, Boolean)]

    Get path statuses based on daterange.

    Get path statuses based on daterange. This tests each path with pathIsGood (which by default checks that there is at least on file in that directory)

    Definition Classes
    TimeSeqPathedSource
  27. def goodHdfsPaths(hdfsMode: Hdfs): Iterable[String]

    Attributes
    protected
    Definition Classes
    FileSource
  28. def hashCode(): Int

    Definition Classes
    TimeSeqPathedSource → AnyRef → Any
  29. def hdfsPaths: Iterable[String]

    Definition Classes
    TimeSeqPathedSourceFileSource
  30. def hdfsReadPathsAreGood(conf: Configuration): Boolean

    Definition Classes
    TimeSeqPathedSourceFileSource
  31. def hdfsScheme: Scheme[JobConf, RecordReader[_, _], OutputCollector[_, _], _, _]

    The scheme to use if the source is on hdfs.

    The scheme to use if the source is on hdfs.

    Definition Classes
    DelimitedSchemeSchemedSource
  32. def hdfsWritePath: String

    Definition Classes
    TimePathedSourceFileSource
  33. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  34. def localPath: String

    A path to use for the local tap.

    A path to use for the local tap.

    Definition Classes
    TimePathedSourceLocalSourceOverride
  35. def localScheme: TextDelimited

    The scheme to use if the source is local.

    The scheme to use if the source is local.

    Definition Classes
    DelimitedSchemeSchemedSource
  36. final def mapTo[U](out: Fields)(mf: (T) ⇒ U)(implicit flowDef: FlowDef, mode: Mode, setter: TupleSetter[U]): Pipe

    Definition Classes
    Mappable
  37. implicit val mf: Manifest[T]

    Definition Classes
    DailySuffixTypedTsvTypedDelimited
  38. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  39. final def notify(): Unit

    Definition Classes
    AnyRef
  40. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  41. def pathIsGood(p: String, conf: Configuration): Boolean

    Determines if a path is 'valid' for this source.

    Determines if a path is 'valid' for this source. In strict mode all paths must be valid. In non-strict mode, all invalid paths will be filtered out.

    Subclasses can override this to validate paths.

    The default implementation is a quick sanity check to look for missing or empty directories. It is necessary but not sufficient -- there are cases where this will return true but there is in fact missing data.

    TODO: consider writing a more in-depth version of this method in TimePathedSource that looks for TODO: missing days / hours etc.

    Attributes
    protected
    Definition Classes
    FileSource
  42. val pattern: String

    Definition Classes
    TimePathedSource
  43. val patterns: Seq[String]

    Definition Classes
    TimeSeqPathedSource
  44. val quote: String

    Definition Classes
    DelimitedScheme
  45. def read(implicit flowDef: FlowDef, mode: Mode): Pipe

    Definition Classes
    Source
  46. val safe: Boolean

    Definition Classes
    DelimitedScheme
  47. val separator: String

    Definition Classes
    TypedDelimitedDelimitedScheme
  48. def setter[U <: T]: TupleSetter[U]

    Definition Classes
    TypedDelimitedTypedSink
  49. final def sinkFields: Fields

    Definition Classes
    TypedDelimitedTypedSink
  50. val sinkMode: SinkMode

    Definition Classes
    SchemedSource
  51. val skipHeader: Boolean

    Definition Classes
    TypedDelimitedDelimitedScheme
  52. def sourceFields: Fields

    Definition Classes
    TypedSource
  53. def sourceId: String

    This is a name the refers to this exact instance of the source (put another way, if s1.

    This is a name the refers to this exact instance of the source (put another way, if s1.sourceId == s2.sourceId, the job should work the same if one is replaced with the other

    Definition Classes
    Source
  54. val strict: Boolean

    Definition Classes
    DelimitedScheme
  55. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  56. def toIterator(implicit config: Config, mode: Mode): Iterator[T]

    Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS.

    Allows you to read a Tap on the submit node NOT FOR USE IN THE MAPPERS OR REDUCERS. Typical use might be to read in Job.next to determine if another job is needed

    Definition Classes
    Mappable
  57. def toString(): String

    Definition Classes
    TimeSeqPathedSource → AnyRef → Any
  58. def transformForRead(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source
  59. def transformForWrite(pipe: Pipe): Pipe

    Attributes
    protected
    Definition Classes
    Source
  60. def transformInTest: Boolean

    The mock passed in to scalding.

    The mock passed in to scalding.JobTest may be considered as a mock of the Tap or the Source. By default, as of 0.9.0, it is considered as a Mock of the Source. If you set this to true, the mock in TestMode will be considered to be a mock of the Tap (which must be transformed) and not the Source.

    Definition Classes
    Source
  61. implicit val tset: TupleSetter[T]

    Definition Classes
    DailySuffixTypedTsvTypedDelimited
  62. val types: Array[Class[_]]

    Definition Classes
    TypedDelimitedDelimitedScheme
  63. val tz: TimeZone

    Definition Classes
    TimeSeqPathedSource
  64. def validateTaps(mode: Mode): Unit

    Definition Classes
    FileSourceSource
  65. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  66. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  67. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  68. def writeFrom(pipe: Pipe)(implicit flowDef: FlowDef, mode: Mode): Pipe

    write the pipe but return the input so it can be chained into the next operation

    write the pipe but return the input so it can be chained into the next operation

    Definition Classes
    Source
  69. val writeHeader: Boolean

    Definition Classes
    TypedDelimitedDelimitedScheme

Deprecated Value Members

  1. def readAtSubmitter[T](implicit mode: Mode, conv: TupleConverter[T]): Stream[T]

    Definition Classes
    Source
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) replace with Mappable.toIterator

Inherited from TypedDelimited[T]

Inherited from typed.TypedSink[T]

Inherited from Mappable[T]

Inherited from typed.TypedSource[T]

Inherited from DelimitedScheme

Inherited from DailySuffixSource

Inherited from TimePathedSource

Inherited from TimeSeqPathedSource

Inherited from FileSource

Inherited from LocalSourceOverride

Inherited from SchemedSource

Inherited from Source

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped