combine the config, flowDef and the Mode to produce a flow
combine the config, flowDef and the Mode to produce a flow
Copy this job By default, this uses reflection and the single argument Args constructor
Copy this job By default, this uses reflection and the single argument Args constructor
This is the exact config that is passed to the Cascading FlowConnector.
This is the exact config that is passed to the Cascading FlowConnector. By default: if there are no spill thresholds in mode.config, we replace with defaultSpillThreshold we overwrite io.serializations with ioSerializations we overwrite cascading.tuple.element.comparator.default to defaultComparator we add some scalding keys for debugging/logging
Tip: override this method, call super, and ++ your additional map to add or overwrite more options
This returns Map[AnyRef, AnyRef] for compatibility with older code
Override this to control how dates are parsed
Override this to control how dates are parsed
Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences.
Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences. If you only use the Typed-API, consider changing this to return None
Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc.
Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output. Perhaps only fromFields=ALL will make sense 2) If one of from or to is a strict super set of the other, SWAP is used. 3) If they are equal, REPLACE is used. 4) Otherwise, ALL is used.
Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.
Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.
This is ignored if there is a value set in the incoming jobConf on Hadoop
The basic idea is to groupBy the dst key with BOTH the nodeset and the edge rows.
The basic idea is to groupBy the dst key with BOTH the nodeset and the edge rows. the nodeset rows have the old page-rank, the edge rows are reversed, so we can get the incoming page-rank from the nodes that point to each destination.
We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.
We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.
Multi-entry fields.
Multi-entry fields. This are higher priority than Product conversions so that List will not conflict with Product.
override this function to change how you generate a pipe of (Long, String, Double) where the first entry is the nodeid, the second is the list of neighbors, as a comma (no spaces) separated string representation of the numeric nodeids, the third is the initial page rank (if not starting from a previous run, this should be 1.
override this function to change how you generate a pipe of (Long, String, Double) where the first entry is the nodeid, the second is the list of neighbors, as a comma (no spaces) separated string representation of the numeric nodeids, the third is the initial page rank (if not starting from a previous run, this should be 1.0
NOTE: if you want to run until convergence, the initialize method must read the same EXACT format as the output method writes. This is your job!
These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations
These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations
Use this if a map or reduce phase takes a while before emitting tuples.
Use this if a map or reduce phase takes a while before emitting tuples.
Here is where we check for convergence and then run the next job if we're not converged
Useful to convert f : Any* to Fields.
Useful to convert f : Any* to Fields. This handles mixed cases ("hey", 'you). Not sure we should be this flexible, but given that Cascading will throw an exception before scheduling the job, I guess this is okay.
you should never call this directly, it is here to make the DSL work.
you should never call this directly, it is here to make the DSL work. Just know, you can treat a Pipe as a RichPipe within a Job
Handles treating any TupleN as a Fields object.
Handles treating any TupleN as a Fields object. This is low priority because List is also a Product, but this method will not work for List (because List is Product2(head, tail) and so productIterator won't work as expected. Lists are handled by an implicit in FieldConversions, which have higher priority.
This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.
This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.
This is here so that Mappable.
This is here so that Mappable.toIterator can find an implicit config
This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc.
This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...
Note that Mappable is a subclass of Source, and Mappable already has mapTo and flatMapTo BUT WITHOUT incoming fields used (see the Mappable trait). This creates some confusion when using these methods (this is an unfortunate mistake in our design that was not noticed until later). To remove ambiguity, explicitly call .read on any Source that you begin operating with a mapTo/flatMapTo.
Specify a callback to run before the start of each flow step.
Specify a callback to run before the start of each flow step.
Defaults to what Config.getReducerEstimator specifies.
ExecutionContext.buildFlow
'* means Fields.
'* means Fields.ALL, otherwise we take the .name
This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.
This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient
Options: --input: the three column TSV with node, comma-sep-out-neighbors, initial pagerank (set to 1.0 first) --ouput: the name for the TSV you want to write to, same as above. optional arguments: --errorOut: name of where to write the L1 error between the input page-rank and the output if this is omitted, we don't compute the error --iterations: how many iterations to run inside this job. Default is 1, 10 is about as much as cascading can handle. --jumpprob: probability of a random jump, default is 0.15 --convergence: if this is set, after every "--iterations" steps, we check the error and see if we should continue. Since the error check is expensive (involving a join), you should avoid doing this too frequently. 10 iterations is probably a good number to set. --temp: this is the name where we will store a temporary output so we can compare to the previous for convergence checking. If convergence is set, this MUST be.