WeightedPageRankFromMatrix
Related Doc:
package examples
Load or generate on first iteration the matrix M^ given A.
combine the config, flowDef and the Mode to produce a flow
combine the config, flowDef and the Mode to produce a flow
Copy this job By default, this uses reflection and the single argument Args constructor
Copy this job By default, this uses reflection and the single argument Args constructor
This is the exact config that is passed to the Cascading FlowConnector.
This is the exact config that is passed to the Cascading FlowConnector. By default: if there are no spill thresholds in mode.config, we replace with defaultSpillThreshold we overwrite io.serializations with ioSerializations we overwrite cascading.tuple.element.comparator.default to defaultComparator we add some scalding keys for debugging/logging
Tip: override this method, call super, and ++ your additional map to add or overwrite more options
This returns Map[AnyRef, AnyRef] for compatibility with older code
Override this to control how dates are parsed
Override this to control how dates are parsed
Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences.
Override this if you want to customize comparisons/hashing for your job the config method overwrites using this before sending to cascading The one we use by default is needed used to make Joins in the Fields-API more robust to Long vs Int differences. If you only use the Typed-API, consider changing this to return None
Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output.
Rather than give the full power of cascading's selectors, we have a simpler set of rules encoded below: 1) if the input is non-definite (ALL, GROUP, ARGS, etc...) ALL is the output. Perhaps only fromFields=ALL will make sense 2) If one of from or to is a strict super set of the other, SWAP is used. 3) If they are equal, REPLACE is used. 4) Otherwise, ALL is used.
Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.
Keep 100k tuples in memory by default before spilling Turn this up as high as you can without getting OOM.
This is ignored if there is a value set in the incoming jobConf on Hadoop
We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.
We can't set the field Manifests because cascading doesn't (yet) expose field type information in the Fields API.
Multi-entry fields.
Multi-entry fields. This are higher priority than Product conversions so that List will not conflict with Product.
These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations
These are user-defined serializations IN-ADDITION to (but deduped) with the required serializations
Use this if a map or reduce phase takes a while before emitting tuples.
Use this if a map or reduce phase takes a while before emitting tuples.
Measure convergence by calculating the total of the absolute difference between the previous and next vectors.
Measure convergence by calculating the total of the absolute difference between the previous and next vectors. This stores the result after calculation.
Recurse and iterate again iff we are under the max number of iterations and vector has not converged.
Recurse and iterate again iff we are under the max number of iterations and vector has not converged.
Useful to convert f : Any* to Fields.
Useful to convert f : Any* to Fields. This handles mixed cases ("hey", 'you). Not sure we should be this flexible, but given that Cascading will throw an exception before scheduling the job, I guess this is okay.
you should never call this directly, it is here to make the DSL work.
you should never call this directly, it is here to make the DSL work. Just know, you can treat a Pipe as a RichPipe within a Job
Load or generate on first iteration the prior vector given d and n.
Handles treating any TupleN as a Fields object.
Handles treating any TupleN as a Fields object. This is low priority because List is also a Product, but this method will not work for List (because List is Product2(head, tail) and so productIterator won't work as expected. Lists are handled by an implicit in FieldConversions, which have higher priority.
This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.
This is implicit so that a Source can be used as the argument to a join or other method that accepts Pipe.
This is here so that Mappable.toIterator can find an implicit config
This is here so that Mappable.toIterator can find an implicit config
This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...
This implicit is to enable RichPipe methods directly on Source objects, such as map/flatMap, etc...
Note that Mappable is a subclass of Source, and Mappable already has mapTo and flatMapTo BUT WITHOUT incoming fields used (see the Mappable trait). This creates some confusion when using these methods (this is an unfortunate mistake in our design that was not noticed until later). To remove ambiguity, explicitly call .read on any Source that you begin operating with a mapTo/flatMapTo.
Specify a callback to run before the start of each flow step.
Specify a callback to run before the start of each flow step.
Defaults to what Config.getReducerEstimator specifies.
ExecutionContext.buildFlow
'* means Fields.ALL, otherwise we take the .name
'* means Fields.ALL, otherwise we take the .name
This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient
This is only here for Java jobs which cannot automatically access the implicit Pipe => RichPipe which makes: pipe.write( ) convenient
A weighted PageRank implementation using the Scalding Matrix API. This assumes that all rows and columns are of type
Intand values or egde weights areDouble. If you want an unweighted PageRank, simply set the weights on the edges to 1.Input arguments:
d -- damping factor n -- number of nodes in the graph currentIteration -- start with 0 probably maxIterations -- stop after n iterations convergenceThreshold -- using the sum of the absolute difference between iteration solutions, iterating stops once we reach this threshold rootDir -- the root directory holding all starting, intermediate and final data/output
The expected structure of the rootDir is:
rootDir |- iterations | |- 0 <-- a TSV of (row, value) of size n, value can be 1/n (generate this) | |- n <-- holds future iterations/solutions |- edges <-- a TSV of (row, column, value) for edges in the graph |- onesVector <-- a TSV of (row, 1) of size n (generate this) |- diff <-- a single line representing the difference between the last iterations |- constants <-- built at iteration 0, these are constant for any given matrix/graph |- M_hat |- priorVector
Don't forget to set the number of reducers for this job: -D mapred.reduce.tasks=n