Figure out the total size of the input to the current step and set the number of reducers using the "bytesPerReducer" configuration parameter.
Figure out the total size of the input to the current step and set the number of reducers using the "bytesPerReducer" configuration parameter.
Holds information about the overall flow (.flow), previously-run steps (.predecessorSteps), and the current step (.step).
Number of reducers recommended by the estimator, or None to keep the default.
Get the total size of the file(s) specified by the Hfs, which may contain a glob pattern in its path, so we must be ready to handle that case.
Get the total size of the file(s) specified by the Hfs, which may contain a glob pattern in its path, so we must be ready to handle that case.
Estimator that uses the input size and a fixed "bytesPerReducer" target.
Bytes per reducer can be configured with configuration parameter, defaults to 1 GB.