Callback to spark Utils to fetch file Download a file or directory to target directory.
Callback to spark Utils to fetch file Download a file or directory to target directory. Supports fetching the file in a variety of ways, including HTTP, Hadoop-compatible filesystems, and files on a standard filesystem, based on the URL parameter. Fetching directories is only supported from Hadoop-compatible filesystems.
If useCache
is true, first attempts to fetch the file to a local cache that's shared
across executors running the same application. useCache
is used mainly for
the executors, and not in local mode.
Throws SparkException if the target file already exists and has different contents than the requested file.