org.apache.spark.sql.execution.datasources
OutputWriterFactory
Related Doc:
package datasources
When writing to a HadoopFsRelation, this method gets called by each task on executor side to instantiate new OutputWriters.
When writing to a HadoopFsRelation, this method gets called by each task on executor side to instantiate new OutputWriters.
Path of the file to which this OutputWriter is supposed to write. Note that
this may not point to the final output file. For example, FileOutputFormat writes to
temporary directories and then merge written files back to the final destination. In
this case, path points to a temporary output file under the temporary directory.
Schema of the rows to be written. Partition columns are not included in the schema if the relation being written is partitioned.
The Hadoop MapReduce task context.
1.4.0
Returns a new instance of OutputWriter that will write data to the given path.
Returns a new instance of OutputWriter that will write data to the given path.
This method gets called by each task on executor to write InternalRows to
format-specific files. Compared to the other newInstance(), this is a newer API that
passes only the path that the writer must write to. The writer must write to the exact path
and not modify it (do not add subdirectories, extensions, etc.). All other
file-format-specific information needed to create the writer must be passed
through the OutputWriterFactory implementation.
2.0.0
::Experimental:: A factory that produces OutputWriters. A new OutputWriterFactory is created on driver side for each write job issued when writing to a HadoopFsRelation, and then gets serialized to executor side to create actual OutputWriters on the fly.
1.4.0