com.github.arcizon.spark.filetransfer.util
FileTransferOptions
Companion object FileTransferOptions
class FileTransferOptions extends Serializable with Logging
- Since
0.1.0
- Alphabetic
- By Inheritance
- FileTransferOptions
- Logging
- Serializable
- Serializable
- AnyRef
- Any
- by any2stringadd
- by StringFormat
- by Ensuring
- by ArrowAssoc
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
FileTransferOptions(parameters: Map[String, String])
- parameters
Options specified on the DataFrame API Reader/Writer.
- Since
0.1.0
-
new
FileTransferOptions(parameters: CaseInsensitiveMap[String])
Create a new
FileTransferOptions
by specifying theirparameters
.Create a new
FileTransferOptions
by specifying theirparameters
.- parameters
Options specified on the DataFrame API Reader/Writer.
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
def
+(other: String): String
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to any2stringadd[FileTransferOptions] performed by method any2stringadd in scala.Predef.
- Definition Classes
- any2stringadd
-
def
->[B](y: B): (FileTransferOptions, B)
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to ArrowAssoc[FileTransferOptions] performed by method ArrowAssoc in scala.Predef.
- Definition Classes
- ArrowAssoc
- Annotations
- @inline()
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
dfOptions: Map[String, String]
All other options except the extracted options from the above to be passed along to the DataFrame API for read/write.
All other options except the extracted options from the above to be passed along to the DataFrame API for read/write. For example, csv options like delimiter for the
csv
file format etc.- Since
0.1.0
-
val
dfsTempPath: String
Value set with the option key dfsTempPath.
Value set with the option key dfsTempPath. Default value set to option value from
localTempPath
.- Since
0.1.0
- Note
If dfsTempPath is not provided then the downloaded file from remote host would be on the local path which is not splittable and therefore, parallelism might not be achieved during the file scan to load data.
-
def
ensuring(cond: (FileTransferOptions) ⇒ Boolean, msg: ⇒ Any): FileTransferOptions
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to Ensuring[FileTransferOptions] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: (FileTransferOptions) ⇒ Boolean): FileTransferOptions
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to Ensuring[FileTransferOptions] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: Boolean, msg: ⇒ Any): FileTransferOptions
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to Ensuring[FileTransferOptions] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: Boolean): FileTransferOptions
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to Ensuring[FileTransferOptions] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
fileFormat: FileFormat.Value
Value set with the option key fileFormat.
Value set with the option key fileFormat.
- Since
0.1.0
- Exceptions thrown
java.lang.RuntimeException
Missing 'fileFormat' option.
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
formatted(fmtstr: String): String
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to StringFormat[FileTransferOptions] performed by method StringFormat in scala.Predef.
- Definition Classes
- StringFormat
- Annotations
- @inline()
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
val
host: String
Value set with the option key host.
Value set with the option key host. Default value set to localhost.
- Since
0.1.0
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
keyFilePath: Option[String]
Value set with the option key keyFilePath.
Value set with the option key keyFilePath.
- Since
0.1.0
-
val
localTempPath: String
Value set with the option key localTempPath.
Value set with the option key localTempPath. Default value set to java system property of
java.io.tmpdir
with a fallback value of/tmp
.- Since
0.1.0
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
val
passphrase: Option[String]
Value set with the option key passphrase.
Value set with the option key passphrase.
- Since
0.1.0
-
val
password: Option[String]
Value set with the option key password.
Value set with the option key password.
- Since
0.1.0
-
val
path: String
Value set with the option key path.
Value set with the option key path.
- Since
0.1.0
- Exceptions thrown
java.lang.RuntimeException
Missing 'path' option.
-
val
port: Int
Value set with the option key port.
Value set with the option key port. Default value set to 22.
- Since
0.1.0
-
val
protocol: Protocol.Value
Value set with the option key protocol.
Value set with the option key protocol. Default value set to sftp.
- Since
0.1.0
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
val
uploadFilePrefix: String
Value set with the option key uploadFilePrefix.
Value set with the option key uploadFilePrefix. Default value set to
part
.- Since
0.1.0
-
val
username: String
Value set with the option key username.
Value set with the option key username.
- Since
0.1.0
- Exceptions thrown
java.lang.RuntimeException
Missing 'username' option.
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
→[B](y: B): (FileTransferOptions, B)
- Implicit
- This member is added by an implicit conversion from FileTransferOptions to ArrowAssoc[FileTransferOptions] performed by method ArrowAssoc in scala.Predef.
- Definition Classes
- ArrowAssoc
Documentation for Spark File Transfer
Notable classes include: