FSUtils

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def keepNotPresent[O](fs: FileSystem, inParentFolder: Path, toTest: Seq[O])(getObjectPath: (O) ⇒ Path): Seq[O]

Check is objects in the toTest collection can be mapped to existing folders in the HDFS and returns objects that have not been mapped into the HDFS folder yet.
Check is objects in the toTest collection can be mapped to existing folders in the HDFS and returns objects that have not been mapped into the HDFS folder yet.
Implementation is quite efficient as it uses HDFS PathFilter and does not use globs or full lists that could be quite big.
For example: Inputs: 1) HDFS folder inParentFolder contains partition folders, one per day from 2017/01/01 to 2017/03/15 2) toTest is a suggested range of dates from 2017/03/10 to 2017/03/20
Output: 1) list of dates from 2017/03/16 to 2017/03/20
inParentFolder
- HDFS folder that contains folders that could be mapped to tested objects
getObjectPath
- maps hdfs path to tested object
returns
objects from toTest that could not be mapped into the folder inParentFolder via function getObjectPath
def listPartitions(fs: FileSystem, folder: String): Seq[(String, String)]

Lists Hive partition column name and its value, by looking into the folder.
Lists Hive partition column name and its value, by looking into the folder.
returns
(PARTITON COLUMN NAME, VALUE)
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def moveAll(fs: FileSystem, subs: Seq[String], fromPath: Path, toPath: Path): Boolean

Moves all sub-folders in fromPath into toPath.
Moves all sub-folders in fromPath into toPath. If a folder exists in the destination, it is overwritten. It uses and efficient approach to minimise the number of call to HDFS for checks and validations which could add significant amount of time to the end to end execution.
fs
- current hadoop file system
subs
- sub folders to move, usually thsese are folders in the staging folder
fromPath
- parent folder in which sub folders are
toPath
- into which folder to move the subs folders, if any already exist, then need to be overwritten
def moveOverwriteFolder(fs: FileSystem, toMove: Path, toPath: Path): Boolean

Moves toMove into toPath.
Moves toMove into toPath. Parent folder of the toPath is created if it does not exist
fs
- FileSystem which can be HDFS or Local.
toMove
- full path to the folder to be moved.
toPath
- full path to be moved into, includes the folder name itself.
returns
true if move was successful.
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def removeFolder(fs: FileSystem, folder: String): Unit

Deletes folder with all of its content, if it does not exist than does nothing.
def removeSubFoldersPresentInList(fs: FileSystem, folder: Path, subs: Seq[String]): Boolean

Check if there are any existing folders with the same name in the path and removes them.
Check if there are any existing folders with the same name in the path and removes them. The main benefit is that it performs checks in one round-trip to HDFS which in case of day zero scenarios could take a lot of time.
folder
- parent folder in which to check for existing sub-folders
subs
- names to check, if the name is not present, than ignore it, if present, remove it
returns
- true if everything was fine
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package filesystem

object FSUtils extends Logging

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def keepNotPresent[O](fs: FileSystem, inParentFolder: Path, toTest: Seq[O])(getObjectPath: (O) ⇒ Path): Seq[O]

def listPartitions(fs: FileSystem, folder: String): Seq[(String, String)]

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def moveAll(fs: FileSystem, subs: Seq[String], fromPath: Path, toPath: Path): Boolean

def moveOverwriteFolder(fs: FileSystem, toMove: Path, toPath: Path): Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def removeFolder(fs: FileSystem, folder: String): Unit

def removeSubFoldersPresentInList(fs: FileSystem, folder: Path, subs: Seq[String]): Boolean

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped