Point3DRDD

Instance Constructors

new Point3DRDD(spark: SparkSession, filename: String, colnames: String, isSpherical: Boolean, format: String, options: HashMap[String, String])

Constructor of Point3DRDD which is suitable for py4j.
Constructor of Point3DRDD which is suitable for py4j. It calls Point3DRDDFromV2PythonHelper instead of Point3DRDDFromV2. All args are the same but options which is a java.util.HashMap, and storageLevel which is removed and set to StorageLevel.MEMORY_ONLY (user cannot set the storage level in pyspark3d for the moment).
new Point3DRDD(spark: SparkSession, filename: String, colnames: String, isSpherical: Boolean, format: String, options: Map[String, String] = Map("" -> ""), storageLevel: StorageLevel = StorageLevel.NONE)

Construct a RDD[Point3D] from whatever data source registered in Spark.
Construct a RDD[Point3D] from whatever data source registered in Spark. For more information about available official connectors: https://spark-packages.org/?q=tags%3A%22Data%20Sources%22
We currently include: CSV, JSON, TXT, FITS, ROOT, HDF5, Avro, Parquet...
```
// Here is an example with a CSV file containing
// 3 spherical coordinates columns labeled Z_COSMO,RA,Dec.

// Filename
val fn = "path/to/file.csv"
// Spark datasource
val format = "csv"
// Options to pass to the DataFrameReader - optional
val options = Map("header" -> "true")

// Load the data as RDD[Point3D]
val rdd = new Point3DRDD(spark, fn, "Z_COSMO,RA,Dec", true, format, options)
```
spark
: (SparkSession) The spark session
filename
: (String) File name where the data is stored.
colnames
: (String) Comma-separated names of (x, y, z) columns. Example: "Z_COSMO,RA,Dec".
isSpherical
: (Boolean) If true, it assumes that the coordinates of the Point3D are (r, theta, phi). Otherwise, it assumes cartesian coordinates (x, y, z).
format
: (String) The name of the data source as registered in Spark. For example:
- text
- csv
- json
- com.astrolabsoftware.sparkfits or fits
- org.dianahep.sparkroot
- gov.llnl.spark.hdf or hdf5
options
: (Map[String, String]) Options to pass to the DataFrameReader. Default is no options.
storageLevel
: (StorageLevel) Storage level for the raw RDD (unpartitioned). Default is StorageLevel.NONE. See https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence for more information.
returns
(RDD[Point3D])
new Point3DRDD(rdd: RDD[Point3D], isSpherical: Boolean, storageLevel: StorageLevel)

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
var boundary: BoxEnvelope

Definition Classes
Shape3DRDD
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getDataEnvelope(): BoxEnvelope

Definition Classes
Shape3DRDD
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val isSpherical: Boolean

Definition Classes
Point3DRDD → Shape3DRDD
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def partition(partitioner: SpatialPartitioner)(implicit c: ClassTag[Point3D]): RDD[Point3D]

Repartion a RDD[T] according to a custom partitioner.
Repartion a RDD[T] according to a custom partitioner.
partitioner
: (SpatialPartitioner) Instance of SpatialPartitioner or any extension of it.
returns
(RDD[T]) Repartitioned RDD[T].

Definition Classes
Shape3DRDD
val rawRDD: RDD[Point3D]

RDD containing the initial data formated as T.
RDD containing the initial data formated as T.

Definition Classes
Point3DRDD → Shape3DRDD
def spatialPartitioning(gridtype: String, numPartitions: Int = 1)(implicit c: ClassTag[Point3D]): RDD[Point3D]

Apply a spatial partitioning to this.rawRDD, and return a RDD[T] with the new partitioning.
Apply a spatial partitioning to this.rawRDD, and return a RDD[T] with the new partitioning. The list of available partitioning can be found in utils/GridType. By default, the outgoing level of parallelism is the same as the incoming one (i.e. same number of partitions).
gridtype
: (String) Type of partitioning to apply. See utils/GridType.
numPartitions
: (Int) Number of partitions for the partitioned RDD. By default (-1), the number of partitions is that of the raw RDD. You can force it to be different by setting manually this parameter. Be aware of shuffling though...
returns
(RDD[T]) RDD whose elements are T (Point3D, Sphere, etc...)

Definition Classes
Shape3DRDD
def spatialPartitioning(partitioner: SpatialPartitioner)(implicit c: ClassTag[Point3D]): RDD[Point3D]

Apply any Spatial Partitioner to this.rawRDD[T], and return a RDD[T] with the new partitioning.
Apply any Spatial Partitioner to this.rawRDD[T], and return a RDD[T] with the new partitioning.
partitioner
: (SpatialPartitioner) Spatial partitioner as defined in utils.GridType
returns
(RDD[T]) RDD whose elements are T (Point3D, Sphere, etc...)

Definition Classes
Shape3DRDD
def spatialPartitioningPython(gridtype: String, numPartitions: Int = 1): RDD[Point3D]

Constructor of spatialPartitioning which is suitable for py4j.
Constructor of spatialPartitioning which is suitable for py4j. py4j does not handle generics, so we explicitly specify the types here. See discussion here: https://github.com/bartdag/py4j/issues/328
Apply a spatial partitioning to this.rawRDD, and return a RDD[Point3D] with the new partitioning. The list of available partitioning can be found in utils/GridType. By default, the outgoing level of parallelism is the same as the incoming one (i.e. same number of partitions).
gridtype
: (String) Type of partitioning to apply. See utils/GridType.
numPartitions
: (Int) Number of partitions for the partitioned RDD. By default (-1), the number of partitions is that of the raw RDD. You can force it to be different by setting manually this parameter. Be aware of shuffling though...
returns
(RDD[Point3D]) RDD whose elements are Point3D.
def spatialPartitioningPython(partitioner: SpatialPartitioner): RDD[Point3D]

Constructor of spatialPartitioning which is suitable for py4j.
Constructor of spatialPartitioning which is suitable for py4j. py4j does not handle generics, so we explicitly specify the types here. See discussion here: https://github.com/bartdag/py4j/issues/328
Apply any Spatial Partitioner to this.rawRDD[Point3D], and return a RDD[Point3D] with the new partitioning.
partitioner
: (SpatialPartitioner) Spatial partitioner as defined in utils.GridType
returns
(RDD[Point3D]) RDD whose elements are Point3D
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toCenterCoordinateRDD(rdd: RDD[Point3D]): RDD[List[Double]]

Return a RDD whose elements are the lists of center coordinates.
Return a RDD whose elements are the lists of center coordinates.
rdd
: (RDD[T]) Input RDD[T]
returns
(RDD[List[Double]]) RDD whose elements are the lists of center coordinates.

Definition Classes
Shape3DRDD
def toCenterCoordinateRDDPython(rdd: RDD[Point3D]): RDD[List[Double]]

Constructor of toCenterCoordinateRDD which is suitable for py4j, i.e.
Constructor of toCenterCoordinateRDD which is suitable for py4j, i.e. it replaces Scala Lists with Java Lists.
Return a RDD whose elements are the lists of center coordinates.
rdd
: (RDD[T]) Input RDD[T]
returns
(RDD[java.util.List[Double]]) RDD whose elements are the (Java) lists of center coordinates.

Definition Classes
Shape3DRDD
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object Point3DRDD | package spatial3DRDD

class Point3DRDD extends Shape3DRDD[Point3D]

Instance Constructors

new Point3DRDD(spark: SparkSession, filename: String, colnames: String, isSpherical: Boolean, format: String, options: HashMap[String, String])

new Point3DRDD(spark: SparkSession, filename: String, colnames: String, isSpherical: Boolean, format: String, options: Map[String, String] = Map("" -> ""), storageLevel: StorageLevel = StorageLevel.NONE)

new Point3DRDD(rdd: RDD[Point3D], isSpherical: Boolean, storageLevel: StorageLevel)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

var boundary: BoxEnvelope

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def getDataEnvelope(): BoxEnvelope

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

val isSpherical: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def partition(partitioner: SpatialPartitioner)(implicit c: ClassTag[Point3D]): RDD[Point3D]

val rawRDD: RDD[Point3D]

def spatialPartitioning(gridtype: String, numPartitions: Int = 1)(implicit c: ClassTag[Point3D]): RDD[Point3D]

def spatialPartitioning(partitioner: SpatialPartitioner)(implicit c: ClassTag[Point3D]): RDD[Point3D]

def spatialPartitioningPython(gridtype: String, numPartitions: Int = 1): RDD[Point3D]

def spatialPartitioningPython(partitioner: SpatialPartitioner): RDD[Point3D]

final def synchronized[T0](arg0: ⇒ T0): T0

def toCenterCoordinateRDD(rdd: RDD[Point3D]): RDD[List[Double]]

def toCenterCoordinateRDDPython(rdd: RDD[Point3D]): RDD[List[Double]]

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Shape3DRDD[Point3D]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped