SphereRDD
Related Docs:
object SphereRDD
| package spatial3DRDD
Construct a RDD[ShellEnvelope] from FITS data.
Construct a RDD[ShellEnvelope] from FITS data.
val fn = "src/test/resources/cartesian_spheres.fits" val sphereRDD = new SphereRDD(spark, fn, 1, "x,y,z,radius", false)
: (SparkSession) The spark session
: (String) File name where the data is stored
: (Int) HDU to load.
: (String) Comma-separated names of (x, y, z, r) columns to read. Example: "Z_COSMO,RA,Dec,Radius".
: (Boolean) If true, it assumes that the coordinates of the center of the ShellEnvelope are (r, theta, phi). Otherwise, it assumes cartesian coordinates (x, y, z). Default is false.
(RDD[ShellEnvelope)
Construct a RDD[ShellEnvelope] from CSV, JSON or TXT data.
Construct a RDD[ShellEnvelope] from CSV, JSON or TXT data.
// CSV val fn = "src/test/resources/cartesian_spheres.csv" val rdd = new SphereRDD(spark, fn, "x,y,z,radius", false) // JSON val fn = "src/test/resources/cartesian_spheres.json" val rdd = new SphereRDD(spark, fn, "x,y,z,radius", false) // TXT val fn = "src/test/resources/cartesian_spheres.txt" val rdd = new SphereRDD(spark, fn, "x,y,z,radius", false)
: (SparkSession) The spark session
: (String) File name where the data is stored. Extension must be explicitly written (.cvs, .json, or .txt)
: (String) Comma-separated names of (x, y, z, r) columns to read. Example: "Z_COSMO,RA,Dec,Radius".
: (Boolean) If true, it assumes that the coordinates of the center of the ShellEnvelope are (r, theta, phi). Otherwise, it assumes cartesian coordinates (x, y, z). Default is false.
(RDD[ShellEnvelope])
Repartion a RDD[T] according to a custom partitioner.
Repartion a RDD[T] according to a custom partitioner.
: (SpatialPartitioner) Instance of SpatialPartitioner or any extension of it.
(RDD[T]) Repartitioned RDD[T].
RDD containing the initial data formated as T.
RDD containing the initial data formated as T.
Apply a spatial partitioning to this.rawRDD, and return a RDD[T] with the new partitioning.
Apply a spatial partitioning to this.rawRDD, and return a RDD[T] with the new partitioning. The list of available partitioning can be found in utils/GridType. By default, the outgoing level of parallelism is the same as the incoming one (i.e. same number of partitions).
: (GridType) Type of partitioning to apply. See utils/GridType.
: (Int) Number of partitions for the partitioned RDD. By default (-1), the number of partitions is that of the raw RDD. You can force it to be different by setting manually this parameter. Be aware of shuffling though...
(RDD[T]) RDD whose elements are T (Point3D, Sphere, etc...)
Apply any Spatial Partitioner to this.rawRDD[T], and return a RDD[T] with the new partitioning.
Apply any Spatial Partitioner to this.rawRDD[T], and return a RDD[T] with the new partitioning.
: (SpatialPartitioner) Spatial partitioner as defined in utils.GridType
(RDD[T]) RDD whose elements are T (Point3D, Sphere, etc...)