Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final val TimeZoneUTC: TimeZone
5.  def anyToDays(obj: Any): Int

   Converts an Java object to days.

   Converts an Java object to days.

   obj

   Either an object of java.sql.Date or java.time.LocalDate.

   returns

   The number of days since 1970-01-01.

6.  def anyToMicros(obj: Any): Long

   Converts an Java object to microseconds.

   Converts an Java object to microseconds.

   obj

   Either an object of java.sql.Timestamp or java.time.{Instant,LocalDateTime}.

   returns

   The number of micros since the epoch.

7.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

8.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

9.  def convertTz(micros: Long, fromZone: ZoneId, toZone: ZoneId): Long

   Converts the timestamp micros from one timezone to another.

   Converts the timestamp micros from one timezone to another.

   Time-zone rules, such as daylight savings, mean that not every local date-time is valid for the toZone time zone, thus the local date-time may be adjusted.

10.  def daysToLocalDate(days: Int): LocalDate

    Obtains an instance of java.time.LocalDate from the epoch day count.

11.  def daysToMicros(days: Int, zoneId: ZoneId): Long

    Converts days since 1970-01-01 at the given zone ID to microseconds since 1970-01-01 00:00:00Z.

12.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

13.  def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

14.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

15.  def fromJavaDate(date: Date): Int

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part.

    Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part. The resulted days are rebased from the hybrid to Proleptic Gregorian calendar. The days rebasing is performed via UTC time zone for simplicity because the difference between two calendars is the same in any given time zone and UTC time zone.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions. The goal of the shift is to get a local date derived from the number of days that has the same date fields (year, month, day) as the original date at the default JVM time zone.

    date

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of days since the epoch in Proleptic Gregorian calendar.

16.  def fromJavaTimestamp(t: Timestamp): Long

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z. It extracts date-time fields from the input, builds a local timestamp in Proleptic Gregorian calendar from the fields, and binds the timestamp to the system time zone. The resulted instant is converted to microseconds since the epoch.

    The conversion is performed via the system time zone because it is used internally in java.sql.Timestamp while extracting date-time fields.

    The goal of the function is to have the same local date-time in the original calendar - the hybrid calendar (Julian + Gregorian) and in the target calendar which is Proleptic Gregorian calendar, see SPARK-26651.

    t

    It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.

    returns

    The number of micros since epoch from java.sql.Timestamp.

17.  def fromJavaTimestampNoRebase(t: Timestamp): Long

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.

    Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.

    t

    an instance of java.sql.Timestamp.

    returns

    The number of micros since epoch from java.sql.Timestamp.

18.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

19.  def getLocalDateTime(micros: Long, zoneId: ZoneId): LocalDateTime

    Attributes

    protected

20.  def getTimeZone(timeZoneId: String): TimeZone
21.  def getZoneId(timeZoneId: String): ZoneId
22.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

23.  def instantToMicros(instant: Instant): Long

    Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant.

    Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant. The epoch microsecond count is a simple incrementing count of microseconds where microsecond 0 is 1970-01-01 00:00:00Z.

24.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

25.  def localDateTimeToMicros(localDateTime: LocalDateTime): Long
26.  def localDateToDays(localDate: LocalDate): Int

    Converts the local date to the number of days since 1970-01-01.

27.  def microsToDays(micros: Long, zoneId: ZoneId): Int

    Converts microseconds since 1970-01-01 00:00:00Z to days since 1970-01-01 at the given zone ID.

28.  def microsToInstant(micros: Long): Instant

    Obtains an instance of java.time.Instant using microseconds from the epoch of 1970-01-01 00:00:00Z.

29.  def microsToLocalDateTime(micros: Long): LocalDateTime
30.  def microsToMillis(micros: Long): Long

    Converts the timestamp to milliseconds since epoch.

    Converts the timestamp to milliseconds since epoch. In Spark timestamp values have microseconds precision, so this conversion is lossy.

31.  def millisToMicros(millis: Long): Long

    Converts milliseconds since the epoch to microseconds.

32.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

33.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

34.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

35.  def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId], Boolean)

    Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date.

    Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date. value. The return type is Option in order to distinguish between 0L and null. The following formats are allowed:

    [+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [+-]yyyy*-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] T[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]

    where zone_id should have one of the forms:

    • Z - Zulu time zone UTC+0
    • +|-[h]h:[m]m
    • A short id, see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html#SHORT_IDS
    • An id with one of the prefixes UTC+, UTC-, GMT+, GMT-, UT+ or UT-, and a suffix in the formats:
      • +|-h[h]
      • +|-hh[:]mm
      • +|-hh:mm:ss
      • +|-hhmmss

    • Region-based zone IDs in the form area/city, such as Europe/Paris

    returns

    timestamp segments, time zone id and whether the input is just time without a date. If the input string can't be parsed as timestamp, the result timestamp segments are empty.

36.  def stringToDate(s: UTF8String): Option[Int]

    Trims and parses a given UTF8 date string to a corresponding Int value.

    Trims and parses a given UTF8 date string to a corresponding Int value. The return type is Option in order to distinguish between 0 and null. The following formats are allowed:

    [+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d * [+-]yyyy*-[m]m-[d]dT*

37.  def stringToDateAnsi(s: UTF8String, context: SQLQueryContext = null): Int
38.  def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[Long]

    Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value.

    Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats

39.  def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId, context: SQLQueryContext = null): Long
40.  def stringToTimestampWithoutTimeZone(s: UTF8String, allowTimeZone: Boolean): Option[Long]

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch.

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch. The result will be independent of time zones.

    If the input string contains a component associated with time zone, the method will return None if allowTimeZone is set to false. If allowTimeZone is set to true, the method will simply discard the time zone component. Enable the check to detect situations like parsing a timestamp with time zone as TimestampNTZType.

    The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats.

41.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

42.  def toJavaDate(days: Int): Date

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian).

    Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian). It rebases the given days from Proleptic Gregorian to the hybrid calendar at UTC time zone for simplicity because the difference between two calendars doesn't depend on any time zone. The result is shifted by the time zone offset in wall clock to have the same date fields (year, month, day) at the default JVM time zone as the input daysSinceEpoch in Proleptic Gregorian calendar.

    Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions.

    days

    The number of days since 1970-01-01 in Proleptic Gregorian calendar.

    returns

    A local date in the hybrid calendar as java.sql.Date from number of days since epoch.

43.  def toJavaTimestamp(micros: Long): Timestamp

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.

    The conversion is based on the JVM system time zone because the java.sql.Timestamp uses the time zone internally.

    The method performs the conversion via local timestamp fields to have the same date-time representation as year, month, day, ..., seconds in the original calendar and in the target calendar.

    micros

    The number of microseconds since 1970-01-01T00:00:00.000000Z.

    returns

    A java.sql.Timestamp from number of micros since epoch.

44.  def toJavaTimestampNoRebase(micros: Long): Timestamp

    Converts microseconds since the epoch to an instance of java.sql.Timestamp.

    Converts microseconds since the epoch to an instance of java.sql.Timestamp.

    micros

    The number of microseconds since 1970-01-01T00:00:00.000000Z.

    returns

    A java.sql.Timestamp from number of micros since epoch.

45.  def toString(): String

    Definition Classes

    AnyRef → Any

46.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

47.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

48.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()