package types
Contains a type system for attributes produced by relations, including complex types like structs, arrays and maps.
- Source
- package.scala
- Alphabetic
- By Inheritance
- types
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
case class
ArrayType(elementType: DataType, containsNull: Boolean) extends DataType with Product with Serializable
The data type for collections of multiple values.
The data type for collections of multiple values. Internally these are represented as columns that contain a
.scala.collection.Seq
Please use
DataTypes.createArrayType()
to create a specific instance.An ArrayType object comprises two fields,
elementType: DataType
andcontainsNull: Boolean
. The field ofelementType
is used to specify the type of array elements. The field ofcontainsNull
is used to specify if the array hasnull
values.- elementType
The data type of values.
- containsNull
Indicates if values have
null
values
- Annotations
- @Stable()
- Since
1.3.0
-
abstract
class
AtomicType extends DataType
An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.
An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.
- Attributes
- protected[sql]
-
class
BinaryType extends AtomicType
The data type representing
Array[Byte]
values.The data type representing
Array[Byte]
values. Please use the singletonDataTypes.BinaryType
.- Annotations
- @Stable()
-
class
BooleanType extends AtomicType
The data type representing
Boolean
values.The data type representing
Boolean
values. Please use the singletonDataTypes.BooleanType
.- Annotations
- @Stable()
- Since
1.3.0
-
class
ByteType extends IntegralType
The data type representing
Byte
values.The data type representing
Byte
values. Please use the singletonDataTypes.ByteType
.- Annotations
- @Stable()
- Since
1.3.0
-
class
CalendarIntervalType extends DataType
The data type representing calendar intervals.
The data type representing calendar intervals. The calendar interval is stored internally in three components: an integer value representing the number of
months
in this interval, an integer value representing the number ofdays
in this interval, a long value representing the number ofmicroseconds
in this interval.Please use the singleton
DataTypes.CalendarIntervalType
to refer the type.- Annotations
- @Stable()
- Since
1.5.0
- Note
Calendar intervals support comparison and ordering since 3.0.0.
-
case class
CharType(length: Int) extends HiveStringType with Product with Serializable
Hive char type.
Hive char type. Similar to other HiveStringType's, these datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
-
abstract
class
DataType extends AbstractDataType
The base type of all Spark SQL data types.
The base type of all Spark SQL data types.
- Annotations
- @Stable() @JsonSerialize() @JsonDeserialize()
- Since
1.3.0
- class DataTypes extends AnyRef
-
class
DateType extends AtomicType
The date type represents a valid date in the proleptic Gregorian calendar.
The date type represents a valid date in the proleptic Gregorian calendar. Valid range is [0001-01-01, 9999-12-31].
Please use the singleton
DataTypes.DateType
to refer the type.- Annotations
- @Stable()
- Since
1.3.0
-
final
class
Decimal extends Ordered[Decimal] with Serializable
A mutable implementation of BigDecimal that can hold a Long if values are small enough.
A mutable implementation of BigDecimal that can hold a Long if values are small enough.
The semantics of the fields are as follows: - _precision and _scale represent the SQL precision and scale we are looking for - If decimalVal is set, it represents the whole decimal value - Otherwise, the decimal value is longVal / (10 ** _scale)
Note, for values between -1.0 and 1.0, precision digits are only counted after dot.
- Annotations
- @Unstable()
-
case class
DecimalType(precision: Int, scale: Int) extends FractionalType with Product with Serializable
The data type representing
java.math.BigDecimal
values.The data type representing
java.math.BigDecimal
values. A Decimal that must have fixed precision (the maximum number of digits) and scale (the number of digits on right side of dot).The precision can be up to 38, scale can also be up to 38 (less or equal to precision).
The default precision and scale is (10, 0).
Please use
DataTypes.createDecimalType()
to create a specific instance.- Annotations
- @Stable()
- Since
1.3.0
-
class
DoubleType extends FractionalType
The data type representing
Double
values.The data type representing
Double
values. Please use the singletonDataTypes.DoubleType
.- Annotations
- @Stable()
- Since
1.3.0
-
class
FloatType extends FractionalType
The data type representing
Float
values.The data type representing
Float
values. Please use the singletonDataTypes.FloatType
.- Annotations
- @Stable()
- Since
1.3.0
-
sealed abstract
class
HiveStringType extends AtomicType
A hive string type for compatibility.
A hive string type for compatibility. These datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
-
class
IntegerType extends IntegralType
The data type representing
Int
values.The data type representing
Int
values. Please use the singletonDataTypes.IntegerType
.- Annotations
- @Stable()
- Since
1.3.0
-
class
LongType extends IntegralType
The data type representing
Long
values.The data type representing
Long
values. Please use the singletonDataTypes.LongType
.- Annotations
- @Stable()
- Since
1.3.0
-
case class
MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean) extends DataType with Product with Serializable
The data type for Maps.
The data type for Maps. Keys in a map are not allowed to have
null
values.Please use
DataTypes.createMapType()
to create a specific instance.- keyType
The data type of map keys.
- valueType
The data type of map values.
- valueContainsNull
Indicates if map values have
null
values.
- Annotations
- @Stable()
-
sealed
class
Metadata extends Serializable
Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata].
Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata]. JSON is used for serialization.
The default constructor is private. User should use either MetadataBuilder or
Metadata.fromJson()
to create Metadata instances.- Annotations
- @Stable()
- Since
1.3.0
-
class
MetadataBuilder extends AnyRef
Builder for Metadata.
Builder for Metadata. If there is a key collision, the latter will overwrite the former.
- Annotations
- @Stable()
- Since
1.3.0
-
class
NullType extends DataType
The data type representing
NULL
values.The data type representing
NULL
values. Please use the singletonDataTypes.NullType
.- Annotations
- @Stable()
- Since
1.3.0
-
abstract
class
NumericType extends AtomicType
Numeric data types.
Numeric data types.
- Annotations
- @Stable()
- Since
1.3.0
-
case class
ObjectType(cls: Class[_]) extends DataType with Product with Serializable
Represents a JVM object that is passing through Spark SQL expression evaluation.
- class SQLUserDefinedType extends Annotation with Annotation with ClassfileAnnotation
-
class
ShortType extends IntegralType
The data type representing
Short
values.The data type representing
Short
values. Please use the singletonDataTypes.ShortType
.- Annotations
- @Stable()
- Since
1.3.0
-
class
StringType extends AtomicType
The data type representing
String
values.The data type representing
String
values. Please use the singletonDataTypes.StringType
.- Annotations
- @Stable()
- Since
1.3.0
-
case class
StructField(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty) extends Product with Serializable
A field inside a StructType.
A field inside a StructType.
- name
The name of this field.
- dataType
The data type of this field.
- nullable
Indicates if values of this field can be
null
values.- metadata
The metadata of this field. The metadata should be preserved during transformation if the content of the column is not modified, e.g, in selection.
- Annotations
- @Stable()
- Since
1.3.0
-
case class
StructType(fields: Array[StructField]) extends DataType with Seq[StructField] with Product with Serializable
A StructType object can be constructed by
A StructType object can be constructed by
StructType(fields: Seq[StructField])
For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a
null
will be returned.Scala Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val struct = StructType( StructField("a", IntegerType, true) :: StructField("b", LongType, false) :: StructField("c", BooleanType, false) :: Nil) // Extract a single StructField. val singleField = struct("b") // singleField: StructField = StructField(b,LongType,false) // If this struct does not have a field called "d", it throws an exception. struct("d") // java.lang.IllegalArgumentException: d does not exist. // ... // Extract multiple StructFields. Field names are provided in a set. // A StructType object will be returned. val twoFields = struct(Set("b", "c")) // twoFields: StructType = // StructType(StructField(b,LongType,false), StructField(c,BooleanType,false)) // Any names without matching fields will throw an exception. // For the case shown below, an exception is thrown due to "d". struct(Set("b", "c", "d")) // java.lang.IllegalArgumentException: d does not exist. // ...
A org.apache.spark.sql.Row object is used as a value of the StructType.
Scala Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val innerStruct = StructType( StructField("f1", IntegerType, true) :: StructField("f2", LongType, false) :: StructField("f3", BooleanType, false) :: Nil) val struct = StructType( StructField("a", innerStruct, true) :: Nil) // Create a Row with the schema defined by struct val row = Row(Row(1, 2, true))
- Annotations
- @Stable()
- Since
1.3.0
-
class
TimestampType extends AtomicType
The timestamp type represents a time instant in microsecond precision.
The timestamp type represents a time instant in microsecond precision. Valid range is [0001-01-01T00:00:00.000000Z, 9999-12-31T23:59:59.999999Z] where the left/right-bound is a date and time of the proleptic Gregorian calendar in UTC+00:00.
Please use the singleton
DataTypes.TimestampType
to refer the type.- Annotations
- @Stable()
- Since
1.3.0
-
case class
VarcharType(length: Int) extends HiveStringType with Product with Serializable
Hive varchar type.
Hive varchar type. Similar to other HiveStringType's, these datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
Value Members
-
val
HIVE_TYPE_STRING: String
Metadata key used to store the raw hive type string in the metadata of StructField.
Metadata key used to store the raw hive type string in the metadata of StructField. This is relevant for datatypes that do not have a direct Spark SQL counterpart, such as CHAR and VARCHAR. We need to preserve the original type in order to invoke the correct object inspector in Hive.
-
object
AnyDataType extends AbstractDataType with Serializable
An
AbstractDataType
that matches any concrete data types.An
AbstractDataType
that matches any concrete data types.- Attributes
- protected[sql]
-
object
ArrayType extends AbstractDataType with Serializable
Companion object for ArrayType.
Companion object for ArrayType.
- Annotations
- @Stable()
- Since
1.3.0
- object AtomicType
-
object
BinaryType extends BinaryType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
BooleanType extends BooleanType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
ByteType extends ByteType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
CalendarIntervalType extends CalendarIntervalType with Product with Serializable
- Annotations
- @Stable()
- Since
1.5.0
-
object
DataType
- Annotations
- @Stable()
- Since
1.3.0
-
object
DateType extends DateType with Product with Serializable
The companion case object and the DateType class is separated so the companion object also subclasses the class.
The companion case object and the DateType class is separated so the companion object also subclasses the class. Otherwise, the companion object would be of type "DateType$" in byte code. The DateType class is defined with a private constructor so its companion object is the only possible instantiation.
- Annotations
- @Stable()
- Since
1.3.0
-
object
Decimal extends Serializable
- Annotations
- @Unstable()
-
object
DecimalType extends AbstractDataType with Serializable
Extra factory methods and pattern matchers for Decimals.
Extra factory methods and pattern matchers for Decimals.
- Annotations
- @Stable()
- Since
1.3.0
-
object
DoubleType extends DoubleType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
FloatType extends FloatType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
- object HiveStringType
-
object
IntegerType extends IntegerType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
LongType extends LongType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
MapType extends AbstractDataType with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
Metadata extends Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
NullType extends NullType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
- object ObjectType extends AbstractDataType with Serializable
-
object
ShortType extends ShortType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
StringType extends StringType with Product with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
StructType extends AbstractDataType with Serializable
- Annotations
- @Stable()
- Since
1.3.0
-
object
TimestampType extends TimestampType with Product with Serializable
The companion case object and its class is separated so the companion object also subclasses the TimestampType class.
The companion case object and its class is separated so the companion object also subclasses the TimestampType class. Otherwise, the companion object would be of type "TimestampType$" in byte code. Defined with a private constructor so the companion object is the only possible instantiation.
- Annotations
- @Stable()
- Since
1.3.0