streaming

Spark Streaming functionality. org.apache.spark.streaming.StreamingContext serves as the main entry point to Spark Streaming, while org.apache.spark.streaming.dstream.DStream is the data type representing a continuous sequence of RDDs, representing a continuous stream of data.

In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKey and reduceByKey. These operations are automatically available on any DStream of the right type (e.g. DStream[(Int, Int)] through implicit conversions.

For the Java API of Spark Streaming, take a look at the org.apache.spark.streaming.api.java.JavaStreamingContext which serves as the entry point, and the org.apache.spark.streaming.api.java.JavaDStream and the org.apache.spark.streaming.api.java.JavaPairDStream which have the DStream functionality.

Source: package.scala

Linear Supertypes

AnyRef, Any

Type Members

case class Duration(millis: Long) extends Product with Serializable
sealed abstract class State[S] extends AnyRef

:: Experimental :: Abstract class for getting and updating the state in mapping function used in the mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).
sealed abstract class StateSpec[KeyType, ValueType, StateType, MappedType] extends Serializable

:: Experimental :: Abstract class representing all the specifications of the DStream transformation mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).
class StreamingContext extends Logging

Main entry point for Spark Streaming functionality.
class StreamingContextState extends Enum[StreamingContextState]
case class Time(millis: Long) extends Product with Serializable

This is a simple class that represents an absolute instant of time.

Value Members

object Durations
object Milliseconds

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of milliseconds.
object Minutes

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of minutes.
object Seconds

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of seconds.
object StateSpec extends Serializable

:: Experimental :: Builder object for creating instances of StateSpec that is used for specifying the parameters of the DStream transformation mapWithState that is used for specifying the parameters of the DStream transformation mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).
object StreamingContext extends Logging

StreamingContext object contains a number of utility functions related to the StreamingContext class.
object Time extends Serializable
package api
package dstream

Various implementations of DStream's.
package flume

Spark streaming receiver for Flume.
package kafka

Kafka receiver for spark streaming,
package kinesis
package mqtt

MQTT receiver for Spark Streaming.
package receiver
package scheduler
package twitter

Twitter feed receiver for spark streaming.
package util
package zeromq

Zeromq receiver for spark streaming.

package streaming

Type Members

case class Duration(millis: Long) extends Product with Serializable

sealed abstract class State[S] extends AnyRef

sealed abstract class StateSpec[KeyType, ValueType, StateType, MappedType] extends Serializable

class StreamingContext extends Logging

class StreamingContextState extends Enum[StreamingContextState]

case class Time(millis: Long) extends Product with Serializable

Value Members

object Durations

object Milliseconds

object Minutes

object Seconds

object StateSpec extends Serializable

object StreamingContext extends Logging

object Time extends Serializable

package api

package dstream

package flume

package kafka

package kinesis

package mqtt

package receiver

package scheduler

package twitter

package util

package zeromq

Inherited from AnyRef

Inherited from Any

Ungrouped