JavaSparkContext

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def accumulable[T, R](initialValue: T, param: AccumulableParam[T, R]): Accumulable[T, R]

   Create an Accumulable shared variable of the given type, to which tasks can "add" values with add.

   Create an Accumulable shared variable of the given type, to which tasks can "add" values with add. Only the master can access the accumuable's value.

7. def accumulator[T](initialValue: T, accumulatorParam: AccumulatorParam[T]): Accumulator[T]

   Create an Accumulator variable of a given type, which tasks can "add" values to using the add method.

   Create an Accumulator variable of a given type, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

8. def accumulator(initialValue: Double): Accumulator[Double]

   Create an Accumulator double variable, which tasks can "add" values to using the add method.

   Create an Accumulator double variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

9. def accumulator(initialValue: Int): Accumulator[Integer]

   Create an Accumulator integer variable, which tasks can "add" values to using the add method.

   Create an Accumulator integer variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

10. def addFile(path: String): Unit

    Add a file to be downloaded with this Spark job on every node.

    Add a file to be downloaded with this Spark job on every node. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, use SparkFiles.get(path) to find its download location.

11. def addJar(path: String): Unit

    Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

    Adds a JAR dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

12. final def asInstanceOf[T0]: T0

    Definition Classes

    Any

13. def broadcast[T](value: T): Broadcast[T]

    Broadcast a read-only variable to the cluster, returning a org.apache.spark.Broadcast object for reading it in distributed functions.

    Broadcast a read-only variable to the cluster, returning a org.apache.spark.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

14. def checkpointFile[T](path: String): JavaRDD[T]

    Attributes

    protected

15. def clearFiles(): Unit

    Clear the job's list of files added by addFile so that they do not get downloaded to any new nodes.

16. def clearJars(): Unit

    Clear the job's list of JARs added by addJar so that they do not get downloaded to any new nodes.

17. def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws()

18. def doubleAccumulator(initialValue: Double): Accumulator[Double]

    Create an Accumulator double variable, which tasks can "add" values to using the add method.

    Create an Accumulator double variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

19. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

20. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

21. def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws()

22. final def getClass(): java.lang.Class[_]

    Definition Classes

    AnyRef → Any

23. def getSparkHome(): Optional[String]

    Get Spark's home location from either a value set through the constructor, or the spark.

    Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.

24. def hadoopConfiguration(): Configuration

    Returns the Hadoop configuration used for the Hadoop code (e.

    Returns the Hadoop configuration used for the Hadoop code (e.g. file systems) we reuse.

25. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

26. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

27. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

28. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

29. def hashCode(): Int

    Definition Classes

    AnyRef → Any

30. def intAccumulator(initialValue: Int): Accumulator[Integer]

    Create an Accumulator integer variable, which tasks can "add" values to using the add method.

    Create an Accumulator integer variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

31. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

32. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

33. def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

34. def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

35. final def notify(): Unit

    Definition Classes

    AnyRef

36. final def notifyAll(): Unit

    Definition Classes

    AnyRef

37. def objectFile[T](path: String): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

38. def objectFile[T](path: String, minSplits: Int): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

39. def parallelize[T](list: List[T]): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

40. def parallelize[T](list: List[T], numSlices: Int): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

41. def parallelizeDoubles(list: List[Double]): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

42. def parallelizeDoubles(list: List[Double], numSlices: Int): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

43. def parallelizePairs[K, V](list: List[(K, V)]): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

44. def parallelizePairs[K, V](list: List[(K, V)], numSlices: Int): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

45. val sc: SparkContext

46. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile.

47. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile with given key and value types.

48. def setCheckpointDir(dir: String): Unit

    Set the directory under which RDDs are going to be checkpointed.

    Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster. If the directory does not exist, it will be created. If the directory exists, an exception will be thrown to prevent accidental overriding of checkpoint files.

49. def setCheckpointDir(dir: String, useExisting: Boolean): Unit

    Set the directory under which RDDs are going to be checkpointed.

    Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster. If the directory does not exist, it will be created. If the directory exists and useExisting is set to true, then the exisiting directory will be used. Otherwise an exception will be thrown to prevent accidental overriding of checkpoint files in the existing directory.

50. def stop(): Unit

    Shut down the SparkContext.

51. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

52. def textFile(path: String, minSplits: Int): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

53. def textFile(path: String): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

54. def toString(): String

    Definition Classes

    AnyRef → Any

55. def union(first: JavaDoubleRDD, rest: List[JavaDoubleRDD]): JavaDoubleRDD

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

56. def union[K, V](first: JavaPairRDD[K, V], rest: List[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

57. def union[T](first: JavaRDD[T], rest: List[JavaRDD[T]]): JavaRDD[T]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

58. def union[K, V](rdds: <repeated...>[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Definition Classes

    JavaSparkContextVarargsWorkaround

59. def union(rdds: <repeated...>[JavaDoubleRDD]): JavaDoubleRDD

    Definition Classes

    JavaSparkContextVarargsWorkaround

60. def union[T](rdds: <repeated...>[JavaRDD[T]]): JavaRDD[T]

    Definition Classes

    JavaSparkContextVarargsWorkaround

61. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

62. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

63. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()