public class JavaSparkContext
extends Object
implements java.io.Closeable
SparkContext
that returns
JavaRDD
s and works with Java collections instead of Scala ones.
Only one SparkContext may be active per JVM. You must stop()
the active SparkContext before
creating a new one. This limitation may eventually be removed; see SPARK-2243 for more details.
Constructor and Description |
---|
JavaSparkContext()
Create a JavaSparkContext that loads settings from system properties (for instance, when
launching with ./bin/spark-submit).
|
JavaSparkContext(SparkConf conf) |
JavaSparkContext(SparkContext sc) |
JavaSparkContext(String master,
String appName) |
JavaSparkContext(String master,
String appName,
SparkConf conf) |
JavaSparkContext(String master,
String appName,
String sparkHome,
String jarFile) |
JavaSparkContext(String master,
String appName,
String sparkHome,
String[] jars) |
JavaSparkContext(String master,
String appName,
String sparkHome,
String[] jars,
java.util.Map<String,String> environment) |
Modifier and Type | Method and Description |
---|---|
<T,R> Accumulable<T,R> |
accumulable(T initialValue,
AccumulableParam<T,R> param)
Deprecated.
use AccumulatorV2. Since 2.0.0.
|
<T,R> Accumulable<T,R> |
accumulable(T initialValue,
String name,
AccumulableParam<T,R> param)
Deprecated.
use AccumulatorV2. Since 2.0.0.
|
Accumulator<Double> |
accumulator(double initialValue)
Deprecated.
use sc().doubleAccumulator(). Since 2.0.0.
|
Accumulator<Double> |
accumulator(double initialValue,
String name)
Deprecated.
use sc().doubleAccumulator(String). Since 2.0.0.
|
Accumulator<Integer> |
accumulator(int initialValue)
Deprecated.
use sc().longAccumulator(). Since 2.0.0.
|
Accumulator<Integer> |
accumulator(int initialValue,
String name)
Deprecated.
use sc().longAccumulator(String). Since 2.0.0.
|
<T> Accumulator<T> |
accumulator(T initialValue,
AccumulatorParam<T> accumulatorParam)
Deprecated.
use AccumulatorV2. Since 2.0.0.
|
<T> Accumulator<T> |
accumulator(T initialValue,
String name,
AccumulatorParam<T> accumulatorParam)
Deprecated.
use AccumulatorV2. Since 2.0.0.
|
void |
addFile(String path)
Add a file to be downloaded with this Spark job on every node.
|
void |
addFile(String path,
boolean recursive)
Add a file to be downloaded with this Spark job on every node.
|
void |
addJar(String path)
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
|
String |
appName() |
JavaPairRDD<String,PortableDataStream> |
binaryFiles(String path)
Read a directory of binary files from HDFS, a local file system (available on all nodes),
or any Hadoop-supported file system URI as a byte array.
|
JavaPairRDD<String,PortableDataStream> |
binaryFiles(String path,
int minPartitions)
Read a directory of binary files from HDFS, a local file system (available on all nodes),
or any Hadoop-supported file system URI as a byte array.
|
JavaRDD<byte[]> |
binaryRecords(String path,
int recordLength)
Load data from a flat binary file, assuming the length of each record is constant.
|
<T> Broadcast<T> |
broadcast(T value)
Broadcast a read-only variable to the cluster, returning a
Broadcast object for reading it in distributed functions. |
void |
cancelAllJobs()
Cancel all jobs that have been scheduled or are running.
|
void |
cancelJobGroup(String groupId)
Cancel active jobs for the specified group.
|
void |
clearCallSite()
Pass-through to SparkContext.setCallSite.
|
void |
clearJobGroup()
Clear the current thread's job group ID and its description.
|
void |
close() |
Integer |
defaultMinPartitions()
Default min number of partitions for Hadoop RDDs when not given by user
|
Integer |
defaultParallelism()
Default level of parallelism to use when not given by user (e.g.
|
Accumulator<Double> |
doubleAccumulator(double initialValue)
Deprecated.
use sc().doubleAccumulator(). Since 2.0.0.
|
Accumulator<Double> |
doubleAccumulator(double initialValue,
String name)
Deprecated.
use sc().doubleAccumulator(String). Since 2.0.0.
|
<T> JavaRDD<T> |
emptyRDD()
Get an RDD that has no partitions or elements.
|
static JavaSparkContext |
fromSparkContext(SparkContext sc) |
Optional<String> |
getCheckpointDir() |
SparkConf |
getConf()
Return a copy of this JavaSparkContext's configuration.
|
String |
getLocalProperty(String key)
Get a local property set in this thread, or null if it is missing.
|
java.util.Map<Integer,JavaRDD<?>> |
getPersistentRDDs()
Returns a Java map of JavaRDDs that have marked themselves as persistent via cache() call.
|
Optional<String> |
getSparkHome()
Get Spark's home location from either a value set through the constructor,
or the spark.home Java property, or the SPARK_HOME environment variable
(in that order of preference).
|
org.apache.hadoop.conf.Configuration |
hadoopConfiguration()
Returns the Hadoop configuration used for the Hadoop code (e.g.
|
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> |
hadoopFile(String path,
Class<F> inputFormatClass,
Class<K> keyClass,
Class<V> valueClass)
Get an RDD for a Hadoop file with an arbitrary InputFormat
|
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> |
hadoopFile(String path,
Class<F> inputFormatClass,
Class<K> keyClass,
Class<V> valueClass,
int minPartitions)
Get an RDD for a Hadoop file with an arbitrary InputFormat.
|
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> |
hadoopRDD(org.apache.hadoop.mapred.JobConf conf,
Class<F> inputFormatClass,
Class<K> keyClass,
Class<V> valueClass)
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any
other necessary info (e.g.
|
<K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> |
hadoopRDD(org.apache.hadoop.mapred.JobConf conf,
Class<F> inputFormatClass,
Class<K> keyClass,
Class<V> valueClass,
int minPartitions)
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any
other necessary info (e.g.
|
Accumulator<Integer> |
intAccumulator(int initialValue)
Deprecated.
use sc().longAccumulator(). Since 2.0.0.
|
Accumulator<Integer> |
intAccumulator(int initialValue,
String name)
Deprecated.
use sc().longAccumulator(String). Since 2.0.0.
|
Boolean |
isLocal() |
static String[] |
jarOfClass(Class<?> cls)
Find the JAR from which a given class was loaded, to make it easy for users to pass
their JARs to SparkContext.
|
static String[] |
jarOfObject(Object obj)
Find the JAR that contains the class of a particular object, to make it easy for users
to pass their JARs to SparkContext.
|
java.util.List<String> |
jars() |
String |
master() |
<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> |
newAPIHadoopFile(String path,
Class<F> fClass,
Class<K> kClass,
Class<V> vClass,
org.apache.hadoop.conf.Configuration conf)
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat
and extra configuration options to pass to the input format.
|
<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> |
newAPIHadoopRDD(org.apache.hadoop.conf.Configuration conf,
Class<F> fClass,
Class<K> kClass,
Class<V> vClass)
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat
and extra configuration options to pass to the input format.
|
<T> JavaRDD<T> |
objectFile(String path)
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and
BytesWritable values that contain a serialized partition.
|
<T> JavaRDD<T> |
objectFile(String path,
int minPartitions)
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and
BytesWritable values that contain a serialized partition.
|
<T> JavaRDD<T> |
parallelize(java.util.List<T> list)
Distribute a local Scala collection to form an RDD.
|
<T> JavaRDD<T> |
parallelize(java.util.List<T> list,
int numSlices)
Distribute a local Scala collection to form an RDD.
|
JavaDoubleRDD |
parallelizeDoubles(java.util.List<Double> list)
Distribute a local Scala collection to form an RDD.
|
JavaDoubleRDD |
parallelizeDoubles(java.util.List<Double> list,
int numSlices)
Distribute a local Scala collection to form an RDD.
|
<K,V> JavaPairRDD<K,V> |
parallelizePairs(java.util.List<scala.Tuple2<K,V>> list)
Distribute a local Scala collection to form an RDD.
|
<K,V> JavaPairRDD<K,V> |
parallelizePairs(java.util.List<scala.Tuple2<K,V>> list,
int numSlices)
Distribute a local Scala collection to form an RDD.
|
SparkContext |
sc() |
<K,V> JavaPairRDD<K,V> |
sequenceFile(String path,
Class<K> keyClass,
Class<V> valueClass)
Get an RDD for a Hadoop SequenceFile.
|
<K,V> JavaPairRDD<K,V> |
sequenceFile(String path,
Class<K> keyClass,
Class<V> valueClass,
int minPartitions)
Get an RDD for a Hadoop SequenceFile with given key and value types.
|
void |
setCallSite(String site)
Pass-through to SparkContext.setCallSite.
|
void |
setCheckpointDir(String dir)
Set the directory under which RDDs are going to be checkpointed.
|
void |
setJobGroup(String groupId,
String description)
Assigns a group ID to all the jobs started by this thread until the group ID is set to a
different value or cleared.
|
void |
setJobGroup(String groupId,
String description,
boolean interruptOnCancel)
Assigns a group ID to all the jobs started by this thread until the group ID is set to a
different value or cleared.
|
void |
setLocalProperty(String key,
String value)
Set a local property that affects jobs submitted from this thread, and all child
threads, such as the Spark fair scheduler pool.
|
void |
setLogLevel(String logLevel)
Control our logLevel.
|
String |
sparkUser() |
Long |
startTime() |
JavaSparkStatusTracker |
statusTracker() |
void |
stop()
Shut down the SparkContext.
|
JavaRDD<String> |
textFile(String path)
Read a text file from HDFS, a local file system (available on all nodes), or any
Hadoop-supported file system URI, and return it as an RDD of Strings.
|
JavaRDD<String> |
textFile(String path,
int minPartitions)
Read a text file from HDFS, a local file system (available on all nodes), or any
Hadoop-supported file system URI, and return it as an RDD of Strings.
|
static SparkContext |
toSparkContext(JavaSparkContext jsc) |
JavaDoubleRDD |
union(JavaDoubleRDD... rdds) |
JavaDoubleRDD |
union(JavaDoubleRDD first,
java.util.List<JavaDoubleRDD> rest)
Build the union of two or more RDDs.
|
<K,V> JavaPairRDD<K,V> |
union(JavaPairRDD<K,V>... rdds) |
<K,V> JavaPairRDD<K,V> |
union(JavaPairRDD<K,V> first,
java.util.List<JavaPairRDD<K,V>> rest)
Build the union of two or more RDDs.
|
<T> JavaRDD<T> |
union(JavaRDD<T>... rdds) |
<T> JavaRDD<T> |
union(JavaRDD<T> first,
java.util.List<JavaRDD<T>> rest)
Build the union of two or more RDDs.
|
String |
version()
The version of Spark on which this application is running.
|
JavaPairRDD<String,String> |
wholeTextFiles(String path)
Read a directory of text files from HDFS, a local file system (available on all nodes), or any
Hadoop-supported file system URI.
|
JavaPairRDD<String,String> |
wholeTextFiles(String path,
int minPartitions)
Read a directory of text files from HDFS, a local file system (available on all nodes), or any
Hadoop-supported file system URI.
|
public JavaSparkContext(SparkContext sc)
public JavaSparkContext()
public JavaSparkContext(SparkConf conf)
conf
- a SparkConf
object specifying Spark parameterspublic JavaSparkContext(String master, String appName)
master
- Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).appName
- A name for your application, to display on the cluster web UIpublic JavaSparkContext(String master, String appName, SparkConf conf)
master
- Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).appName
- A name for your application, to display on the cluster web UIconf
- a SparkConf
object specifying other Spark parameterspublic JavaSparkContext(String master, String appName, String sparkHome, String jarFile)
master
- Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).appName
- A name for your application, to display on the cluster web UIsparkHome
- The SPARK_HOME directory on the slave nodesjarFile
- JAR file to send to the cluster. This can be a path on the local file system
or an HDFS, HTTP, HTTPS, or FTP URL.public JavaSparkContext(String master, String appName, String sparkHome, String[] jars)
master
- Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).appName
- A name for your application, to display on the cluster web UIsparkHome
- The SPARK_HOME directory on the slave nodesjars
- Collection of JARs to send to the cluster. These can be paths on the local file
system or HDFS, HTTP, HTTPS, or FTP URLs.public JavaSparkContext(String master, String appName, String sparkHome, String[] jars, java.util.Map<String,String> environment)
master
- Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).appName
- A name for your application, to display on the cluster web UIsparkHome
- The SPARK_HOME directory on the slave nodesjars
- Collection of JARs to send to the cluster. These can be paths on the local file
system or HDFS, HTTP, HTTPS, or FTP URLs.environment
- Environment variables to set on worker nodespublic static JavaSparkContext fromSparkContext(SparkContext sc)
public static SparkContext toSparkContext(JavaSparkContext jsc)
public static String[] jarOfClass(Class<?> cls)
cls
- (undocumented)public static String[] jarOfObject(Object obj)
obj
- (undocumented)public SparkContext sc()
public JavaSparkStatusTracker statusTracker()
public Boolean isLocal()
public String sparkUser()
public String master()
public String appName()
public java.util.List<String> jars()
public Long startTime()
public String version()
public Integer defaultParallelism()
public Integer defaultMinPartitions()
public <T> JavaRDD<T> parallelize(java.util.List<T> list, int numSlices)
public <T> JavaRDD<T> emptyRDD()
public <T> JavaRDD<T> parallelize(java.util.List<T> list)
public <K,V> JavaPairRDD<K,V> parallelizePairs(java.util.List<scala.Tuple2<K,V>> list, int numSlices)
public <K,V> JavaPairRDD<K,V> parallelizePairs(java.util.List<scala.Tuple2<K,V>> list)
public JavaDoubleRDD parallelizeDoubles(java.util.List<Double> list, int numSlices)
public JavaDoubleRDD parallelizeDoubles(java.util.List<Double> list)
public JavaRDD<String> textFile(String path)
path
- (undocumented)public JavaRDD<String> textFile(String path, int minPartitions)
path
- (undocumented)minPartitions
- (undocumented)public JavaPairRDD<String,String> wholeTextFiles(String path, int minPartitions)
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn
Do
JavaPairRDD<String, String> rdd = sparkContext.wholeTextFiles("hdfs://a-hdfs-path")
then rdd
contains
(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)
minPartitions
- A suggestion value of the minimal splitting number for input data.path
- (undocumented)public JavaPairRDD<String,String> wholeTextFiles(String path)
path
- (undocumented)wholeTextFiles(path: String, minPartitions: Int)
.public JavaPairRDD<String,PortableDataStream> binaryFiles(String path, int minPartitions)
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn
Do
JavaPairRDD<String, byte[]> rdd = sparkContext.dataStreamFiles("hdfs://a-hdfs-path")
then rdd
contains
(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)
minPartitions
- A suggestion value of the minimal splitting number for input data.path
- (undocumented)public JavaPairRDD<String,PortableDataStream> binaryFiles(String path)
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn
Do
JavaPairRDD<String, byte[]> rdd = sparkContext.dataStreamFiles("hdfs://a-hdfs-path")
,
then rdd
contains
(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)
path
- (undocumented)public JavaRDD<byte[]> binaryRecords(String path, int recordLength)
path
- Directory to the input data filesrecordLength
- (undocumented)public <K,V> JavaPairRDD<K,V> sequenceFile(String path, Class<K> keyClass, Class<V> valueClass, int minPartitions)
path
- (undocumented)keyClass
- (undocumented)valueClass
- (undocumented)minPartitions
- (undocumented)map
function.public <K,V> JavaPairRDD<K,V> sequenceFile(String path, Class<K> keyClass, Class<V> valueClass)
path
- (undocumented)keyClass
- (undocumented)valueClass
- (undocumented)map
function.public <T> JavaRDD<T> objectFile(String path, int minPartitions)
path
- (undocumented)minPartitions
- (undocumented)public <T> JavaRDD<T> objectFile(String path)
path
- (undocumented)public <K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> JavaPairRDD<K,V> hadoopRDD(org.apache.hadoop.mapred.JobConf conf, Class<F> inputFormatClass, Class<K> keyClass, Class<V> valueClass, int minPartitions)
conf
- JobConf for setting up the dataset. Note: This will be put into a Broadcast.
Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
sure you won't modify the conf. A safe approach is always creating a new conf for
a new RDD.inputFormatClass
- Class of the InputFormatkeyClass
- Class of the keysvalueClass
- Class of the valuesminPartitions
- Minimum number of Hadoop Splits to generate.
map
function.public <K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> JavaPairRDD<K,V> hadoopRDD(org.apache.hadoop.mapred.JobConf conf, Class<F> inputFormatClass, Class<K> keyClass, Class<V> valueClass)
conf
- JobConf for setting up the dataset. Note: This will be put into a Broadcast.
Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
sure you won't modify the conf. A safe approach is always creating a new conf for
a new RDD.inputFormatClass
- Class of the InputFormatkeyClass
- Class of the keysvalueClass
- Class of the values
map
function.public <K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> JavaPairRDD<K,V> hadoopFile(String path, Class<F> inputFormatClass, Class<K> keyClass, Class<V> valueClass, int minPartitions)
path
- (undocumented)inputFormatClass
- (undocumented)keyClass
- (undocumented)valueClass
- (undocumented)minPartitions
- (undocumented)map
function.public <K,V,F extends org.apache.hadoop.mapred.InputFormat<K,V>> JavaPairRDD<K,V> hadoopFile(String path, Class<F> inputFormatClass, Class<K> keyClass, Class<V> valueClass)
path
- (undocumented)inputFormatClass
- (undocumented)keyClass
- (undocumented)valueClass
- (undocumented)map
function.public <K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> JavaPairRDD<K,V> newAPIHadoopFile(String path, Class<F> fClass, Class<K> kClass, Class<V> vClass, org.apache.hadoop.conf.Configuration conf)
path
- (undocumented)fClass
- (undocumented)kClass
- (undocumented)vClass
- (undocumented)conf
- (undocumented)map
function.public <K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>> JavaPairRDD<K,V> newAPIHadoopRDD(org.apache.hadoop.conf.Configuration conf, Class<F> fClass, Class<K> kClass, Class<V> vClass)
conf
- Configuration for setting up the dataset. Note: This will be put into a Broadcast.
Therefore if you plan to reuse this conf to create multiple RDDs, you need to make
sure you won't modify the conf. A safe approach is always creating a new conf for
a new RDD.fClass
- Class of the InputFormatkClass
- Class of the keysvClass
- Class of the values
map
function.public <T> JavaRDD<T> union(JavaRDD<T> first, java.util.List<JavaRDD<T>> rest)
public <K,V> JavaPairRDD<K,V> union(JavaPairRDD<K,V> first, java.util.List<JavaPairRDD<K,V>> rest)
public JavaDoubleRDD union(JavaDoubleRDD first, java.util.List<JavaDoubleRDD> rest)
public Accumulator<Integer> intAccumulator(int initialValue)
Accumulator
integer variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.initialValue
- (undocumented)public Accumulator<Integer> intAccumulator(int initialValue, String name)
Accumulator
integer variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)public Accumulator<Double> doubleAccumulator(double initialValue)
Accumulator
double variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.initialValue
- (undocumented)public Accumulator<Double> doubleAccumulator(double initialValue, String name)
Accumulator
double variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)public Accumulator<Integer> accumulator(int initialValue)
Accumulator
integer variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.initialValue
- (undocumented)public Accumulator<Integer> accumulator(int initialValue, String name)
Accumulator
integer variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)public Accumulator<Double> accumulator(double initialValue)
Accumulator
double variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.initialValue
- (undocumented)public Accumulator<Double> accumulator(double initialValue, String name)
Accumulator
double variable, which tasks can "add" values
to using the add
method. Only the master can access the accumulator's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)public <T> Accumulator<T> accumulator(T initialValue, AccumulatorParam<T> accumulatorParam)
Accumulator
variable of a given type, which tasks can "add"
values to using the add
method. Only the master can access the accumulator's value
.initialValue
- (undocumented)accumulatorParam
- (undocumented)public <T> Accumulator<T> accumulator(T initialValue, String name, AccumulatorParam<T> accumulatorParam)
Accumulator
variable of a given type, which tasks can "add"
values to using the add
method. Only the master can access the accumulator's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)accumulatorParam
- (undocumented)public <T,R> Accumulable<T,R> accumulable(T initialValue, AccumulableParam<T,R> param)
Accumulable
shared variable of the given type, to which tasks
can "add" values with add
. Only the master can access the accumulable's value
.initialValue
- (undocumented)param
- (undocumented)public <T,R> Accumulable<T,R> accumulable(T initialValue, String name, AccumulableParam<T,R> param)
Accumulable
shared variable of the given type, to which tasks
can "add" values with add
. Only the master can access the accumulable's value
.
This version supports naming the accumulator for display in Spark's web UI.
initialValue
- (undocumented)name
- (undocumented)param
- (undocumented)public <T> Broadcast<T> broadcast(T value)
Broadcast
object for reading it in distributed functions.
The variable will be sent to each cluster only once.value
- (undocumented)public void stop()
public void close()
close
in interface java.io.Closeable
close
in interface AutoCloseable
public Optional<String> getSparkHome()
public void addFile(String path)
path
passed can be either a local file, a file in HDFS (or other Hadoop-supported
filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
use SparkFiles.get(fileName)
to find its download location.path
- (undocumented)public void addFile(String path, boolean recursive)
path
passed can be either a local file, a file in HDFS (or other Hadoop-supported
filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs,
use SparkFiles.get(fileName)
to find its download location.
A directory can be given if the recursive option is set to true. Currently directories are only supported for Hadoop-supported filesystems.
path
- (undocumented)recursive
- (undocumented)public void addJar(String path)
path
passed can be either a local file, a file in HDFS (or other Hadoop-supported
filesystems), or an HTTP, HTTPS or FTP URI.path
- (undocumented)public org.apache.hadoop.conf.Configuration hadoopConfiguration()
public void setCheckpointDir(String dir)
dir
- (undocumented)public Optional<String> getCheckpointDir()
public SparkConf getConf()
public void setCallSite(String site)
site
- (undocumented)public void clearCallSite()
public void setLocalProperty(String key, String value)
These properties are inherited by child threads spawned from this thread. This may have unexpected consequences when working with thread pools. The standard java implementation of thread pools have worker threads spawn other worker threads. As a result, local properties may propagate unpredictably.
key
- (undocumented)value
- (undocumented)public String getLocalProperty(String key)
org.apache.spark.api.java.JavaSparkContext.setLocalProperty
.key
- (undocumented)public void setLogLevel(String logLevel)
logLevel
- The desired log level as a string.
Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARNpublic void setJobGroup(String groupId, String description, boolean interruptOnCancel)
Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application programmers can use this method to group all those jobs together and give a group description. Once set, the Spark web UI will associate such jobs with this group.
The application can also use org.apache.spark.api.java.JavaSparkContext.cancelJobGroup
to cancel all running jobs in this group. For example,
// In the main thread:
sc.setJobGroup("some_job_to_cancel", "some job description");
rdd.map(...).count();
// In a separate thread:
sc.cancelJobGroup("some_job_to_cancel");
If interruptOnCancel is set to true for the job group, then job cancellation will result in Thread.interrupt() being called on the job's executor threads. This is useful to help ensure that the tasks are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS may respond to Thread.interrupt() by marking nodes as dead.
groupId
- (undocumented)description
- (undocumented)interruptOnCancel
- (undocumented)public void setJobGroup(String groupId, String description)
groupId
- (undocumented)description
- (undocumented)setJobGroup(groupId: String, description: String, interruptThread: Boolean)
.
This method sets interruptOnCancel to false.public void clearJobGroup()
public void cancelJobGroup(String groupId)
org.apache.spark.api.java.JavaSparkContext.setJobGroup
for more information.groupId
- (undocumented)public void cancelAllJobs()
public java.util.Map<Integer,JavaRDD<?>> getPersistentRDDs()
public JavaDoubleRDD union(JavaDoubleRDD... rdds)
@SafeVarargs public final <K,V> JavaPairRDD<K,V> union(JavaPairRDD<K,V>... rdds)