DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.
Abstract class for transformers that transform one dataset into another.
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Abstract class for estimators that fit models to data.
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Abstract class for models that are fitted by estimators.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
A simple pipeline, which acts as an estimator. A Pipeline consists of a sequence of stages, each of which is either an Estimator or a Transformer. When Pipeline.fit() is called, the stages are executed in order. If a stage is an Estimator, its Estimator.fit() method will be called on the input dataset to fit a model. Then the model, which is a transformer, will be used to transform the dataset as the input to the next stage. If a stage is a Transformer, its Transformer.transform() method will be called to produce the dataset for the next stage. The fitted model from a Pipeline is a PipelineModel, which consists of fitted models and transformers, corresponding to the pipeline stages. If there are no stages, the pipeline acts as an identity transformer.
New in version 1.3.0.
Creates a copy of this instance.
| Parameters: | extra – extra parameters |
|---|---|
| Returns: | new instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Get pipeline stages.
New in version 1.3.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
New in version 2.0.0.
Save this ML instance to the given path, a shortcut of write().save(path).
New in version 2.0.0.
Sets params for Pipeline.
New in version 1.3.0.
Set pipeline stages.
| Parameters: | value – a list of transformers or estimators |
|---|---|
| Returns: | the pipeline instance |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
New in version 2.0.0.
Represents a compiled pipeline with transformers and fitted models.
New in version 1.3.0.
Creates a copy of this instance.
| Parameters: | extra – extra parameters |
|---|---|
| Returns: | new instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
New in version 2.0.0.
Save this ML instance to the given path, a shortcut of write().save(path).
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
New in version 2.0.0.
A param with self-contained documentation.
New in version 1.3.0.
Components that take parameters. This also provides an internal param map to store parameter values attached to the instance.
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Binarize a column of continuous features given a threshold.
>>> df = spark.createDataFrame([(0.5,)], ["values"])
>>> binarizer = Binarizer(threshold=1.0, inputCol="values", outputCol="features")
>>> binarizer.transform(df).head().features
0.0
>>> binarizer.setParams(outputCol="freqs").transform(df).head().freqs
0.0
>>> params = {binarizer.threshold: -0.5, binarizer.outputCol: "vector"}
>>> binarizer.transform(df, params).head().vector
1.0
>>> binarizerPath = temp_path + "/binarizer"
>>> binarizer.save(binarizerPath)
>>> loadedBinarizer = Binarizer.load(binarizerPath)
>>> loadedBinarizer.getThreshold() == binarizer.getThreshold()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this Binarizer.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Maps a column of continuous features to a column of feature buckets.
>>> df = spark.createDataFrame([(0.1,), (0.4,), (1.2,), (1.5,)], ["values"])
>>> bucketizer = Bucketizer(splits=[-float("inf"), 0.5, 1.4, float("inf")],
... inputCol="values", outputCol="buckets")
>>> bucketed = bucketizer.transform(df).collect()
>>> bucketed[0].buckets
0.0
>>> bucketed[1].buckets
0.0
>>> bucketed[2].buckets
1.0
>>> bucketed[3].buckets
2.0
>>> bucketizer.setParams(outputCol="b").transform(df).head().b
0.0
>>> bucketizerPath = temp_path + "/bucketizer"
>>> bucketizer.save(bucketizerPath)
>>> loadedBucketizer = Bucketizer.load(bucketizerPath)
>>> loadedBucketizer.getSplits() == bucketizer.getSplits()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this Bucketizer.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame(
... [(Vectors.dense([0.0, 0.0, 18.0, 1.0]), 1.0),
... (Vectors.dense([0.0, 1.0, 12.0, 0.0]), 0.0),
... (Vectors.dense([1.0, 0.0, 15.0, 0.1]), 0.0)],
... ["features", "label"])
>>> selector = ChiSqSelector(numTopFeatures=1, outputCol="selectedFeatures")
>>> model = selector.fit(df)
>>> model.transform(df).head().selectedFeatures
DenseVector([1.0])
>>> model.selectedFeatures
[3]
>>> chiSqSelectorPath = temp_path + "/chi-sq-selector"
>>> selector.save(chiSqSelectorPath)
>>> loadedSelector = ChiSqSelector.load(chiSqSelectorPath)
>>> loadedSelector.getNumTopFeatures() == selector.getNumTopFeatures()
True
>>> modelPath = temp_path + "/chi-sq-selector-model"
>>> model.save(modelPath)
>>> loadedModel = ChiSqSelectorModel.load(modelPath)
>>> loadedModel.selectedFeatures == model.selectedFeatures
True
New in version 2.0.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of featuresCol or its default value.
Gets the value of labelCol or its default value.
Gets the value of numTopFeatures or its default value.
New in version 2.0.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of featuresCol.
Sets the value of numTopFeatures.
New in version 2.0.0.
Sets params for this ChiSqSelector.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Model fitted by ChiSqSelector.
New in version 2.0.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
List of indices to select (filter). Must be ordered asc.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Extracts a vocabulary from document collections and generates a CountVectorizerModel.
>>> df = spark.createDataFrame(
... [(0, ["a", "b", "c"]), (1, ["a", "b", "b", "c", "a"])],
... ["label", "raw"])
>>> cv = CountVectorizer(inputCol="raw", outputCol="vectors")
>>> model = cv.fit(df)
>>> model.transform(df).show(truncate=False)
+-----+---------------+-------------------------+
|label|raw |vectors |
+-----+---------------+-------------------------+
|0 |[a, b, c] |(3,[0,1,2],[1.0,1.0,1.0])|
|1 |[a, b, b, c, a]|(3,[0,1,2],[2.0,2.0,1.0])|
+-----+---------------+-------------------------+
...
>>> sorted(model.vocabulary) == ['a', 'b', 'c']
True
>>> countVectorizerPath = temp_path + "/count-vectorizer"
>>> cv.save(countVectorizerPath)
>>> loadedCv = CountVectorizer.load(countVectorizerPath)
>>> loadedCv.getMinDF() == cv.getMinDF()
True
>>> loadedCv.getMinTF() == cv.getMinTF()
True
>>> loadedCv.getVocabSize() == cv.getVocabSize()
True
>>> modelPath = temp_path + "/count-vectorizer-model"
>>> model.save(modelPath)
>>> loadedModel = CountVectorizerModel.load(modelPath)
>>> loadedModel.vocabulary == model.vocabulary
True
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Set the params for the CountVectorizer
New in version 1.6.0.
Returns an MLWriter instance for this ML instance.
Model fitted by CountVectorizer.
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A feature transformer that takes the 1D discrete cosine transform of a real vector. No zero padding is performed on the input vector. It returns a real vector of the same length representing the DCT. The return vector is scaled such that the transform matrix is unitary (aka scaled DCT-II).
See also
>>> from pyspark.ml.linalg import Vectors
>>> df1 = spark.createDataFrame([(Vectors.dense([5.0, 8.0, 6.0]),)], ["vec"])
>>> dct = DCT(inverse=False, inputCol="vec", outputCol="resultVec")
>>> df2 = dct.transform(df1)
>>> df2.head().resultVec
DenseVector([10.969..., -0.707..., -2.041...])
>>> df3 = DCT(inverse=True, inputCol="resultVec", outputCol="origVec").transform(df2)
>>> df3.head().origVec
DenseVector([5.0, 8.0, 6.0])
>>> dctPath = temp_path + "/dct"
>>> dct.save(dctPath)
>>> loadedDtc = DCT.load(dctPath)
>>> loadedDtc.getInverse()
False
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this DCT.
New in version 1.6.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided “weight” vector. In other words, it scales each column of the dataset by a scalar multiplier.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([2.0, 1.0, 3.0]),)], ["values"])
>>> ep = ElementwiseProduct(scalingVec=Vectors.dense([1.0, 2.0, 3.0]),
... inputCol="values", outputCol="eprod")
>>> ep.transform(df).head().eprod
DenseVector([2.0, 2.0, 9.0])
>>> ep.setParams(scalingVec=Vectors.dense([2.0, 3.0, 5.0])).transform(df).head().eprod
DenseVector([4.0, 3.0, 15.0])
>>> elementwiseProductPath = temp_path + "/elementwise-product"
>>> ep.save(elementwiseProductPath)
>>> loadedEp = ElementwiseProduct.load(elementwiseProductPath)
>>> loadedEp.getScalingVec() == ep.getScalingVec()
True
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this ElementwiseProduct.
New in version 1.5.0.
Sets the value of scalingVec.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Maps a sequence of terms to their term frequencies using the hashing trick. Currently we use Austin Appleby’s MurmurHash 3 algorithm (MurmurHash3_x86_32) to calculate the hash code value for the term object. Since a simple modulo is used to transform the hash function to a column index, it is advisable to use a power of two as the numFeatures parameter; otherwise the features will not be mapped evenly to the columns.
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ["words"])
>>> hashingTF = HashingTF(numFeatures=10, inputCol="words", outputCol="features")
>>> hashingTF.transform(df).head().features
SparseVector(10, {0: 1.0, 1: 1.0, 2: 1.0})
>>> hashingTF.setParams(outputCol="freqs").transform(df).head().freqs
SparseVector(10, {0: 1.0, 1: 1.0, 2: 1.0})
>>> params = {hashingTF.numFeatures: 5, hashingTF.outputCol: "vector"}
>>> hashingTF.transform(df, params).head().vector
SparseVector(5, {0: 1.0, 1: 1.0, 2: 1.0})
>>> hashingTFPath = temp_path + "/hashing-tf"
>>> hashingTF.save(hashingTFPath)
>>> loadedHashingTF = HashingTF.load(hashingTFPath)
>>> loadedHashingTF.getNumFeatures() == hashingTF.getNumFeatures()
True
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of numFeatures or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of numFeatures.
Sets params for this HashingTF.
New in version 1.3.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Compute the Inverse Document Frequency (IDF) given a collection of documents.
>>> from pyspark.ml.linalg import DenseVector
>>> df = spark.createDataFrame([(DenseVector([1.0, 2.0]),),
... (DenseVector([0.0, 1.0]),), (DenseVector([3.0, 0.2]),)], ["tf"])
>>> idf = IDF(minDocFreq=3, inputCol="tf", outputCol="idf")
>>> model = idf.fit(df)
>>> model.idf
DenseVector([0.0, 0.0])
>>> model.transform(df).head().idf
DenseVector([0.0, 0.0])
>>> idf.setParams(outputCol="freqs").fit(df).transform(df).collect()[1].freqs
DenseVector([0.0, 0.0])
>>> params = {idf.minDocFreq: 1, idf.outputCol: "vector"}
>>> idf.fit(df, params).transform(df).head().vector
DenseVector([0.2877, 0.0])
>>> idfPath = temp_path + "/idf"
>>> idf.save(idfPath)
>>> loadedIdf = IDF.load(idfPath)
>>> loadedIdf.getMinDocFreq() == idf.getMinDocFreq()
True
>>> modelPath = temp_path + "/idf-model"
>>> model.save(modelPath)
>>> loadedModel = IDFModel.load(modelPath)
>>> loadedModel.transform(df).head().idf == model.transform(df).head().idf
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of minDocFreq.
New in version 1.4.0.
Sets params for this IDF.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by IDF.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A Transformer that maps a column of indices back to a new column of corresponding string values. The index-string mapping is either from the ML attributes of the input column, or from user-supplied labels (which take precedence over ML attributes). See StringIndexer for converting strings into indices.
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this IndexToString.
New in version 1.6.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Rescale each feature individually to range [-1, 1] by dividing through the largest maximum absolute value in each feature. It does not shift/center the data, and thus does not destroy any sparsity.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([1.0]),), (Vectors.dense([2.0]),)], ["a"])
>>> maScaler = MaxAbsScaler(inputCol="a", outputCol="scaled")
>>> model = maScaler.fit(df)
>>> model.transform(df).show()
+-----+------+
| a|scaled|
+-----+------+
|[1.0]| [0.5]|
|[2.0]| [1.0]|
+-----+------+
...
>>> scalerPath = temp_path + "/max-abs-scaler"
>>> maScaler.save(scalerPath)
>>> loadedMAScaler = MaxAbsScaler.load(scalerPath)
>>> loadedMAScaler.getInputCol() == maScaler.getInputCol()
True
>>> loadedMAScaler.getOutputCol() == maScaler.getOutputCol()
True
>>> modelPath = temp_path + "/max-abs-scaler-model"
>>> model.save(modelPath)
>>> loadedModel = MaxAbsScalerModel.load(modelPath)
>>> loadedModel.maxAbs == model.maxAbs
True
New in version 2.0.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this MaxAbsScaler.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Model fitted by MaxAbsScaler.
New in version 2.0.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling. The rescaled value for feature E is calculated as,
Rescaled(e_i) = (e_i - E_min) / (E_max - E_min) * (max - min) + min
For the case E_max == E_min, Rescaled(e_i) = 0.5 * (max + min)
Note that since zero values will probably be transformed to non-zero values, output of the transformer will be DenseVector even for sparse input.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([0.0]),), (Vectors.dense([2.0]),)], ["a"])
>>> mmScaler = MinMaxScaler(inputCol="a", outputCol="scaled")
>>> model = mmScaler.fit(df)
>>> model.originalMin
DenseVector([0.0])
>>> model.originalMax
DenseVector([2.0])
>>> model.transform(df).show()
+-----+------+
| a|scaled|
+-----+------+
|[0.0]| [0.0]|
|[2.0]| [1.0]|
+-----+------+
...
>>> minMaxScalerPath = temp_path + "/min-max-scaler"
>>> mmScaler.save(minMaxScalerPath)
>>> loadedMMScaler = MinMaxScaler.load(minMaxScalerPath)
>>> loadedMMScaler.getMin() == mmScaler.getMin()
True
>>> loadedMMScaler.getMax() == mmScaler.getMax()
True
>>> modelPath = temp_path + "/min-max-scaler-model"
>>> model.save(modelPath)
>>> loadedModel = MinMaxScalerModel.load(modelPath)
>>> loadedModel.originalMin == model.originalMin
True
>>> loadedModel.originalMax == model.originalMax
True
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this MinMaxScaler.
New in version 1.6.0.
Returns an MLWriter instance for this ML instance.
Model fitted by MinMaxScaler.
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A feature transformer that converts the input array of strings into an array of n-grams. Null values in the input array are ignored. It returns an array of n-grams where each n-gram is represented by a space-separated string of words. When the input is empty, an empty array is returned. When the input array length is less than n (number of elements per n-gram), no n-grams are returned.
>>> df = spark.createDataFrame([Row(inputTokens=["a", "b", "c", "d", "e"])])
>>> ngram = NGram(n=2, inputCol="inputTokens", outputCol="nGrams")
>>> ngram.transform(df).head()
Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b', u'b c', u'c d', u'd e'])
>>> # Change n-gram length
>>> ngram.setParams(n=4).transform(df).head()
Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b c d', u'b c d e'])
>>> # Temporarily modify output column.
>>> ngram.transform(df, {ngram.outputCol: "output"}).head()
Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], output=[u'a b c d', u'b c d e'])
>>> ngram.transform(df).head()
Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b c d', u'b c d e'])
>>> # Must use keyword arguments to specify params.
>>> ngram.setParams("text")
Traceback (most recent call last):
...
TypeError: Method setParams forces keyword arguments.
>>> ngramPath = temp_path + "/ngram"
>>> ngram.save(ngramPath)
>>> loadedNGram = NGram.load(ngramPath)
>>> loadedNGram.getN() == ngram.getN()
True
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this NGram.
New in version 1.5.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Normalize a vector to have unit norm using the given p-norm.
>>> from pyspark.ml.linalg import Vectors
>>> svec = Vectors.sparse(4, {1: 4.0, 3: 3.0})
>>> df = spark.createDataFrame([(Vectors.dense([3.0, -4.0]), svec)], ["dense", "sparse"])
>>> normalizer = Normalizer(p=2.0, inputCol="dense", outputCol="features")
>>> normalizer.transform(df).head().features
DenseVector([0.6, -0.8])
>>> normalizer.setParams(inputCol="sparse", outputCol="freqs").transform(df).head().freqs
SparseVector(4, {1: 0.8, 3: 0.6})
>>> params = {normalizer.p: 1.0, normalizer.inputCol: "dense", normalizer.outputCol: "vector"}
>>> normalizer.transform(df, params).head().vector
DenseVector([0.4286, -0.5714])
>>> normalizerPath = temp_path + "/normalizer"
>>> normalizer.save(normalizerPath)
>>> loadedNormalizer = Normalizer.load(normalizerPath)
>>> loadedNormalizer.getP() == normalizer.getP()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this Normalizer.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index. For example with 5 categories, an input value of 2.0 would map to an output vector of [0.0, 0.0, 1.0, 0.0]. The last category is not included by default (configurable via dropLast) because it makes the vector entries sum up to one, and hence linearly dependent. So an input value of 4.0 maps to [0.0, 0.0, 0.0, 0.0]. Note that this is different from scikit-learn’s OneHotEncoder, which keeps all categories. The output vectors are sparse.
See also
StringIndexer for converting categorical values into category indices
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed")
>>> model = stringIndexer.fit(stringIndDf)
>>> td = model.transform(stringIndDf)
>>> encoder = OneHotEncoder(inputCol="indexed", outputCol="features")
>>> encoder.transform(td).head().features
SparseVector(2, {0: 1.0})
>>> encoder.setParams(outputCol="freqs").transform(td).head().freqs
SparseVector(2, {0: 1.0})
>>> params = {encoder.dropLast: False, encoder.outputCol: "test"}
>>> encoder.transform(td, params).head().test
SparseVector(3, {0: 1.0})
>>> onehotEncoderPath = temp_path + "/onehot-encoder"
>>> encoder.save(onehotEncoderPath)
>>> loadedEncoder = OneHotEncoder.load(onehotEncoderPath)
>>> loadedEncoder.getDropLast() == encoder.getDropLast()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this OneHotEncoder.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
PCA trains a model to project vectors to a lower dimensional space of the top k principal components.
>>> from pyspark.ml.linalg import Vectors
>>> data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),
... (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
... (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
>>> df = spark.createDataFrame(data,["features"])
>>> pca = PCA(k=2, inputCol="features", outputCol="pca_features")
>>> model = pca.fit(df)
>>> model.transform(df).collect()[0].pca_features
DenseVector([1.648..., -4.013...])
>>> model.explainedVariance
DenseVector([0.794..., 0.205...])
>>> pcaPath = temp_path + "/pca"
>>> pca.save(pcaPath)
>>> loadedPca = PCA.load(pcaPath)
>>> loadedPca.getK() == pca.getK()
True
>>> modelPath = temp_path + "/pca-model"
>>> model.save(modelPath)
>>> loadedModel = PCAModel.load(modelPath)
>>> loadedModel.pc == model.pc
True
>>> loadedModel.explainedVariance == model.explainedVariance
True
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Set params for this PCA.
New in version 1.5.0.
Returns an MLWriter instance for this ML instance.
Model fitted by PCA. Transforms vectors to a lower dimensional space.
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Returns a vector of proportions of variance explained by each principal component.
New in version 2.0.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns a principal components Matrix. Each column is one principal component.
New in version 2.0.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Perform feature expansion in a polynomial space. As said in wikipedia of Polynomial Expansion, “In mathematics, an expansion of a product of sums expresses it as a sum of products by using the fact that multiplication distributes over addition”. Take a 2-variable feature vector as an example: (x, y), if we want to expand it with degree 2, then we get (x, x * x, y, x * y, y * y).
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([0.5, 2.0]),)], ["dense"])
>>> px = PolynomialExpansion(degree=2, inputCol="dense", outputCol="expanded")
>>> px.transform(df).head().expanded
DenseVector([0.5, 0.25, 2.0, 1.0, 4.0])
>>> px.setParams(outputCol="test").transform(df).head().test
DenseVector([0.5, 0.25, 2.0, 1.0, 4.0])
>>> polyExpansionPath = temp_path + "/poly-expansion"
>>> px.save(polyExpansionPath)
>>> loadedPx = PolynomialExpansion.load(polyExpansionPath)
>>> loadedPx.getDegree() == px.getDegree()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this PolynomialExpansion.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features. The number of bins can be set using the numBuckets parameter. The bin ranges are chosen using an approximate algorithm (see the documentation for approxQuantile() for a detailed description). The precision of the approximation can be controlled with the relativeError parameter. The lower and upper bin bounds will be -Infinity and +Infinity, covering all real values.
>>> df = spark.createDataFrame([(0.1,), (0.4,), (1.2,), (1.5,)], ["values"])
>>> qds = QuantileDiscretizer(numBuckets=2,
... inputCol="values", outputCol="buckets", relativeError=0.01)
>>> qds.getRelativeError()
0.01
>>> bucketizer = qds.fit(df)
>>> splits = bucketizer.getSplits()
>>> splits[0]
-inf
>>> print("%2.1f" % round(splits[1], 1))
0.4
>>> bucketed = bucketizer.transform(df).head()
>>> bucketed.buckets
0.0
>>> quantileDiscretizerPath = temp_path + "/quantile-discretizer"
>>> qds.save(quantileDiscretizerPath)
>>> loadedQds = QuantileDiscretizer.load(quantileDiscretizerPath)
>>> loadedQds.getNumBuckets() == qds.getNumBuckets()
True
New in version 2.0.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Gets the value of relativeError or its default value.
New in version 2.0.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of numBuckets.
New in version 2.0.0.
Set the params for the QuantileDiscretizer
New in version 2.0.0.
Sets the value of relativeError.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
A regex based tokenizer that extracts tokens either by using the provided regex pattern (in Java dialect) to split the text (default) or repeatedly matching the regex (if gaps is false). Optional parameters also allow filtering tokens using a minimal length. It returns an array of strings that can be empty.
>>> df = spark.createDataFrame([("A B c",)], ["text"])
>>> reTokenizer = RegexTokenizer(inputCol="text", outputCol="words")
>>> reTokenizer.transform(df).head()
Row(text=u'A B c', words=[u'a', u'b', u'c'])
>>> # Change a parameter.
>>> reTokenizer.setParams(outputCol="tokens").transform(df).head()
Row(text=u'A B c', tokens=[u'a', u'b', u'c'])
>>> # Temporarily modify a parameter.
>>> reTokenizer.transform(df, {reTokenizer.outputCol: "words"}).head()
Row(text=u'A B c', words=[u'a', u'b', u'c'])
>>> reTokenizer.transform(df).head()
Row(text=u'A B c', tokens=[u'a', u'b', u'c'])
>>> # Must use keyword arguments to specify params.
>>> reTokenizer.setParams("text")
Traceback (most recent call last):
...
TypeError: Method setParams forces keyword arguments.
>>> regexTokenizerPath = temp_path + "/regex-tokenizer"
>>> reTokenizer.save(regexTokenizerPath)
>>> loadedReTokenizer = RegexTokenizer.load(regexTokenizerPath)
>>> loadedReTokenizer.getMinTokenLength() == reTokenizer.getMinTokenLength()
True
>>> loadedReTokenizer.getGaps() == reTokenizer.getGaps()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of minTokenLength or its default value.
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of minTokenLength.
New in version 1.4.0.
Sets params for this RegexTokenizer.
New in version 1.4.0.
Sets the value of toLowercase.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Implements the transforms required for fitting a dataset against an R model formula. Currently we support a limited subset of the R operators, including ‘~’, ‘.’, ‘:’, ‘+’, and ‘-‘. Also see the R formula docs.
>>> df = spark.createDataFrame([
... (1.0, 1.0, "a"),
... (0.0, 2.0, "b"),
... (0.0, 0.0, "a")
... ], ["y", "x", "s"])
>>> rf = RFormula(formula="y ~ x + s")
>>> model = rf.fit(df)
>>> model.transform(df).show()
+---+---+---+---------+-----+
| y| x| s| features|label|
+---+---+---+---------+-----+
|1.0|1.0| a|[1.0,1.0]| 1.0|
|0.0|2.0| b|[2.0,0.0]| 0.0|
|0.0|0.0| a|[0.0,1.0]| 0.0|
+---+---+---+---------+-----+
...
>>> rf.fit(df, {rf.formula: "y ~ . - s"}).transform(df).show()
+---+---+---+--------+-----+
| y| x| s|features|label|
+---+---+---+--------+-----+
|1.0|1.0| a| [1.0]| 1.0|
|0.0|2.0| b| [2.0]| 0.0|
|0.0|0.0| a| [0.0]| 0.0|
+---+---+---+--------+-----+
...
>>> rFormulaPath = temp_path + "/rFormula"
>>> rf.save(rFormulaPath)
>>> loadedRF = RFormula.load(rFormulaPath)
>>> loadedRF.getFormula() == rf.getFormula()
True
>>> loadedRF.getFeaturesCol() == rf.getFeaturesCol()
True
>>> loadedRF.getLabelCol() == rf.getLabelCol()
True
>>> str(loadedRF)
'RFormula(y ~ x + s) (uid=...)'
>>> modelPath = temp_path + "/rFormulaModel"
>>> model.save(modelPath)
>>> loadedModel = RFormulaModel.load(modelPath)
>>> loadedModel.uid == model.uid
True
>>> loadedModel.transform(df).show()
+---+---+---+---------+-----+
| y| x| s| features|label|
+---+---+---+---------+-----+
|1.0|1.0| a|[1.0,1.0]| 1.0|
|0.0|2.0| b|[2.0,0.0]| 0.0|
|0.0|0.0| a|[0.0,1.0]| 0.0|
+---+---+---+---------+-----+
...
>>> str(loadedModel)
'RFormulaModel(ResolvedRFormula(label=y, terms=[x,s], hasIntercept=true)) (uid=...)'
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of featuresCol or its default value.
Gets the value of labelCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of featuresCol.
Sets params for RFormula.
New in version 1.5.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Model fitted by RFormula. Fitting is required to determine the factor levels of formula terms.
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Implements the transforms which are defined by SQL statement. Currently we only support SQL syntax like ‘SELECT ... FROM __THIS__’ where ‘__THIS__’ represents the underlying table of the input dataset.
>>> df = spark.createDataFrame([(0, 1.0, 3.0), (2, 2.0, 5.0)], ["id", "v1", "v2"])
>>> sqlTrans = SQLTransformer(
... statement="SELECT *, (v1 + v2) AS v3, (v1 * v2) AS v4 FROM __THIS__")
>>> sqlTrans.transform(df).head()
Row(id=0, v1=1.0, v2=3.0, v3=4.0, v4=3.0)
>>> sqlTransformerPath = temp_path + "/sql-transformer"
>>> sqlTrans.save(sqlTransformerPath)
>>> loadedSqlTrans = SQLTransformer.load(sqlTransformerPath)
>>> loadedSqlTrans.getStatement() == sqlTrans.getStatement()
True
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
The “unit std” is computed using the corrected sample standard deviation, which is computed as the square root of the unbiased sample variance.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([0.0]),), (Vectors.dense([2.0]),)], ["a"])
>>> standardScaler = StandardScaler(inputCol="a", outputCol="scaled")
>>> model = standardScaler.fit(df)
>>> model.mean
DenseVector([1.0])
>>> model.std
DenseVector([1.4142])
>>> model.transform(df).collect()[1].scaled
DenseVector([1.4142])
>>> standardScalerPath = temp_path + "/standard-scaler"
>>> standardScaler.save(standardScalerPath)
>>> loadedStandardScaler = StandardScaler.load(standardScalerPath)
>>> loadedStandardScaler.getWithMean() == standardScaler.getWithMean()
True
>>> loadedStandardScaler.getWithStd() == standardScaler.getWithStd()
True
>>> modelPath = temp_path + "/standard-scaler-model"
>>> model.save(modelPath)
>>> loadedModel = StandardScalerModel.load(modelPath)
>>> loadedModel.std == model.std
True
>>> loadedModel.mean == model.mean
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this StandardScaler.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by StandardScaler.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A feature transformer that filters out stop words from input. Note: null values from input array are preserved unless adding null to stopWords explicitly.
>>> df = spark.createDataFrame([(["a", "b", "c"],)], ["text"])
>>> remover = StopWordsRemover(inputCol="text", outputCol="words", stopWords=["b"])
>>> remover.transform(df).head().words == ['a', 'c']
True
>>> stopWordsRemoverPath = temp_path + "/stopwords-remover"
>>> remover.save(stopWordsRemoverPath)
>>> loadedRemover = StopWordsRemover.load(stopWordsRemoverPath)
>>> loadedRemover.getStopWords() == remover.getStopWords()
True
>>> loadedRemover.getCaseSensitive() == remover.getCaseSensitive()
True
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of caseSensitive or its default value.
New in version 1.6.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Loads the default stop words for the given language. Supported languages: danish, dutch, english, finnish, french, german, hungarian, italian, norwegian, portuguese, russian, spanish, swedish, turkish
New in version 2.0.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of caseSensitive.
New in version 1.6.0.
Sets params for this StopWordRemover.
New in version 1.6.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A label indexer that maps a string column of labels to an ML column of label indices. If the input column is numeric, we cast it to string and index the string values. The indices are in [0, numLabels), ordered by label frequencies. So the most frequent label gets index 0.
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed", handleInvalid='error')
>>> model = stringIndexer.fit(stringIndDf)
>>> td = model.transform(stringIndDf)
>>> sorted(set([(i[0], i[1]) for i in td.select(td.id, td.indexed).collect()]),
... key=lambda x: x[0])
[(0, 0.0), (1, 2.0), (2, 1.0), (3, 0.0), (4, 0.0), (5, 1.0)]
>>> inverter = IndexToString(inputCol="indexed", outputCol="label2", labels=model.labels)
>>> itd = inverter.transform(td)
>>> sorted(set([(i[0], str(i[1])) for i in itd.select(itd.id, itd.label2).collect()]),
... key=lambda x: x[0])
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'a'), (4, 'a'), (5, 'c')]
>>> stringIndexerPath = temp_path + "/string-indexer"
>>> stringIndexer.save(stringIndexerPath)
>>> loadedIndexer = StringIndexer.load(stringIndexerPath)
>>> loadedIndexer.getHandleInvalid() == stringIndexer.getHandleInvalid()
True
>>> modelPath = temp_path + "/string-indexer-model"
>>> model.save(modelPath)
>>> loadedModel = StringIndexerModel.load(modelPath)
>>> loadedModel.labels == model.labels
True
>>> indexToStringPath = temp_path + "/index-to-string"
>>> inverter.save(indexToStringPath)
>>> loadedInverter = IndexToString.load(indexToStringPath)
>>> loadedInverter.getLabels() == inverter.getLabels()
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of handleInvalid or its default value.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of handleInvalid.
Sets params for this StringIndexer.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by StringIndexer.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Ordered list of labels, corresponding to indices to be assigned.
New in version 1.5.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A tokenizer that converts the input string to lowercase and then splits it by white spaces.
>>> df = spark.createDataFrame([("a b c",)], ["text"])
>>> tokenizer = Tokenizer(inputCol="text", outputCol="words")
>>> tokenizer.transform(df).head()
Row(text=u'a b c', words=[u'a', u'b', u'c'])
>>> # Change a parameter.
>>> tokenizer.setParams(outputCol="tokens").transform(df).head()
Row(text=u'a b c', tokens=[u'a', u'b', u'c'])
>>> # Temporarily modify a parameter.
>>> tokenizer.transform(df, {tokenizer.outputCol: "words"}).head()
Row(text=u'a b c', words=[u'a', u'b', u'c'])
>>> tokenizer.transform(df).head()
Row(text=u'a b c', tokens=[u'a', u'b', u'c'])
>>> # Must use keyword arguments to specify params.
>>> tokenizer.setParams("text")
Traceback (most recent call last):
...
TypeError: Method setParams forces keyword arguments.
>>> tokenizerPath = temp_path + "/tokenizer"
>>> tokenizer.save(tokenizerPath)
>>> loadedTokenizer = Tokenizer.load(tokenizerPath)
>>> loadedTokenizer.transform(df).head().tokens == tokenizer.transform(df).head().tokens
True
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this Tokenizer.
New in version 1.3.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
A feature transformer that merges multiple columns into a vector column.
>>> df = spark.createDataFrame([(1, 0, 3)], ["a", "b", "c"])
>>> vecAssembler = VectorAssembler(inputCols=["a", "b", "c"], outputCol="features")
>>> vecAssembler.transform(df).head().features
DenseVector([1.0, 0.0, 3.0])
>>> vecAssembler.setParams(outputCol="freqs").transform(df).head().freqs
DenseVector([1.0, 0.0, 3.0])
>>> params = {vecAssembler.inputCols: ["b", "a"], vecAssembler.outputCol: "vector"}
>>> vecAssembler.transform(df, params).head().vector
DenseVector([0.0, 1.0])
>>> vectorAssemblerPath = temp_path + "/vector-assembler"
>>> vecAssembler.save(vectorAssemblerPath)
>>> loadedAssembler = VectorAssembler.load(vectorAssemblerPath)
>>> loadedAssembler.transform(df).head().freqs == vecAssembler.transform(df).head().freqs
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCols or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets params for this VectorAssembler.
New in version 1.4.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Class for indexing categorical feature columns in a dataset of Vector.
- Automatically identify categorical features (default behavior)
- This helps process a dataset of unknown vectors into a dataset with some continuous features and some categorical features. The choice between continuous and categorical is based upon a maxCategories parameter.
- Set maxCategories to the maximum number of categorical any categorical feature should have.
- E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}. If maxCategories = 2, then feature 0 will be declared categorical and use indices {0, 1}, and feature 1 will be declared continuous.
- Index all features, if all features are categorical
- If maxCategories is set to be very large, then this will build an index of unique values for all features.
- Warning: This can cause problems if features are continuous since this will collect ALL unique values to the driver.
- E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}. If maxCategories >= 3, then both features will be declared categorical.
This returns a model which can transform categorical features to use 0-based indices.
- This is not guaranteed to choose the same category index across multiple runs.
- If a categorical feature includes value 0, then this is guaranteed to map value 0 to index 0. This maintains vector sparsity.
- More stability may be added in the future.
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([(Vectors.dense([-1.0, 0.0]),),
... (Vectors.dense([0.0, 1.0]),), (Vectors.dense([0.0, 2.0]),)], ["a"])
>>> indexer = VectorIndexer(maxCategories=2, inputCol="a", outputCol="indexed")
>>> model = indexer.fit(df)
>>> model.transform(df).head().indexed
DenseVector([1.0, 0.0])
>>> model.numFeatures
2
>>> model.categoryMaps
{0: {0.0: 0, -1.0: 1}}
>>> indexer.setParams(outputCol="test").fit(df).transform(df).collect()[1].test
DenseVector([0.0, 1.0])
>>> params = {indexer.maxCategories: 3, indexer.outputCol: "vector"}
>>> model2 = indexer.fit(df, params)
>>> model2.transform(df).head().vector
DenseVector([1.0, 0.0])
>>> vectorIndexerPath = temp_path + "/vector-indexer"
>>> indexer.save(vectorIndexerPath)
>>> loadedIndexer = VectorIndexer.load(vectorIndexerPath)
>>> loadedIndexer.getMaxCategories() == indexer.getMaxCategories()
True
>>> modelPath = temp_path + "/vector-indexer-model"
>>> model.save(modelPath)
>>> loadedModel = VectorIndexerModel.load(modelPath)
>>> loadedModel.numFeatures == model.numFeatures
True
>>> loadedModel.categoryMaps == model.categoryMaps
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of maxCategories or its default value.
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of maxCategories.
New in version 1.4.0.
Sets params for this VectorIndexer.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by VectorIndexer.
This also appends metadata to the output column, marking features as Numeric (continuous), Nominal (categorical), or Binary (either continuous or categorical). Non-ML metadata is not carried over from the input to the output column.
This maintains vector sparsity.
New in version 1.4.0.
Feature value index. Keys are categorical feature indices (column indices). Values are maps from original features values to 0-based category indices. If a feature is not in this map, it is treated as continuous.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Number of features, i.e., length of Vectors which this transforms.
New in version 1.4.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
The subset of features can be specified with either indices (setIndices()) or names (setNames()). At least one feature must be selected. Duplicate features are not allowed, so there can be no overlap between selected indices and names.
The output vector will order features with the selected indices first (in the order given), followed by the selected names (in the order given).
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([
... (Vectors.dense([-2.0, 2.3, 0.0, 0.0, 1.0]),),
... (Vectors.dense([0.0, 0.0, 0.0, 0.0, 0.0]),),
... (Vectors.dense([0.6, -1.1, -3.0, 4.5, 3.3]),)], ["features"])
>>> vs = VectorSlicer(inputCol="features", outputCol="sliced", indices=[1, 4])
>>> vs.transform(df).head().sliced
DenseVector([2.3, 1.0])
>>> vectorSlicerPath = temp_path + "/vector-slicer"
>>> vs.save(vectorSlicerPath)
>>> loadedVs = VectorSlicer.load(vectorSlicerPath)
>>> loadedVs.getIndices() == vs.getIndices()
True
>>> loadedVs.getNames() == vs.getNames()
True
New in version 1.6.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of inputCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
setParams(self, inputCol=None, outputCol=None, indices=None, names=None): Sets params for this VectorSlicer.
New in version 1.6.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Word2Vec trains a model of Map(String, Vector), i.e. transforms a word into a code for further natural language processing or machine learning process.
>>> sent = ("a b " * 100 + "a c " * 10).split(" ")
>>> doc = spark.createDataFrame([(sent,), (sent,)], ["sentence"])
>>> word2Vec = Word2Vec(vectorSize=5, seed=42, inputCol="sentence", outputCol="model")
>>> model = word2Vec.fit(doc)
>>> model.getVectors().show()
+----+--------------------+
|word| vector|
+----+--------------------+
| a|[0.09461779892444...|
| b|[1.15474212169647...|
| c|[-0.3794820010662...|
+----+--------------------+
...
>>> from pyspark.sql.functions import format_number as fmt
>>> model.findSynonyms("a", 2).select("word", fmt("similarity", 5).alias("similarity")).show()
+----+----------+
|word|similarity|
+----+----------+
| b| 0.25053|
| c| -0.69805|
+----+----------+
...
>>> model.transform(doc).head().model
DenseVector([0.5524, -0.4995, -0.3599, 0.0241, 0.3461])
>>> word2vecPath = temp_path + "/word2vec"
>>> word2Vec.save(word2vecPath)
>>> loadedWord2Vec = Word2Vec.load(word2vecPath)
>>> loadedWord2Vec.getVectorSize() == word2Vec.getVectorSize()
True
>>> loadedWord2Vec.getNumPartitions() == word2Vec.getNumPartitions()
True
>>> loadedWord2Vec.getMinCount() == word2Vec.getMinCount()
True
>>> modelPath = temp_path + "/word2vec-model"
>>> model.save(modelPath)
>>> loadedModel = Word2VecModel.load(modelPath)
>>> loadedModel.getVectors().first().word == model.getVectors().first().word
True
>>> loadedModel.getVectors().first().vector == model.getVectors().first().vector
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of inputCol or its default value.
Gets the value of maxIter or its default value.
Gets the value of maxSentenceLength or its default value.
New in version 2.0.0.
Gets the value of numPartitions or its default value.
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets the value of outputCol or its default value.
Gets a param by its name.
New in version 1.4.0.
Gets the value of seed or its default value.
Gets the value of stepSize or its default value.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of maxSentenceLength.
New in version 2.0.0.
Sets the value of numPartitions.
New in version 1.4.0.
Sets params for this Word2Vec.
New in version 1.4.0.
Sets the value of vectorSize.
New in version 1.4.0.
Sets the value of windowSize.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
Model fitted by Word2Vec.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Find “num” number of words closest in similarity to “word”. word can be a string or vector representation. Returns a dataframe with two fields word and similarity (which gives the cosine similarity).
New in version 1.5.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Returns the vector representation of the words as a dataframe with two fields, word and vector.
New in version 1.5.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Logistic regression. Currently, this class only supports binary classification.
>>> from pyspark.sql import Row
>>> from pyspark.ml.linalg import Vectors
>>> df = sc.parallelize([
... Row(label=1.0, weight=2.0, features=Vectors.dense(1.0)),
... Row(label=0.0, weight=2.0, features=Vectors.sparse(1, [], []))]).toDF()
>>> lr = LogisticRegression(maxIter=5, regParam=0.01, weightCol="weight")
>>> model = lr.fit(df)
>>> model.coefficients
DenseVector([5.5...])
>>> model.intercept
-2.68...
>>> test0 = sc.parallelize([Row(features=Vectors.dense(-1.0))]).toDF()
>>> result = model.transform(test0).head()
>>> result.prediction
0.0
>>> result.probability
DenseVector([0.99..., 0.00...])
>>> result.rawPrediction
DenseVector([8.22..., -8.22...])
>>> test1 = sc.parallelize([Row(features=Vectors.sparse(1, [0], [1.0]))]).toDF()
>>> model.transform(test1).head().prediction
1.0
>>> lr.setParams("vector")
Traceback (most recent call last):
...
TypeError: Method setParams forces keyword arguments.
>>> lr_path = temp_path + "/lr"
>>> lr.save(lr_path)
>>> lr2 = LogisticRegression.load(lr_path)
>>> lr2.getMaxIter()
5
>>> model_path = temp_path + "/lr_model"
>>> model.save(model_path)
>>> model2 = LogisticRegressionModel.load(model_path)
>>> model.coefficients[0] == model2.coefficients[0]
True
>>> model.intercept == model2.intercept
True
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of elasticNetParam or its default value.
Gets the value of featuresCol or its default value.
Gets the value of fitIntercept or its default value.
Gets the value of labelCol or its default value.
Gets the value of maxIter or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Gets the value of predictionCol or its default value.
Gets the value of probabilityCol or its default value.
Gets the value of rawPredictionCol or its default value.
Gets the value of regParam or its default value.
Gets the value of standardization or its default value.
Get threshold for binary classification.
If thresholds is set with length 2 (i.e., binary classification), this returns the equivalent threshold: \(\frac{1}{1 + \frac{thresholds(0)}{thresholds(1)}}\). Otherwise, returns threshold if set or its default value if unset.
New in version 1.4.0.
If thresholds is set, return its value. Otherwise, if threshold is set, return the equivalent thresholds for binary classification: (1-threshold, threshold). If neither are set, throw an error.
New in version 1.5.0.
Gets the value of tol or its default value.
Gets the value of weightCol or its default value.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of elasticNetParam.
Sets the value of featuresCol.
Sets the value of fitIntercept.
Sets params for logistic regression. If the threshold and thresholds Params are both set, they must be equivalent.
New in version 1.3.0.
Sets the value of predictionCol.
Sets the value of probabilityCol.
Sets the value of rawPredictionCol.
Sets the value of standardization.
Sets the value of threshold. Clears value of thresholds if it has been set.
New in version 1.4.0.
Sets the value of thresholds. Clears value of threshold if it has been set.
New in version 1.5.0.
Returns an MLWriter instance for this ML instance.
Model fitted by LogisticRegression.
New in version 1.3.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Evaluates the model on a test dataset.
| Parameters: | dataset – Test dataset to evaluate model on, where dataset is an instance of pyspark.sql.DataFrame |
|---|
New in version 2.0.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Indicates whether a training summary exists for this model instance.
New in version 2.0.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Gets summary (e.g. residuals, mse, r-squared ) of model on training set. An exception is thrown if trainingSummary is None.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Note
Experimental
Abstraction for Logistic Regression Results for a given model.
New in version 2.0.0.
Field in “predictions” which gives the features of each instance as a vector.
New in version 2.0.0.
Note
Experimental
Abstraction for multinomial Logistic Regression Training results. Currently, the training summary ignores the training weights except for the objective trace.
New in version 2.0.0.
Field in “predictions” which gives the features of each instance as a vector.
New in version 2.0.0.
Field in “predictions” which gives the true label of each instance.
New in version 2.0.0.
Objective function (scaled loss + regularization) at each iteration.
New in version 2.0.0.
Dataframe outputted by the model’s transform method.
New in version 2.0.0.
Field in “predictions” which gives the probability of each class as a vector.
New in version 2.0.0.
Note
Experimental
Binary Logistic regression results for a given model.
New in version 2.0.0.
Computes the area under the receiver operating characteristic (ROC) curve.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, F-Measure) curve with beta = 1.0.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Field in “predictions” which gives the features of each instance as a vector.
New in version 2.0.0.
Field in “predictions” which gives the true label of each instance.
New in version 2.0.0.
Returns the precision-recall curve, which is a Dataframe containing two fields recall, precision with (0.0, 1.0) prepended to it.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, precision) curve. Every possible probability obtained in transforming the dataset are used as thresholds used in calculating the precision.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Dataframe outputted by the model’s transform method.
New in version 2.0.0.
Field in “predictions” which gives the probability of each class as a vector.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, recall) curve. Every possible probability obtained in transforming the dataset are used as thresholds used in calculating the recall.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns the receiver operating characteristic (ROC) curve, which is a Dataframe having two fields (FPR, TPR) with (0.0, 0.0) prepended and (1.0, 1.0) appended to it.
See also
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Note
Experimental
Binary Logistic regression training results for a given model.
New in version 2.0.0.
Computes the area under the receiver operating characteristic (ROC) curve.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, F-Measure) curve with beta = 1.0.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Field in “predictions” which gives the features of each instance as a vector.
New in version 2.0.0.
Field in “predictions” which gives the true label of each instance.
New in version 2.0.0.
Objective function (scaled loss + regularization) at each iteration.
New in version 2.0.0.
Returns the precision-recall curve, which is a Dataframe containing two fields recall, precision with (0.0, 1.0) prepended to it.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, precision) curve. Every possible probability obtained in transforming the dataset are used as thresholds used in calculating the precision.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Dataframe outputted by the model’s transform method.
New in version 2.0.0.
Field in “predictions” which gives the probability of each class as a vector.
New in version 2.0.0.
Returns a dataframe with two fields (threshold, recall) curve. Every possible probability obtained in transforming the dataset are used as thresholds used in calculating the recall.
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Returns the receiver operating characteristic (ROC) curve, which is a Dataframe having two fields (FPR, TPR) with (0.0, 0.0) prepended and (1.0, 1.0) appended to it.
See also
Note: This ignores instance weights (setting all to 1.0) from LogisticRegression.weightCol. This will change in later Spark versions.
New in version 2.0.0.
Number of training iterations until termination.
New in version 2.0.0.
Decision tree learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.
>>> from pyspark.ml.linalg import Vectors
>>> from pyspark.ml.feature import StringIndexer
>>> df = spark.createDataFrame([
... (1.0, Vectors.dense(1.0)),
... (0.0, Vectors.sparse(1, [], []))], ["label", "features"])
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed")
>>> si_model = stringIndexer.fit(df)
>>> td = si_model.transform(df)
>>> dt = DecisionTreeClassifier(maxDepth=2, labelCol="indexed")
>>> model = dt.fit(td)
>>> model.numNodes
3
>>> model.depth
1
>>> model.featureImportances
SparseVector(1, {0: 1.0})
>>> print(model.toDebugString)
DecisionTreeClassificationModel (uid=...) of depth 1 with 3 nodes...
>>> test0 = spark.createDataFrame([(Vectors.dense(-1.0),)], ["features"])
>>> result = model.transform(test0).head()
>>> result.prediction
0.0
>>> result.probability
DenseVector([1.0, 0.0])
>>> result.rawPrediction
DenseVector([1.0, 0.0])
>>> test1 = spark.createDataFrame([(Vectors.sparse(1, [0], [1.0]),)], ["features"])
>>> model.transform(test1).head().prediction
1.0
>>> dtc_path = temp_path + "/dtc"
>>> dt.save(dtc_path)
>>> dt2 = DecisionTreeClassifier.load(dtc_path)
>>> dt2.getMaxDepth()
2
>>> model_path = temp_path + "/dtc_model"
>>> model.save(model_path)
>>> model2 = DecisionTreeClassificationModel.load(model_path)
>>> model.featureImportances == model2.featureImportances
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of cacheNodeIds or its default value.
Gets the value of checkpointInterval or its default value.
Gets the value of featuresCol or its default value.
Gets the value of impurity or its default value.
New in version 1.6.0.
Gets the value of labelCol or its default value.
Gets the value of maxBins or its default value.
Gets the value of maxDepth or its default value.
Gets the value of maxMemoryInMB or its default value.
Gets the value of minInfoGain or its default value.
Gets the value of minInstancesPerNode or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Gets the value of predictionCol or its default value.
Gets the value of probabilityCol or its default value.
Gets the value of rawPredictionCol or its default value.
Gets the value of seed or its default value.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of cacheNodeIds.
Sets the value of checkpointInterval.
Sets the value of featuresCol.
Sets the value of maxMemoryInMB.
Sets the value of minInfoGain.
Sets the value of minInstancesPerNode.
Sets params for the DecisionTreeClassifier.
New in version 1.4.0.
Sets the value of predictionCol.
Sets the value of probabilityCol.
Sets the value of rawPredictionCol.
Returns an MLWriter instance for this ML instance.
Model fitted by DecisionTreeClassifier.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Return depth of the decision tree.
New in version 1.5.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Estimate of the importance of each feature.
This generalizes the idea of “Gini” importance to other losses, following the explanation of Gini importance from “Random Forests” documentation by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
New in version 2.0.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Return number of nodes of the decision tree.
New in version 1.5.0.
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Full description of model.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Returns an MLWriter instance for this ML instance.
Gradient-Boosted Trees (GBTs) learning algorithm for classification. It supports binary labels, as well as both continuous and categorical features. Note: Multiclass labels are not currently supported.
The implementation is based upon: J.H. Friedman. “Stochastic Gradient Boosting.” 1999.
Notes on Gradient Boosting vs. TreeBoost: - This implementation is for Stochastic Gradient Boosting, not for TreeBoost. - Both algorithms learn tree ensembles by minimizing loss functions. - TreeBoost (Friedman, 1999) additionally modifies the outputs at tree leaf nodes based on the loss function, whereas the original gradient boosting method does not. - We expect to implement TreeBoost in the future: SPARK-4240
>>> from numpy import allclose
>>> from pyspark.ml.linalg import Vectors
>>> from pyspark.ml.feature import StringIndexer
>>> df = spark.createDataFrame([
... (1.0, Vectors.dense(1.0)),
... (0.0, Vectors.sparse(1, [], []))], ["label", "features"])
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed")
>>> si_model = stringIndexer.fit(df)
>>> td = si_model.transform(df)
>>> gbt = GBTClassifier(maxIter=5, maxDepth=2, labelCol="indexed", seed=42)
>>> model = gbt.fit(td)
>>> model.featureImportances
SparseVector(1, {0: 1.0})
>>> allclose(model.treeWeights, [1.0, 0.1, 0.1, 0.1, 0.1])
True
>>> test0 = spark.createDataFrame([(Vectors.dense(-1.0),)], ["features"])
>>> model.transform(test0).head().prediction
0.0
>>> test1 = spark.createDataFrame([(Vectors.sparse(1, [0], [1.0]),)], ["features"])
>>> model.transform(test1).head().prediction
1.0
>>> model.totalNumNodes
15
>>> print(model.toDebugString)
GBTClassificationModel (uid=...)...with 5 trees...
>>> gbtc_path = temp_path + "gbtc"
>>> gbt.save(gbtc_path)
>>> gbt2 = GBTClassifier.load(gbtc_path)
>>> gbt2.getMaxDepth()
2
>>> model_path = temp_path + "gbtc_model"
>>> model.save(model_path)
>>> model2 = GBTClassificationModel.load(model_path)
>>> model.featureImportances == model2.featureImportances
True
>>> model.treeWeights == model2.treeWeights
True
>>> model.trees
[DecisionTreeRegressionModel (uid=...) of depth..., DecisionTreeRegressionModel...]
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of cacheNodeIds or its default value.
Gets the value of checkpointInterval or its default value.
Gets the value of featuresCol or its default value.
Gets the value of labelCol or its default value.
Gets the value of maxBins or its default value.
Gets the value of maxDepth or its default value.
Gets the value of maxIter or its default value.
Gets the value of maxMemoryInMB or its default value.
Gets the value of minInfoGain or its default value.
Gets the value of minInstancesPerNode or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Gets the value of predictionCol or its default value.
Gets the value of seed or its default value.
Gets the value of stepSize or its default value.
Gets the value of subsamplingRate or its default value.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of cacheNodeIds.
Sets the value of checkpointInterval.
Sets the value of featuresCol.
Sets the value of maxMemoryInMB.
Sets the value of minInfoGain.
Sets the value of minInstancesPerNode.
Sets params for Gradient Boosted Tree Classification.
New in version 1.4.0.
Sets the value of predictionCol.
Sets the value of subsamplingRate.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by GBTClassifier.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Estimate of the importance of each feature.
Each feature’s importance is the average of its importance across all trees in the ensemble The importance vector is normalized to sum to 1. This method is suggested by Hastie et al. (Hastie, Tibshirani, Friedman. “The Elements of Statistical Learning, 2nd Edition.” 2001.) and follows the implementation from scikit-learn.
New in version 2.0.0.
Number of trees in ensemble.
New in version 2.0.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Full description of model.
New in version 2.0.0.
Total number of nodes, summed over all trees in the ensemble.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Return the weights for each tree
New in version 1.5.0.
Trees in this ensemble. Warning: These have null parent Estimators.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
Random Forest learning algorithm for classification. It supports both binary and multiclass labels, as well as both continuous and categorical features.
>>> import numpy
>>> from numpy import allclose
>>> from pyspark.ml.linalg import Vectors
>>> from pyspark.ml.feature import StringIndexer
>>> df = spark.createDataFrame([
... (1.0, Vectors.dense(1.0)),
... (0.0, Vectors.sparse(1, [], []))], ["label", "features"])
>>> stringIndexer = StringIndexer(inputCol="label", outputCol="indexed")
>>> si_model = stringIndexer.fit(df)
>>> td = si_model.transform(df)
>>> rf = RandomForestClassifier(numTrees=3, maxDepth=2, labelCol="indexed", seed=42)
>>> model = rf.fit(td)
>>> model.featureImportances
SparseVector(1, {0: 1.0})
>>> allclose(model.treeWeights, [1.0, 1.0, 1.0])
True
>>> test0 = spark.createDataFrame([(Vectors.dense(-1.0),)], ["features"])
>>> result = model.transform(test0).head()
>>> result.prediction
0.0
>>> numpy.argmax(result.probability)
0
>>> numpy.argmax(result.rawPrediction)
0
>>> test1 = spark.createDataFrame([(Vectors.sparse(1, [0], [1.0]),)], ["features"])
>>> model.transform(test1).head().prediction
1.0
>>> model.trees
[DecisionTreeClassificationModel (uid=...) of depth..., DecisionTreeClassificationModel...]
>>> rfc_path = temp_path + "/rfc"
>>> rf.save(rfc_path)
>>> rf2 = RandomForestClassifier.load(rfc_path)
>>> rf2.getNumTrees()
3
>>> model_path = temp_path + "/rfc_model"
>>> model.save(model_path)
>>> model2 = RandomForestClassificationModel.load(model_path)
>>> model.featureImportances == model2.featureImportances
True
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of cacheNodeIds or its default value.
Gets the value of checkpointInterval or its default value.
Gets the value of featureSubsetStrategy or its default value.
New in version 1.4.0.
Gets the value of featuresCol or its default value.
Gets the value of impurity or its default value.
New in version 1.6.0.
Gets the value of labelCol or its default value.
Gets the value of maxBins or its default value.
Gets the value of maxDepth or its default value.
Gets the value of maxMemoryInMB or its default value.
Gets the value of minInfoGain or its default value.
Gets the value of minInstancesPerNode or its default value.
Gets the value of numTrees or its default value.
New in version 1.4.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Gets the value of predictionCol or its default value.
Gets the value of probabilityCol or its default value.
Gets the value of rawPredictionCol or its default value.
Gets the value of seed or its default value.
Gets the value of subsamplingRate or its default value.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Sets the value of cacheNodeIds.
Sets the value of checkpointInterval.
Sets the value of featureSubsetStrategy.
New in version 1.4.0.
Sets the value of featuresCol.
Sets the value of maxMemoryInMB.
Sets the value of minInfoGain.
Sets the value of minInstancesPerNode.
Sets params for linear classification.
New in version 1.4.0.
Sets the value of predictionCol.
Sets the value of probabilityCol.
Sets the value of rawPredictionCol.
Sets the value of subsamplingRate.
New in version 1.4.0.
Returns an MLWriter instance for this ML instance.
Model fitted by RandomForestClassifier.
New in version 1.4.0.
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java model with extra params. So both the Python wrapper and the Java model get copied.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Estimate of the importance of each feature.
Each feature’s importance is the average of its importance across all trees in the ensemble The importance vector is normalized to sum to 1. This method is suggested by Hastie et al. (Hastie, Tibshirani, Friedman. “The Elements of Statistical Learning, 2nd Edition.” 2001.) and follows the implementation from scikit-learn.
New in version 2.0.0.
Number of trees in ensemble.
New in version 2.0.0.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.
Checks whether a param is explicitly set by user.
New in version 1.4.0.
Reads an ML instance from the input path, a shortcut of read().load(path).
Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.
New in version 1.3.0.
Returns an MLReader instance for this class.
Save this ML instance to the given path, a shortcut of write().save(path).
Full description of model.
New in version 2.0.0.
Total number of nodes, summed over all trees in the ensemble.
New in version 2.0.0.
Transforms the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | transformed dataset |
New in version 1.3.0.
Return the weights for each tree
New in version 1.5.0.
Trees in this ensemble. Warning: These have null parent Estimators.
New in version 2.0.0.
Returns an MLWriter instance for this ML instance.
Naive Bayes Classifiers. It supports both Multinomial and Bernoulli NB. Multinomial NB can handle finitely supported discrete data. For example, by converting documents into TF-IDF vectors, it can be used for document classification. By making every vector a binary (0/1) data, it can also be used as Bernoulli NB. The input feature values must be nonnegative.
>>> from pyspark.sql import Row
>>> from pyspark.ml.linalg import Vectors
>>> df = spark.createDataFrame([
... Row(label=0.0, features=Vectors.dense([0.0, 0.0])),
... Row(label=0.0, features=Vectors.dense([0.0, 1.0])),
... Row(label=1.0, features=Vectors.dense([1.0, 0.0]))])
>>> nb = NaiveBayes(smoothing=1.0, modelType="multinomial")
>>> model = nb.fit(df)
>>> model.pi
DenseVector([-0.51..., -0.91...])
>>> model.theta
DenseMatrix(2, 2, [-1.09..., -0.40..., -0.40..., -1.09...], 1)
>>> test0 = sc.parallelize([Row(features=Vectors.dense([1.0, 0.0]))]).toDF()
>>> result = model.transform(test0).head()
>>> result.prediction
1.0
>>> result.probability
DenseVector([0.42..., 0.57...])
>>> result.rawPrediction
DenseVector([-1.60..., -1.32...])
>>> test1 = sc.parallelize([Row(features=Vectors.sparse(2, [0], [1.0]))]).toDF()
>>> model.transform(test1).head().prediction
1.0
>>> nb_path = temp_path + "/nb"
>>> nb.save(nb_path)
>>> nb2 = NaiveBayes.load(nb_path)
>>> nb2.getSmoothing()
1.0
>>> model_path = temp_path + "/nb_model"
>>> model.save(model_path)
>>> model2 = NaiveBayesModel.load(model_path)
>>> model.pi == model2.pi
True
>>> model.theta == model2.theta
True
>>> nb = nb.setThresholds([0.01, 10.00])
>>> model3 = nb.fit(df)
>>> result = model3.transform(test0).head()
>>> result.prediction
0.0
New in version 1.5.0.
Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.
| Parameters: | extra – Extra parameters to copy to the new instance |
|---|---|
| Returns: | Copy of this instance |
New in version 1.4.0.
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
New in version 1.4.0.
Returns the documentation of all params with their optionally default values and user-supplied values.
New in version 1.4.0.
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
| Parameters: | extra – extra param values |
|---|---|
| Returns: | merged param map |
New in version 1.4.0.
Fits a model to the input dataset with optional parameters.
| Parameters: |
|
|---|---|
| Returns: | fitted model(s) |
New in version 1.3.0.
Gets the value of featuresCol or its default value.
Gets the value of labelCol or its default value.
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
New in version 1.4.0.
Gets a param by its name.
New in version 1.4.0.
Gets the value of predictionCol or its default value.
Gets the value of probabilityCol or its default value.
Gets the value of rawPredictionCol or its default value.
Gets the value of thresholds or its default value.
Checks whether a param has a default value.
New in version 1.4.0.
Tests whether this instance contains a param with a given (string) name.
New in version 1.4.0.
Checks whether a param is explicitly set by user or has a default value.
New in version 1.4.0.