
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/model_selection/plot_precision_recall.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_model_selection_plot_precision_recall.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_model_selection_plot_precision_recall.py:


================
Precision-Recall
================

Example of Precision-Recall metric to evaluate classifier output quality.

Precision-Recall is a useful measure of success of prediction when the
classes are very imbalanced. In information retrieval, precision is a
measure of result relevancy, while recall is a measure of how many truly
relevant results are returned.

The precision-recall curve shows the tradeoff between precision and
recall for different threshold. A high area under the curve represents
both high recall and high precision, where high precision relates to a
low false positive rate, and high recall relates to a low false negative
rate. High scores for both show that the classifier is returning accurate
results (high precision), as well as returning a majority of all positive
results (high recall).

A system with high recall but low precision returns many results, but most of
its predicted labels are incorrect when compared to the training labels. A
system with high precision but low recall is just the opposite, returning very
few results, but most of its predicted labels are correct when compared to the
training labels. An ideal system with high precision and high recall will
return many results, with all results labeled correctly.

Precision (:math:`P`) is defined as the number of true positives (:math:`T_p`)
over the number of true positives plus the number of false positives
(:math:`F_p`).

:math:`P = \frac{T_p}{T_p+F_p}`

Recall (:math:`R`) is defined as the number of true positives (:math:`T_p`)
over the number of true positives plus the number of false negatives
(:math:`F_n`).

:math:`R = \frac{T_p}{T_p + F_n}`

These quantities are also related to the (:math:`F_1`) score, which is defined
as the harmonic mean of precision and recall.

:math:`F1 = 2\frac{P \times R}{P+R}`

Note that the precision may not decrease with recall. The
definition of precision (:math:`\frac{T_p}{T_p + F_p}`) shows that lowering
the threshold of a classifier may increase the denominator, by increasing the
number of results returned. If the threshold was previously set too high, the
new results may all be true positives, which will increase precision. If the
previous threshold was about right or too low, further lowering the threshold
will introduce false positives, decreasing precision.

Recall is defined as :math:`\frac{T_p}{T_p+F_n}`, where :math:`T_p+F_n` does
not depend on the classifier threshold. This means that lowering the classifier
threshold may increase recall, by increasing the number of true positive
results. It is also possible that lowering the threshold may leave recall
unchanged, while the precision fluctuates.

The relationship between recall and precision can be observed in the
stairstep area of the plot - at the edges of these steps a small change
in the threshold considerably reduces precision, with only a minor gain in
recall.

**Average precision** (AP) summarizes such a plot as the weighted mean of
precisions achieved at each threshold, with the increase in recall from the
previous threshold used as the weight:

:math:`\text{AP} = \sum_n (R_n - R_{n-1}) P_n`

where :math:`P_n` and :math:`R_n` are the precision and recall at the
nth threshold. A pair :math:`(R_k, P_k)` is referred to as an
*operating point*.

AP and the trapezoidal area under the operating points
(:func:`sklearn.metrics.auc`) are common ways to summarize a precision-recall
curve that lead to different results. Read more in the
:ref:`User Guide <precision_recall_f_measure_metrics>`.

Precision-recall curves are typically used in binary classification to study
the output of a classifier. In order to extend the precision-recall curve and
average precision to multi-class or multi-label classification, it is necessary
to binarize the output. One curve can be drawn per label, but one can also draw
a precision-recall curve by considering each element of the label indicator
matrix as a binary prediction (micro-averaging).

.. note::

    See also :func:`sklearn.metrics.average_precision_score`,
             :func:`sklearn.metrics.recall_score`,
             :func:`sklearn.metrics.precision_score`,
             :func:`sklearn.metrics.f1_score`

.. GENERATED FROM PYTHON SOURCE LINES 95-102

In binary classification settings
---------------------------------

Dataset and model
.................

We will use a Linear SVC classifier to differentiate two types of irises.

.. GENERATED FROM PYTHON SOURCE LINES 102-119

.. code-block:: default

    import numpy as np

    from sklearn.datasets import load_iris
    from sklearn.model_selection import train_test_split

    X, y = load_iris(return_X_y=True)

    # Add noisy features
    random_state = np.random.RandomState(0)
    n_samples, n_features = X.shape
    X = np.concatenate([X, random_state.randn(n_samples, 200 * n_features)], axis=1)

    # Limit to the two first classes, and split into training and test
    X_train, X_test, y_train, y_test = train_test_split(
        X[y < 2], y[y < 2], test_size=0.5, random_state=random_state
    )








.. GENERATED FROM PYTHON SOURCE LINES 120-123

Linear SVC will expect each feature to have a similar range of values. Thus,
we will first scale the data using a
:class:`~sklearn.preprocessing.StandardScaler`.

.. GENERATED FROM PYTHON SOURCE LINES 123-132

.. code-block:: default

    from sklearn.pipeline import make_pipeline
    from sklearn.preprocessing import StandardScaler
    from sklearn.svm import LinearSVC

    classifier = make_pipeline(
        StandardScaler(), LinearSVC(random_state=random_state, dual="auto")
    )
    classifier.fit(X_train, y_train)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style>#sk-container-id-55 {color: black;}#sk-container-id-55 pre{padding: 0;}#sk-container-id-55 div.sk-toggleable {background-color: white;}#sk-container-id-55 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-55 label.sk-toggleable__label-arrow:before {content: "▸";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-55 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-55 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-55 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-55 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-55 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-55 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: "▾";}#sk-container-id-55 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-55 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-55 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-55 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-55 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-55 div.sk-parallel-item::after {content: "";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-55 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-55 div.sk-serial::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-55 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-55 div.sk-item {position: relative;z-index: 1;}#sk-container-id-55 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-55 div.sk-item::before, #sk-container-id-55 div.sk-parallel-item::before {content: "";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-55 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-55 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-55 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-55 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-55 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-55 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-55 div.sk-label-container {text-align: center;}#sk-container-id-55 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-55 div.sk-text-repr-fallback {display: none;}</style><div id="sk-container-id-55" class="sk-top-container"><div class="sk-text-repr-fallback"><pre>Pipeline(steps=[(&#x27;standardscaler&#x27;, StandardScaler()),
                    (&#x27;linearsvc&#x27;,
                     LinearSVC(dual=&#x27;auto&#x27;,
                               random_state=RandomState(MT19937) at 0x7FCA209B6040))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class="sk-container" hidden><div class="sk-item sk-dashed-wrapped"><div class="sk-label-container"><div class="sk-label sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-235" type="checkbox" ><label for="sk-estimator-id-235" class="sk-toggleable__label sk-toggleable__label-arrow">Pipeline</label><div class="sk-toggleable__content"><pre>Pipeline(steps=[(&#x27;standardscaler&#x27;, StandardScaler()),
                    (&#x27;linearsvc&#x27;,
                     LinearSVC(dual=&#x27;auto&#x27;,
                               random_state=RandomState(MT19937) at 0x7FCA209B6040))])</pre></div></div></div><div class="sk-serial"><div class="sk-item"><div class="sk-estimator sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-236" type="checkbox" ><label for="sk-estimator-id-236" class="sk-toggleable__label sk-toggleable__label-arrow">StandardScaler</label><div class="sk-toggleable__content"><pre>StandardScaler()</pre></div></div></div><div class="sk-item"><div class="sk-estimator sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually" id="sk-estimator-id-237" type="checkbox" ><label for="sk-estimator-id-237" class="sk-toggleable__label sk-toggleable__label-arrow">LinearSVC</label><div class="sk-toggleable__content"><pre>LinearSVC(dual=&#x27;auto&#x27;, random_state=RandomState(MT19937) at 0x7FCA209B6040)</pre></div></div></div></div></div></div></div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 133-145

Plot the Precision-Recall curve
...............................

To plot the precision-recall curve, you should use
:class:`~sklearn.metrics.PrecisionRecallDisplay`. Indeed, there is two
methods available depending if you already computed the predictions of the
classifier or not.

Let's first plot the precision-recall curve without the classifier
predictions. We use
:func:`~sklearn.metrics.PrecisionRecallDisplay.from_estimator` that
computes the predictions for us before plotting the curve.

.. GENERATED FROM PYTHON SOURCE LINES 145-152

.. code-block:: default

    from sklearn.metrics import PrecisionRecallDisplay

    display = PrecisionRecallDisplay.from_estimator(
        classifier, X_test, y_test, name="LinearSVC", plot_chance_level=True
    )
    _ = display.ax_.set_title("2-class Precision-Recall curve")




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_001.png
   :alt: 2-class Precision-Recall curve
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 153-156

If we already got the estimated probabilities or scores for
our model, then we can use
:func:`~sklearn.metrics.PrecisionRecallDisplay.from_predictions`.

.. GENERATED FROM PYTHON SOURCE LINES 156-163

.. code-block:: default

    y_score = classifier.decision_function(X_test)

    display = PrecisionRecallDisplay.from_predictions(
        y_test, y_score, name="LinearSVC", plot_chance_level=True
    )
    _ = display.ax_.set_title("2-class Precision-Recall curve")




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_002.png
   :alt: 2-class Precision-Recall curve
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 164-175

In multi-label settings
-----------------------

The precision-recall curve does not support the multilabel setting. However,
one can decide how to handle this case. We show such an example below.

Create multi-label data, fit, and predict
.........................................

We create a multi-label dataset, to illustrate the precision-recall in
multi-label settings.

.. GENERATED FROM PYTHON SOURCE LINES 175-187

.. code-block:: default


    from sklearn.preprocessing import label_binarize

    # Use label_binarize to be multi-label like settings
    Y = label_binarize(y, classes=[0, 1, 2])
    n_classes = Y.shape[1]

    # Split into training and test
    X_train, X_test, Y_train, Y_test = train_test_split(
        X, Y, test_size=0.5, random_state=random_state
    )








.. GENERATED FROM PYTHON SOURCE LINES 188-190

We use :class:`~sklearn.multiclass.OneVsRestClassifier` for multi-label
prediction.

.. GENERATED FROM PYTHON SOURCE LINES 190-199

.. code-block:: default

    from sklearn.multiclass import OneVsRestClassifier

    classifier = OneVsRestClassifier(
        make_pipeline(StandardScaler(), LinearSVC(random_state=random_state, dual="auto"))
    )
    classifier.fit(X_train, Y_train)
    y_score = classifier.decision_function(X_test)









.. GENERATED FROM PYTHON SOURCE LINES 200-202

The average precision score in multi-label settings
...................................................

.. GENERATED FROM PYTHON SOURCE LINES 202-218

.. code-block:: default

    from sklearn.metrics import average_precision_score, precision_recall_curve

    # For each class
    precision = dict()
    recall = dict()
    average_precision = dict()
    for i in range(n_classes):
        precision[i], recall[i], _ = precision_recall_curve(Y_test[:, i], y_score[:, i])
        average_precision[i] = average_precision_score(Y_test[:, i], y_score[:, i])

    # A "micro-average": quantifying score on all classes jointly
    precision["micro"], recall["micro"], _ = precision_recall_curve(
        Y_test.ravel(), y_score.ravel()
    )
    average_precision["micro"] = average_precision_score(Y_test, y_score, average="micro")








.. GENERATED FROM PYTHON SOURCE LINES 219-221

Plot the micro-averaged Precision-Recall curve
..............................................

.. GENERATED FROM PYTHON SOURCE LINES 221-232

.. code-block:: default

    from collections import Counter

    display = PrecisionRecallDisplay(
        recall=recall["micro"],
        precision=precision["micro"],
        average_precision=average_precision["micro"],
        prevalence_pos_label=Counter(Y_test.ravel())[1] / Y_test.size,
    )
    display.plot(plot_chance_level=True)
    _ = display.ax_.set_title("Micro-averaged over all classes")




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_003.png
   :alt: Micro-averaged over all classes
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 233-235

Plot Precision-Recall curve for each class and iso-f1 curves
............................................................

.. GENERATED FROM PYTHON SOURCE LINES 235-278

.. code-block:: default

    from itertools import cycle

    import matplotlib.pyplot as plt

    # setup plot details
    colors = cycle(["navy", "turquoise", "darkorange", "cornflowerblue", "teal"])

    _, ax = plt.subplots(figsize=(7, 8))

    f_scores = np.linspace(0.2, 0.8, num=4)
    lines, labels = [], []
    for f_score in f_scores:
        x = np.linspace(0.01, 1)
        y = f_score * x / (2 * x - f_score)
        (l,) = plt.plot(x[y >= 0], y[y >= 0], color="gray", alpha=0.2)
        plt.annotate("f1={0:0.1f}".format(f_score), xy=(0.9, y[45] + 0.02))

    display = PrecisionRecallDisplay(
        recall=recall["micro"],
        precision=precision["micro"],
        average_precision=average_precision["micro"],
    )
    display.plot(ax=ax, name="Micro-average precision-recall", color="gold")

    for i, color in zip(range(n_classes), colors):
        display = PrecisionRecallDisplay(
            recall=recall[i],
            precision=precision[i],
            average_precision=average_precision[i],
        )
        display.plot(ax=ax, name=f"Precision-recall for class {i}", color=color)

    # add the legend for the iso-f1 curves
    handles, labels = display.ax_.get_legend_handles_labels()
    handles.extend([l])
    labels.extend(["iso-f1 curves"])
    # set the legend and the axes
    ax.set_xlim([0.0, 1.0])
    ax.set_ylim([0.0, 1.05])
    ax.legend(handles=handles, labels=labels, loc="best")
    ax.set_title("Extension of Precision-Recall curve to multi-class")

    plt.show()



.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_004.png
   :alt: Extension of Precision-Recall curve to multi-class
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_precision_recall_004.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.306 seconds)


.. _sphx_glr_download_auto_examples_model_selection_plot_precision_recall.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/1.3.X?urlpath=lab/tree/notebooks/auto_examples/model_selection/plot_precision_recall.ipynb
        :alt: Launch binder
        :width: 150 px



    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_precision_recall.py <plot_precision_recall.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_precision_recall.ipynb <plot_precision_recall.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
