PR曲线为什么计算AP而非AUC-The AP of PR curve

发表于 | 分类于 | | 阅读数 次
字数统计: 830 | 阅读时长 ≈ 4

在计算性能指标时，如果仔细观察，会发现PR曲线并不是直接计算ROC，而是AP，average precision，那么他到底有啥区别呢？

PR曲线的AUC和AP/mAP

看了很多，还是有点懵，但不纠结了，先记录下几点吧。。。

结论

首先，结论是AP算的比PR-AUC来的好，后者常常会高估了真实性能，二者采用的方法可以看下面的example，高估的原因在于一般的PR-AUC所采取的是线性插值的方法，比如第二点会提到的那个VOC比赛，采用的就是插值的方法

官方描述：sklearn.metrics .average_precision_score

sklearn.metrics.average_precision_score(y_true, y_score, **, average=’macro’, pos_label=1, sample_weight=None*)[source]

Compute average precision (AP) from prediction scores.

AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight:

where Pn and Rn are the precision and recall at the nth threshold [1]. This implementation is not interpolated and is different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic.

Note: this implementation is restricted to the binary classification task or multilabel classification task.

Read more in the User Guide.

源码

其实看scikit的源码差别就在于：

AUC-PR 用的trapz直接求积分，就是(precision k - precision k-1)/2 * (recall k - recall k-1)

AP-PR 用的 用的是 precision k * (recall k - recall k-1)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 #Parameters
    x : ndarray of shape (n,)
        x coordinates. These must be either monotonic increasing or monotonic
        decreasing.
    y : ndarray of shape, (n,)
        y coordinates.
# AUC
direction = -1
dx = np.diff(x)
area = direction * np.trapz(y, x)
# np.trapz = ret = (d * (y[tuple(slice1)] + y[tuple(slice2)]) / 2.0).sum(axis) 

# AP
precision, recall, _ = precision_recall_curve(y_true, y_score, pos_label=pos_label, sample_weight=sample_weight)
-np.sum(np.diff(recall) * np.array(precision)[:-1])

Example：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_recall_curve
from sklearn.metrics import auc, plot_precision_recall_curve
import matplotlib.pyplot as plt

random_state = 416
# Create dataset for binary classification with 5 predictors
X, y = datasets.make_classification(n_samples=1000,
                                    n_features=5,
                                    n_informative=3,
                                    n_redundant=2,
                                    random_state=random_state)

# Split into training and test
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=.4,
                                                    random_state=random_state)

# Create classifier using logistic regression
classifier = LogisticRegression(random_state=random_state)
classifier.fit(X_train, y_train)

# Get the predicited probability of testing data 
y_score = classifier.predict_proba(X_test)[:, 1]
Use average_precision_score

The average precision (PR AUC) is returned by passing the true label & the probability estimate.

1
2
3
4
# Average precision score
average_precision = average_precision_score(y_test, y_score)
print(average_precision)
## 0.9057007040716777
Use precision_recall_curve&auc

When using auc function to compute the area under the precision-recall curve, as mentioned earlier, the result is not the same as the value from average_precision_score, but it does not differ too much since the number of data points are large enough to mitigate the effect of wiggles.

1
2
3
4
5
6
7
8
# Data to plot precision - recall curve
precision, recall, thresholds = precision_recall_curve(y_test, y_score)
# Use AUC function to calculate the area under the curve of precision recall curve
auc_precision_recall = auc(recall, precision)
print(auc_precision_recall)
## 0.9053487302244206
plt.plot(recall, precision)
plt.show()

image-20211029183259512

Use build-in function to plot precision-recall curve

In version 0.22.0 of scikit learn, plot_precision_recall_curve is added into the metrics module. It is easy to plot the precision-recall curve with sufficient information by using the classifier without any extra steps to generate the prediction of probability,

1
2
3
disp = plot_precision_recall_curve(classifier, X_test, y_test)
disp.ax_.set_title('Binary class Precision-Recall curve: '
                   'AP={0:0.2f}'.format(average_precision))

image-20211029183319952

If you need to compute the area under the curve of precision-recall plot, don’t forget to use average_precision_score to help you get robust result quickly.

example来自：Compute the AUC of Precision-Recall Curve

多分类问题/mAP

是涉及到多分类问题，如VOC比赛里的评判标准，07年采用的11-point方法，10年以后采取的

image-20211029184456351

论文出处：M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn, A. Zisserman, The Pascal Visual Object Classes (VOC) Challenge, IJCV 2010.

具体实现：AP，mAP计算详解（代码全解） - JimmyHua的文章 - 知乎 https://zhuanlan.zhihu.com/p/70667071

img

img

img

还有一篇：讲得很好 ReID任务中的CMC和mAP

0%