使用PyOD进行异常值检测

异常值检测各个领域的关键任务之一。PyOD是Python Outlier Detection的缩写，可以简化多变量数据集中识别异常值的过程。在本文中，我们将介绍PyOD包，并通过实际给出详细的代码示例

PyOD简介

PyOD为异常值检测提供了广泛的算法集合，适用于有监督和无监督的场景。无论处理的是带标签的数据还是未带标签的数据，PyOD都提供了一系列技术来满足特定需求。PyOD的突出特性之一是其用户友好的API，使新手和有经验的从业者都可以轻松的访问它。

示例1：kNN

我们从一个简单的例子开始，利用k近邻(kNN)算法进行离群值检测。

首先从PyOD导入必要的模块

我们生成具有预定义离群率的合成数据来模拟异常值。

contamination = 0.1 # percentage of outliers

n_train = 200 # number of training points

n_test = 100 # number of testing points

X_train, X_test, y_train, y_test = generate_data(

  n_train=n_train, n_test=n_test, contamination=contamination)

初始化kNN检测器，将其与训练数据拟合，并获得离群值预测。

clf_name = 'KNN'

clf = KNN()

clf.fit(X_train)

使用ROC和Precision @ Rank n指标评估训练模型在训练和测试数据集上的性能。

print("\nOn Training Data:")

evaluate_print(clf_name, y_train, clf.decision_scores_)

print("\nOn Test Data:")

evaluate_print(clf_name, y_test, clf.decision_function(X_test))

最后可以使用内置的可视化功能可视化离群检测结果。

这是一个简单的用法示例

示例2 模型集成

异常值检测有时会受到模型不稳定性的影响，特别是在无监督的情况下。所以PyOD提供了模型组合技术来提高鲁棒性。

import numpy as np

from sklearn.model_selection import train_test_split

from scipy.io import loadmat

from pyod.models.knn import KNN

from pyod.models.combination import aom, moa, average, maximization, median

from pyod.utils.utility import standardizer

from pyod.utils.data import generate_data

from pyod.utils.data import evaluate_print

X, y = generate_data(train_only=True) # load data

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4)

# standardizing data for processing

X_train_norm, X_test_norm = standardizer(X_train, X_test)

n_clf = 20 # number of base detectors

# Initialize 20 base detectors for combination

k_list = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140,

          150, 160, 170, 180, 190, 200]

train_scores = np.zeros([X_train.shape[0], n_clf])

test_scores = np.zeros([X_test.shape[0], n_clf])

print('Combining {n_clf} kNN detectors'.format(n_clf=n_clf))

for i in range(n_clf):

  k = k_list[i]

  clf = KNN(n_neighbors=k, method='largest')

  clf.fit(X_train_norm)

  train_scores[:, i] = clf.decision_scores_

  test_scores[:, i] = clf.decision_function(X_test_norm)

# Decision scores have to be normalized before combination

train_scores_norm, test_scores_norm = standardizer(train_scores,

                                                  test_scores)

# Combination by average

y_by_average = average(test_scores_norm)

evaluate_print('Combination by Average', y_test, y_by_average)

# Combination by max

y_by_maximization = maximization(test_scores_norm)

evaluate_print('Combination by Maximization', y_test, y_by_maximization)

# Combination by median

y_by_median = median(test_scores_norm)

evaluate_print('Combination by Median', y_test, y_by_median)

# Combination by aom

y_by_aom = aom(test_scores_norm, n_buckets=5)

evaluate_print('Combination by AOM', y_test, y_by_aom)

# Combination by moa

y_by_moa = moa(test_scores_norm, n_buckets=5)

evaluate_print('Combination by MOA', y_test, y_by_moa)

如果上面代码提示错误，需要安装combo包

pip install combo总结

可以看到，PyOD进行离群值检测是非常方便的，从基本的kNN离群值检测到模型组合，PyOD都提供了一个全面的整合，这使得我们可以轻松高效地处理异常值检测任务。

最后pyod的文档和官网https://pyod.readthedocs.io/en/latest/