5种方法锁定Pandas缺失值

皮大大

发布于 2023-08-25 09:43:31

1370

发布于 2023-08-25 09:43:31

文章被收录于专栏：机器学习/数据可视化机器学习/数据可视化

5大绝技锁定缺失值所在行

本文记录的是：如何锁定Pandas中缺失值所在的行

数据

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "A":list(range(1,11)),
    "B":list(range(11,21)),
    "C":list(range(21,31)),
    "D":list(range(31,41))
})

df

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	1	11	21	31
1	2	12	22	32
2	3	13	23	33
3	4	14	24	34
4	5	15	25	35
5	6	16	26	36
6	7	17	27	37
7	8	18	28	38
8	9	19	29	39
9	10	20	30	40

设置空值

df.iloc[2,0] = np.nan
df.iloc[3,1] = np.nan
df.iloc[3,2] = np.nan
df.iloc[5,2] = np.nan
df.iloc[9,1] = np.nan

df

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	1.0	11.0	21.0	31
1	2.0	12.0	22.0	32
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
4	5.0	15.0	25.0	35
5	6.0	16.0	NaN	36
6	7.0	17.0	27.0	37
7	8.0	18.0	28.0	38
8	9.0	19.0	29.0	39
9	10.0	NaN	30.0	40

统计空值个数

# 统计每列下空值的个数
df.isnull().sum()

A    1
B    2
C    2
D    0
dtype: int64

确定空值所在行

方法1

# 1、每个位置是否为空

df.isnull()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	False	False	False	False
1	False	False	False	False
2	True	False	False	False
3	False	True	True	False
4	False	False	False	False
5	False	False	True	False
6	False	False	False	False
7	False	False	False	False
8	False	False	False	False
9	False	True	False	False

# 2、转置供能

df.isnull().T

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	0	1	2	3	4	5	6	7	8	9
A	False	False	True	False	False	False	False	False	False	False
B	False	False	False	True	False	False	False	False	False	True
C	False	False	False	True	False	True	False	False	False	False
D	False	False	False	False	False	False	False	False	False	False

# 3、any表示至少有一个空值

df.isnull().T.any()

0    False
1    False
2     True
3     True
4    False
5     True
6    False
7    False
8    False
9     True
dtype: bool

# 4、确定空值的行

df[df.isnull().T.any()]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

方法2

df1 = df.isnull()
df1

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	False	False	False	False
1	False	False	False	False
2	True	False	False	False
3	False	True	True	False
4	False	False	False	False
5	False	False	True	False
6	False	False	False	False
7	False	False	False	False
8	False	False	False	False
9	False	True	False	False

新生成一个列E，是前面4个列的求和。求和的时候只要出现一个True，则为True

df1["E"] = df1["A"] + df1["B"] + df1["C"] + df1["D"]

df1

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D	E
0	False	False	False	False	False
1	False	False	False	False	False
2	True	False	False	False	True
3	False	True	True	False	True
4	False	False	False	False	False
5	False	False	True	False	True
6	False	False	False	False	False
7	False	False	False	False	False
8	False	False	False	False	False
9	False	True	False	False	True

# 把df1["E"]列传进来

df[df1["E"]]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

方法3

df.isnull().values==True

array([[False, False, False, False],
       [False, False, False, False],
       [ True, False, False, False],
       [False,  True,  True, False],
       [False, False, False, False],
       [False, False,  True, False],
       [False, False, False, False],
       [False, False, False, False],
       [False, False, False, False],
       [False,  True, False, False]])

df[df.isnull().values==True]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

可以看到结果中出现了重复的行，这个因为第4行中有2个缺失值，需要去重：

# 删除重复值
df[df.isnull().values==True].drop_duplicates()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

方法4

# 每个位置判断【不是空值-notnull】
pd.notnull(df)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	True	True	True	True
1	True	True	True	True
2	False	True	True	True
3	True	False	False	True
4	True	True	True	True
5	True	True	False	True
6	True	True	True	True
7	True	True	True	True
8	True	True	True	True
9	True	False	True	True

df[~pd.notnull(df).all(axis=1)]  # 执行取反操作

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

# notnull 和 notna 等效
df[~pd.notna(df).all(axis=1)]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
2	NaN	13.0	23.0	33
3	4.0	NaN	NaN	34
5	6.0	16.0	NaN	36
9	10.0	NaN	30.0	40

方法5

# 统计每个位置是否为空

df.isnull()

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>

	A	B	C	D
0	False	False	False	False
1	False	False	False	False
2	True	False	False	False
3	False	True	True	False
4	False	False	False	False
5	False	False	True	False
6	False	False	False	False
7	False	False	False	False
8	False	False	False	False
9	False	True	False	False

# 按照行统计：只要存在一个空值即为True

(df.isnull()).any(axis=1)

0    False
1    False
2     True
3     True
4    False
5     True
6    False
7    False
8    False
9     True
dtype: bool

df[(df.isnull()).any(axis=1)]

.dataframe tbody tr th:only-of-type { vertical-align: middle; } <pre><code>.dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; } </code></pre>