python | 简单的数据分析

做数据分析的两大利器：python和R语言，这里介绍一个我用python学习的案例

第一步，设置工作目录

#encoding:utf8
import os
os.chdir("G:\\Anaconda3\\Scripts\\lecture01\\Feature_engineering_and_model_tuning\\Feature-engineering_and_Parameter_Tuning_XGBoost")

第二步，加载包

import pandas as pd
import numpy as np
%matplotlib inline

第三步，载入数据

#载入数据:
train = pd.read_csv('Train.csv',encoding = "ISO-8859-1")
test = pd.read_csv('Test.csv',encoding = "ISO-8859-1")

第四步，查看数据

• 维数

train.shape, test.shape

((87020, 26), (37717, 24))

• 数据类型

#看看数据的基本情况
train.dtypes

ID object Gender object City object Monthly_Income int64 DOB object Lead_Creation_Date object Loan_Amount_Applied float64 Loan_Tenure_Applied float64 Existing_EMI float64 Employer_Name object Salary_Account object Mobile_Verified object Var5 int64 Var1 object Loan_Amount_Submitted float64 Loan_Tenure_Submitted float64 Interest_Rate float64 Processing_Fee float64 EMI_Loan_Submitted float64 Filled_Form object Device_Type object Var2 object Source object Var4 int64 LoggedIn int64 Disbursed int64 dtype: object

• 查看数据

#拿前5条出来看看
train.head(5)

• 数据合并

#合成一个总的data
train['source']= 'train'
test['source'] = 'test'
data=pd.concat([train, test],ignore_index=True)
data.shape

(124737, 27)

• 查看异常值
  • 空值

data.apply(lambda x: sum(x.isnull()))

City 1401 DOB 0 Device_Type 0 Disbursed 37717 EMI_Loan_Submitted 84901 Employer_Name 113 Existing_EMI 111 Filled_Form 0 Gender 0 ID 0 Interest_Rate 84901 Lead_Creation_Date 0 Loan_Amount_Applied 111 Loan_Amount_Submitted 49535 Loan_Tenure_Applied 111 Loan_Tenure_Submitted 49535 LoggedIn 37717 Mobile_Verified 0 Monthly_Income 0 Processing_Fee 85346 Salary_Account 16801 Source 0 Var1 0 Var2 0 Var4 0 Var5 0 source 0 dtype: int64 - 查看每一列的取值种数

var = ['Gender','Salary_Account','Mobile_Verified','Var1','Filled_Form','Device_Type','Var2','Source']
for v in var:
    print ('\n%s这一列数据的不同取值和出现的次数\n'%v)
    print (data[v].value_counts())

Gender这一列数据的不同取值和出现的次数

Male 71398 Female 53339 Name: Gender, dtype: int64

Salary_Account这一列数据的不同取值和出现的次数

HDFC Bank 25180 ICICI Bank 19547 State Bank of India 17110 Axis Bank 12590 Citibank 3398 Kotak Bank 2955 IDBI Bank 2213 Punjab National Bank 1747 Bank of India 1713 Bank of Baroda 1675 Standard Chartered Bank 1434 Canara Bank 1385 Union Bank of India 1330 Yes Bank 1120 ING Vysya 996 Corporation bank 948 Indian Overseas Bank 901 State Bank of Hyderabad 854 Indian Bank 773 Oriental Bank of Commerce 761 IndusInd Bank 711 Andhra Bank 706 Central Bank of India 648 Syndicate Bank 614 Bank of Maharasthra 576 HSBC 474 State Bank of Bikaner & Jaipur 448 Karur Vysya Bank 435 State Bank of Mysore 385 Federal Bank 377 Vijaya Bank 354 Allahabad Bank 345 UCO Bank 344 State Bank of Travancore 333 Karnataka Bank 279 United Bank of India 276 Dena Bank 268 Saraswat Bank 265 State Bank of Patiala 263 South Indian Bank 223 Deutsche Bank 176 Abhyuday Co-op Bank Ltd 161 The Ratnakar Bank Ltd 113 Tamil Nadu Mercantile Bank 103 Punjab & Sind bank 84 J&K Bank 78 Lakshmi Vilas bank 69 Dhanalakshmi Bank Ltd 66 State Bank of Indore 32 Catholic Syrian Bank 27 India Bulls 21 B N P Paribas 15 Firstrand Bank Limited 11 GIC Housing Finance Ltd 10 Bank of Rajasthan 8 Kerala Gramin Bank 4 Industrial And Commercial Bank Of China Limited 3 Ahmedabad Mercantile Cooperative Bank 1 Name: Salary_Account, dtype: int64

Mobile_Verified这一列数据的不同取值和出现的次数

Y 80928 N 43809 Name: Mobile_Verified, dtype: int64

Var1这一列数据的不同取值和出现的次数

HBXX 84901 HBXC 12952 HBXB 6502 HAXA 4214 HBXA 3042 HAXB 2879 HBXD 2818 HAXC 2171 HBXH 1387 HCXF 990 HAYT 710 HAVC 570 HAXM 386 HCXD 348 HCYS 318 HVYS 252 HAZD 161 HCXG 114 HAXF 22 Name: Var1, dtype: int64

Filled_Form这一列数据的不同取值和出现的次数

N 96740 Y 27997 Name: Filled_Form, dtype: int64

Device_Type这一列数据的不同取值和出现的次数

Web-browser 92105 Mobile 32632 Name: Device_Type, dtype: int64

Var2这一列数据的不同取值和出现的次数

B 53481 G 47338 C 20366 E 1855 D 918 F 770 A 9 Name: Var2, dtype: int64

Source这一列数据的不同取值和出现的次数

S122 55249 S133 42900 S159 7999 S143 6140 S127 2804 S137 2450 S134 1900 S161 1109 S151 1018 S157 929 S153 705 S144 447 S156 432 S158 294 S123 112 S141 83 S162 60 S124 43 S150 19 S160 11 S136 5 S138 5 S155 5 S139 4 S129 4 S135 2 S142 1 S140 1 S154 1 S125 1 S130 1 S126 1 S132 1 S131 1 Name: Source, dtype: int64

• 单个特征分析

计算字段数

#City字段处理
len(data['City'].unique())

删掉属性

data.drop('City',axis=1,inplace=True)

• 缺失值用中位数填补

#找中位数去填补缺省值（因为缺省的不多）
data['Loan_Amount_Applied'].fillna(data['Loan_Amount_Applied'].median(),inplace=True)

• 数据集的One-Hot编码

data = pd.get_dummies(data, columns=var_to_encode)
data.columns