Linux 系统出现异常排查思路

16 系统出现异常排查思路

16.1 查看用户信息

16.1.1查看当前的用户

# who

04:39:39 up 1:30, 1 user, load average: 0.01, 0.01, 0.00

USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT

root pts/0 192.168.215.1 04:27 0.00s 0.16s 0.02s w

16.1.2查看最近登录的用户

# last

***************

root pts/2 hadoop2 Sun Oct 16 15:52 - 15:52 (00:00)

root pts/1 192.168.215.1 Sun Oct 16 15:39 - down (00:23)

hadoop pts/0 :0.0 Sun Oct 16 00:33 - down (15:30)

hadoop tty1 :0 Sun Oct 16 00:31 - down (15:31)

reboot system boot 2.6.32-573.el6.x Sun Oct 16 08:16 - 16:03 (07:47)

16.2 查看直线执行的命令

# history

***************

683 last

684 clear

685 last

686 clear

687 history

16.3查看现在运行的进程

# pstree -a

init

├─NetworkManager --pid-file=/var/run/NetworkManager/NetworkManager.pid

├─abrtd

├─acpid

├─atd

├─auditd

│ └─{auditd}

├─bonobo-activati --ac-activate --ior-output-fd=12

*******************

# ps aux

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1 0.0 0.0 19352 1544 ? Ss 03:09 0:02 /sbin/init

root 2 0.0 0.0 0 0 ? S 03:09 0:00 [kthreadd]

root 3 0.0 0.0 0 0 ? S 03:09 0:00 [migration/0]

root 4 0.0 0.0 0 0 ? S 03:09 0:00 [ksoftirqd/0]

root 5 0.0 0.0 0 0 ? S 03:09 0:00 [stopper/0]

16.4查看网络服务的进程

16.4.1查看正在运行的端口

# netstat -nltl

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN

tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN

tcp 0 0 :::2181 :::* LISTEN

tcp 0 0 :::37129 :::* LISTEN

16.4.2正在活跃的端口

# netstat -nulp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

udp 0 0 0.0.0.0:631 0.0.0.0:* 2089/cupsd

16.4.3 查看UNIX活跃的端口

# netstat -nxlp

Active UNIX domain sockets (only servers)

Proto RefCnt Flags Type State I-Node PID/Program name Path

unix 2 [ ACC ] STREAM LISTENING 13954 2136/hald @/var/run/hald/dbus-WAkpL6y5o7

unix 2 [ ACC ] STREAM LISTENING 16245 2614/gnome-session @/tmp/.ICE-unix/2614

unix 2 [ ACC ] STREAM LISTENING 15966 2524/Xorg @/tmp/.X11-unix/X0

unix 2 [ ACC ] STREAM LISTENING 13947 2136/hald @/var/run/hald/dbus-QUMwKtSaJ5

unix 2 [ ACC ] STREAM LISTENING 13818 2089/cupsd /var/run/cups/cups.sock

*********************

16.5查看CPU与内存

16.5.1查看空闲的内存以及内存与硬盘之间的SWAP

# free -m

total used free shared buffers cached

Mem: 1862 475 1386 1 27 202

-/+ buffers/cache: 245 1616

Swap: 2047 0 2047

# free -g

总计 已用 空闲 共享 缓冲/缓存 可用

内存： 15 7 1 0 6 6

交换： 1 0 1

16.6查看运行的详细信息

# uptime

04:59:59 up 1:50, 1 user, load average: 0.00, 0.00, 0.00

当前时间 04:59:59

系统已运行的时间 1:50

当前在线用户 1 user

平均负载：0.00, 0.00, 0.00，最近1分钟、5分钟、15分钟系统的负载

16.7动态查看运行的内存,CPU等信息

# top

top - 12:26:46 up 16:21, 1 user, load average: 0.00, 0.00, 0.00

Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.7%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st

Mem: 1895288k total, 665188k used, 1230100k free, 20628k buffers

Swap: 2097144k total, 0k used, 2097144k free, 80392k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2269 root 20 0 15056 1080 832 R 2.0 0.1 0:00.01 top

1 root 20 0 19356 1536 1228 S 0.0 0.1 0:01.81 init

2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0

4 root 20 0 0 0 0 S 0.0 0.0 0:01.13 ksoftirqd/0

5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0

6 root RT 0 0 0 0 S 0.0 0.0 0:00.14 watchdog/0

7 root 20 0 0 0 0 S 0.0 0.0 0:41.30 events/0

8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup

9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper

***********************

16.8 硬件信息

16.8.1系统中所有PCI总线设备或连接到该总线上的所有设备

# lspci

00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)

00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)

00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)

00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)

00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)

16.8.2查看硬件方面的信息

# ethtool eth0

*******************

Handle 0x0229, DMI type 33, 31 bytes

64-bit Memory Error Information

Type: OK

Granularity: Unknown

Operation: Unknown

Vendor Syndrome: Unknown

Memory Array Address: Unknown

Device Address: Unknown

Resolution: Unknown

Handle 0x022A, DMI type 126, 4 bytes

Inactive

Handle 0x022B, DMI type 127, 4 bytes

End Of Table

16.9 IO的性能

16.9.1 查看磁盘的使用情况

# iostat

Linux 2.6.32-573.el6.x86_64 (hadoop1) 10/21/2016 _x86_64_(1 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle

0.17 0.00 0.56 2.15 0.00 97.11

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

sda 1.49 75.27 10.68 645224 91568

16.9.2 动态的查看服务器的状态值

# vmstat 2 10

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----

r b swpd free buff cache si so bi bo in cs us sy id wa st

0 0 0 1322196 30688 298892 0 0 37 5 39 57 0 1 97 2 0

0 0 0 1322140 30688 298920 0 0 0 0 57 84 1 1 99 0 0

*********************

16.9.3实时的对系统的监控

# mpstat 2 10

Linux 2.6.32-573.el6.x86_64 (hadoop1) 10/21/2016 _x86_64_(1 CPU)

05:37:26 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle

05:37:28 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

05:37:30 AM all 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 99.50

05:37:32 AM all 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 99.50

*********************

16.9.4动态显示当前的操作IO的进程

# yum -y install dstat

# dstat --top-io --top-bio

----most-expensive---- ----most-expensive----

i/o process | block i/o process

bash 53k 316B|init 19k 198B

sshd: root@ 301B 340B|tpvmlpd2 0 4096B

sshd: root@ 136B 180B|jbd2/sda2-8 0 56k

16.10文件系统以及外接磁盘的信息

16.10.1查看当前的挂在的设备

# mount

/dev/sda2 on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

/dev/sda1 on /boot type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

vmware-vmblock on /var/run/vmblock-fuse type fuse.vmware-vmblock (rw,nosuid,nodev,default_permissions,allow_other)

16.10.2查看是否有专用的文件系统

打开一下文件进行编辑

# cat /etc/fstab

#

# /etc/fstab

# Created by anaconda on Sun Oct 16 07:55:57 2016

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

UUID=b89c0aae-3284-4835-9b1b-04986146cd96 / ext4 defaults 1 1

UUID=a1313d92-6873-402d-95a6-add6cd1321c6 /boot ext4 defaults 1 2

UUID=6a5cde98-2fc5-4d8f-976c-92acb39ab2a9 swap swap defaults 0 0

tmpfs /dev/shm tmpfs defaults 0 0

devpts /dev/pts devpts gid=5,mode=620 0 0

sysfs /sys sysfs defaults 0 0

proc /proc proc defaults 0 0

16.10.3查看文件系统的挂在的选项

# vgs

16.10.4查看物理卷的信息

# pvs

16.11查看磁盘的剩余情况

# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda2 18G 6.2G 11G 38% /

tmpfs 932M 72K 932M 1% /dev/shm

/dev/sda1 283M 41M 228M 16% /boot

16.12列出当前系统打开文件的工具

# lsof +D / /* beware not to kill your box */

***************

lsof 3907 root mem REG 8,2 22536 265965 /lib64/libdl-2.12.so

lsof 3907 root mem REG 8,2 1926480 265960 /lib64/libc-2.12.so

lsof 3907 root mem REG 8,2 124624 265966 /lib64/libselinux.so.1

lsof 3907 root mem REG 8,2 99158576 394281 /usr/lib/locale/locale-archive

16.12 内核与网络

16.12.1显示在/proc/sys目录中的内核参数

**************

net.ipv6.nf_conntrack_frag6_high_thresh = 4194304

net.ipv6.ip6frag_secret_interval = 600

net.ipv6.mld_max_msf = 64

net.nf_conntrack_max = 65536

net.unix.max_dgram_qlen = 10

abi.vsyscall32 = 1

crypto.fips_enabled = 0

16.12.2 显示设备的详细信息

irq的序号， 在各自cpu上发生中断的次数，可编程中断控制器，设备名称（request_irq的dev_name字段）

# cat /proc/interrupts

CPU0

0: 261 IO-APIC-edge timer

1: 8 IO-APIC-edge i8042

4: 4838 IO-APIC-edge

8: 1 IO-APIC-edge rtc0

9: 0 IO-APIC-fasteoi acpi

查看链接数据库的信息

# cat /proc/net/ip_conntrack /* may take some time on busy servers */

**************

cat: sys/: Is a directory

cat: tmp/: Is a directory

cat: usr/: Is a directory

cat: var/: Is a directory

16.13查看网络套接字连接情况

# netstat

************

unix 3 [ ] STREAM CONNECTED 13648

unix 3 [ ] STREAM CONNECTED 13647

unix 3 [ ] DGRAM 10073

unix 3 [ ] DGRAM 10072

16.14获取socket统计信息

# ss -s

Total: 602 (kernel 610)

TCP: 15 (estab 4, closed 0, orphaned 0, synrecv 0, timewait 0/0), ports 8

Transport Total IP IPv6

* 610 - -

RAW 0 0 0

UDP 1 1 0

TCP 15 5 10

INET 16 6 10

FRAG 0 0 0

16.15日志消息与内核信息的查看

16.15.1 显示linux内核的环形缓冲区信息

# dmesg [ tail / less / grep / more ]

*************

eth0: no IPv6 routers present

lp: driver loaded but no devices found

ppdev: user-space parallel port driver

hrtimer: interrupt took 2588670 ns

16.15.2查看系统报错日志

# less /var/log/messages

Oct 16 08:16:22 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.

Oct 16 08:16:22 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1604" x-info="http://www.rsyslog.com"] start

Oct 16 08:16:22 localhost kernel: Initializing cgroup subsys cpuset

Oct 16 08:16:22 localhost kernel: Initializing cgroup subsys cpu

*************

16.15.3 安全信息和系统登录与网络连接的信息

# less /var/log/secure

Oct 16 08:17:06 localhost sshd[8287]: Server listening on 0.0.0.0 port 22.

Oct 16 08:17:06 localhost sshd[8287]: Server listening on :: port 22.

Oct 16 00:22:58 localhost polkitd(authority=local): Registered Authentication Agent for session /org/freedesktop/ConsoleKit/Session1 (system bus name :1.25 [/usr/libexec/polkit-gnome-authentication-agent-1], object path /org/gnome/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

********************

16.16查看定时的任务

16.16.1查看定时任务的运行频率

# ls /etc/cron* + cat

/etc/cron.daily:

cups logrotate makewhatis.cron mlocate.cron prelink readahead.cron tmpwatch

/etc/cron.hourly:

0anacron

/etc/cron.monthly:

readahead-monthly.cron

/etc/cron.weekly:

16.1.2 查看用户是否执行了隐藏的命令

# for user in $(cat /etc/passwd | cut -f1 -d:); do crontab -l -u $user; done

no crontab for root

no crontab for bin

no crontab for daemon