Tuesday, November 13, 2012

Linux Wait I/O what process is causing it ?

How to analyze Wait I/O problem ?

the first thing is to control what kernel version you have.

> 2.6.20 ?

You can use the IOTOP or LATENCYTOP utility that uses the IO_ACCOUNT capability of new kernel

< 2.6.20 ?

block_dump
block_dump enables block I/O debugging when set to a nonzero value.
When this flag is set, Linux reports all disk read and write operations that take place, and all block dirtyings done to files. The output of block_dump is written to the kernel output, and it can be retrieved using "dmesg".
When you use block_dump and your kernel logging level also includes kernel debugging messages, you probably want to turn off klogd, otherwise the output of block_dump will be logged, causing disk activity that is not normally there.

- First you must clear all log in "dmesg"
#dmesg -c 

- Stop any syslog you have for loggin only disk activity
 #/etc/init.d/syslog stop 

- Enable Block_Dump
# echo 1 > /proc/sys/vm/block_dump 
( also sysctl vm.block_dump for read the value and sysctl vm.block_dump=1 to enable the feature)

- Start the reporting: 
#while true; do sleep 1; dmesg -c; done | perl iodump (download iodump script )

- CTRL-C Stop the system from dumping these messages

- Disable Block_Dump after the analisys
#echo 0 > /proc/sys/vm/block_dump 
(also sysctl vm.block_dump for read the value and sysctl vm.block_dump=0 to disable the feature)

- Re-Start any syslog
# /etc/init.d/syslog start



dmesg -c
/etc/init.d/syslog stop 
echo 1 > /proc/sys/vm/block_dump 
touch /tmp/disklog 
watch "dmesg -c >> /tmp/disklog" 
CTRL-C when you're done collecting data 
echo 0 > /proc/sys/vm/block_dump 
/etc/init.d/syslog start
cat /tmp/disklog | awk -F"[() \t]" '/(READ|WRITE|dirtied)/ {activity[$1]++} END {for (x in activity) print x, activity[x]}'| sort -nr -k2 

No comments: