Sharing

2013年9月2日 星期一

wa (Waiting for I/O) from top command is high


遇到有台機器動作很慢, 用 top 一看,loading 滿重的, 而且 wa 高達 96.9%, 看起來應該是 io 出了問題

$ top
top - 03:08:30 up 34 days, 15:48,  1 user,  load average: 41.70, 43.33, 43.77
Tasks: 323 total,   1 running, 322 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  1.4%sy,  0.0%ni,  0.0%id, 96.9%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   1019112k total,   953076k used,    66036k free,     1092k buffers
Swap:  4111356k total,  2204620k used,  1906736k free,    22320k cached

http://serverfault.com/questions/155882/wa-waiting-for-i-o-from-top-command-is-big


$ iostat 1
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.03    0.00    1.03   97.94    0.00    0.00

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
scd0              0.00         0.00         0.00          0          0
sda             493.81      7327.84       738.14       7108        716
sdb               0.00         0.00         0.00          0          0




2013年9月1日 星期日

Software RAID mdadm



Get Volume List

$ mdadm --detail --scan
ARRAY /dev/md125 metadata=0.90 spares=1 UUID=ad192d4a:114eb1c4:7dadf8c1:9019de71
ARRAY /dev/md127 metadata=0.90 UUID=8b53927c:8e569dae:c589cde9:18eb45a5
ARRAY /dev/md126 metadata=0.90 UUID=ab76aade:4b9bd34d:c589cde9:18eb45a5

Get Volume Detail


$ mdadm --detail /dev/md125
/dev/md125:
        Version : 0.90
  Creation Time : Mon Jul 30 04:13:15 2012
     Raid Level : raid1
     Array Size : 29296576 (27.94 GiB 30.00 GB)
  Used Dev Size : 29296576 (27.94 GiB 30.00 GB)
   Raid Devices : 2
  Total Devices : 3
Preferred Minor : 125
    Persistence : Superblock is persistent

    Update Time : Mon Sep  2 04:12:09 2013
          State : clean 
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

$ mdadm --detail /dev/md127
/dev/md127:
        Version : 0.90
  Creation Time : Wed Oct 17 15:43:53 2012
     Raid Level : raid6
     Array Size : 46884321792 (44712.37 GiB 48009.55 GB)
  Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
   Raid Devices : 26
  Total Devices : 26
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Mon Sep  2 04:14:36 2013
          State : active, checking 
 Active Devices : 26
Working Devices : 26
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

   Check Status : 99% complete

           UUID : 8b53927c:8e569dae:c589cde9:18eb45a5 (local to host osdp-backup1.sjc1)
         Events : 0.120544

Get Checking Progress

$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md126 : active raid1 sdb4[1] sda4[0]
      1922262976 blocks [2/2] [UU]
      
md127 : active raid6 sdc1[0] sdw1[20] sdac1[22] sdu1[18] sdm1[10] sdj1[7] sdl1[9] sdab1[25] sdg1[4] sdo1[12] sdt1[17] sdq1[14] sdr1[15] sdp1[13] sdn1[11] sdz1[23] sdv1[19] sde1[2] sds1[16] sdx1[21] sdd1[1] sdi1[6] sdaa1[24] sdf1[3] sdh1[5] sdk1[8]
      46884321792 blocks level 6, 64k chunk, algorithm 2 [26/26] [UUUUUUUUUUUUUUUUUUUUUUUUUU]
      [===================>.]  check = 99.6% (1945938304/1953513408) finish=118.6min speed=1064K/sec
      
md125 : active raid1 sda2[0] sdb2[1] sdaj1[2](S)
      29296576 blocks [2/2] [UU]
      

Reference

Linux Man Page
Wiki
鳥哥的 Linux 私房菜