Export Ceph RBD with iSCSI

原本要使用 LIO , 但因為 Linux Kernel 2.6.38 似乎還沒把 iSCSI 整合進來, 所以就改用 iET (iSCSI Enterprise Target) 來試驗整個 Ceph 提供的 RBD (Rados Block Device)

如果對 iSCSI 還很不了解, 可以先從這兩篇開始了解

• http://linux.vbird.org/linux_server/0460iscsi.php
• http://benjr.tw/node/74
• http://babyface2.com/NetAdmin/10200611iSCSI/
• 
  

至於設定的部份可以參考這幾篇

• http://jiangshuiy.iteye.com/blog/518478
• http://163.20.124.8/~t301000/wordpress/?p=388
• http://blogs.yyes.chc.edu.tw/post/2/3412
• http://www.howtoforge.com/using-iscsi-on-ubuntu-10.04-initiator-and-target
• http://www.cyberciti.biz/tips/howto-setup-linux-iscsi-target-sanwith-tgt.html

第一步先從簡單的 partition 開始熟悉 iSCSI

設定 Target

root@ubuntu1104-64-5:/dev$ apt-get install iscsitarget
root@ubuntu1104-64-5:/dev$ apt-get install open-iscsi

安裝完出現警告訊息： iscsitarget not enabled in "/etc/default/iscsitarget", not starting... ... (warning). 這個問題需編輯設定檔：/etc/default/iscsitarget 將「ISCSITARGET_ENABLE=false」改為「ISCSITARGET_ENABLE=true」，iSCSI Target 服務才能作用。

接下來要設定 iET 的 /etc/iet/ietd.conf

iSNSServer 172.16.33.5
iSNSAccessControl No
Target iqn.2011-09.com.example:storage.lun1
# ubuntu1104--64--5-lvol1 是之前就切好的一塊 partition, 我們用fileio 來把它 release 出去
# ScsiId 及 ScsiSN 現在不知道是做什麼的, 暫時不管它
Lun 0 Path=/dev/mapper/ubuntu1104--64--5-lvol1,Type=fileio,ScsiId=xyz,ScsiSN=xyz
 

root@ubuntu1104-64-5:/dev$ /etc/init.d/iscsitarget restart
 * Removing iSCSI enterprise target devices:
   ...done.
 * Stopping iSCSI enterprise target service:
   ...done.
 * Removing iSCSI enterprise target modules:
   ...done.
 * Starting iSCSI enterprise target service
   ...done.
   ...done.

然後設定允許的連線 /etc/iet/initiators.allow, 末行顯示 ALL ALL 表示預設開放所有來源與目的連線，測試初期就先保持這樣了。

設定 Initiator

先把 /etc/iscsi/iscsid.conf 內的 node.startup 設成 automatic

# 重啟 initiator 的服務
root@ubuntu1104-64-6:/etc/iscsi$ /etc/init.d/open-iscsi restart
* Disconnecting iSCSI targets
...done.
* Stopping iSCSI initiator service
...done.
* Starting iSCSI initiator service iscsid
...done.
* Setting up iSCSI targets
...done.

# 然後尋找我們剛剛開出來的 iSCSI
root@ubuntu1104-64-6:/etc/iscsi$ iscsiadm -m discovery -t st -p 172.16.33.5
172.16.33.5:3260,1 iqn.2011-09.com.example:storage.lun1

root@ubuntu1104-64-6:/etc/iscsi# iscsiadm -m node
172.16.33.5:3260,1 iqn.2011-09.com.example:storage.lun1

# 如果有連結到, 應該會在這邊看到它的目錄
root@ubuntu1104-64-6:/etc/iscsi$ ll /etc/iscsi/nodes/
total 12
drw------- 3 root root 4096 2011-09-20 16:06 ./
drwxr-xr-x 5 root root 4096 2011-09-20 15:20 ../
drw------- 3 root root 4096 2011-09-20 16:06 iqn.2011-09.com.example:storage.lun1/

# 登入到這個節點, 不過基本上前面會自動登入, 所以在這個步驟可能會看到"already exists"
root@ubuntu1104-64-6:/etc/iscsi$ iscsiadm -m node -T iqn.2011-09.com.example:storage.lun1 -p 172.16.33.5 -l
Logging in to [iface: default, target: iqn.2011-09.com.example:storage.lun1, portal: 172.16.33.5,3260]
Login to [iface: default, target: iqn.2011-09.com.example:storage.lun1, portal: 172.16.33.5,3260]: successful

# 用 fdisk 看一下, 會看到多了個 sdb 這個 device, 然後有一個 sdb0 partition
root@ubuntu1104-64-6:/etc/iscsi$ fdisk -l
Disk /dev/sdb: 53.7 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe4a1139a

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       51200    52428784   83  Linux


# 格式化這個 partition
root@ubuntu1104-64-6:/etc/iscsi$ mkfs.ext4 /dev/sdb1
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3276800 inodes, 13107196 blocks
655359 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

# mount 起來
root@ubuntu1104-64-6:/data$ mount /dev/sdb1 /data/scsi

# 看一下所有 mount 的狀況
root@ubuntu1104-64-6:/data$ mount
/dev/mapper/ubuntu1104--64--6-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
/dev/mapper/ubuntu1104--64--6-lvol0 on /data/osd.2 type btrfs (rw,noatime)
/dev/mapper/ubuntu1104--64--6-lvol1 on /data/osd.3 type btrfs (rw,noatime)
/dev/sdb1 on /data/scsi type ext4 (rw)

# 看一下 partition 的使用狀況
root@ubuntu1104-64-6:/data$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/ubuntu1104--64--6-root
47328184   2595704  42328336   6% /
none                  12358244       224  12358020   1% /dev
none                  12366300         0  12366300   0% /dev/shm
none                  12366300        48  12366252   1% /var/run
none                  12366300         0  12366300   0% /var/lock
/dev/sda1               233191     45272    175478  21% /boot
/dev/mapper/ubuntu1104--64--6-lvol0
52428800   1033620  49277220   3% /data/osd.2
/dev/mapper/ubuntu1104--64--6-lvol1
52428800   1034584  49276440   3% /data/osd.3
/dev/sdb1             51606124    184136  48800552   1% /data/scsi

如果要把 iscsi 的連結斷掉

iscsiadm -m node -T iqn.2011-09.com.example:storage.lun1 -p 172.16.33.5 -u

Create Ceph RBD

參考連結: http://ceph.newdream.net/wiki/RBD

# 製造一個 rbd , 大小為 5G
root@ubuntu1104-64-5:/dev/rbd/rbd$ rbd create goo --size 5120

# 看目前在 rbd 內的 list
root@ubuntu1104-64-5:/dev/rbd/rbd$ rbd list
foo

# 從 rbd 去觀察 foo
root@ubuntu1104-64-5:~$ rbd info foo
rbd image 'foo':
        size 5120 MB in 1280 objects
        order 22 (4096 KB objects)
        block_name_prefix: rb.0.3
        parent:  (pool -1)

# 從 rados 去看一下 rbd 的狀況, 可以發現多了一個 foo.rbd
root@ubuntu1104-64-5:~$ rados ls -p rbd
foo.rbd
rb.0.1.000000000000
rb.0.1.000000000001
rbd_directory
rbd_info

root@ubuntu1104-64-5:~$ modprobe rbd

# 把 rbd 加到系統內, 等下才看的到這個 device
root@ubuntu1104-64-5:/dev$ echo "172.16.33.5 name=admin,secret=AQDeRGdOMNL3MhAAuzvelwICjpYhLIk7IMcX2g== rbd foo" > /sys/bus/rbd/add
root@ubuntu1104-64-5:/dev$ mknod /dev/rbd0 b 254 0


# 看一下 rbd device 有沒有出現
root@ubuntu1104-64-5:~$ ls /sys/bus/rbd/devices
0
root@ubuntu1104-64-5:~$ ls /dev/rbd/rbd
foo:0

# format rbd0
root@ubuntu1104-64-5:/dev$ mkfs -t ext3 /dev/rbd0
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310720 blocks
65536 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information:
done

# mount 起來
root@ubuntu1104-64-5:/dev$ mount -t ext3 /dev/rbd0 /mnt

root@ubuntu1104-64-5:/dev$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu1104--64--5-root
                       46G  6.6G   37G  16% /
none                   12G  228K   12G   1% /dev
none                   12G     0   12G   0% /dev/shm
none                   12G   64K   12G   1% /var/run
none                   12G     0   12G   0% /var/lock
/dev/sda1             228M   45M  172M  21% /boot
/dev/mapper/ubuntu1104--64--5-lvol2
                       50G  1.1G   47G   3% /data/osd.0
/dev/mapper/ubuntu1104--64--5-lvol0
                       50G  1.1G   47G   3% /data/osd.1
/dev/rbd0             5.0G  139M  4.6G   3% /mnt

# 測試完畢, 再將它 umount
root@ubuntu1104-64-5:/dev$ umount /mnt

Export RBD via iSCSI

可以參考 http://ceph.newdream.net/wiki/ISCSI
首先改 /etc/iet/ietd.conf 裡面的設定

Target iqn.2011-09.net.newdream.ceph:rados.iscsi.001
        # 記得要使用 blockio, 而不是 fileio
        Lun 0 Path=/dev/rbd0,Type=blockio

然後分別重啟 target 及 Initiator, 步驟和上面的一樣,

root@ubuntu1104-64-7:~$ fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c4797

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          32      248832   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              32       60802   488134657    5  Extended
/dev/sda5              32       60802   488134656   8e  Linux LVM

Disk /dev/sdb: 5368 MB, 5368709120 bytes
166 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 10292 * 512 = 5269504 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

# 因為是用 blockio 當成一個 device export 出來, 所以上面沒有任何 partition
Disk /dev/sdb doesn't contain a valid partition table

# 直接 mount 起來
root@ubuntu1104-64-7:~$ mount /dev/sdb /mnt

# 看一下狀況, 多了一個 5G 的 sdb
root@ubuntu1104-64-7:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu1104--64--7-root
                      435G   11G  402G   3% /
none                   12G  216K   12G   1% /dev
none                   12G     0   12G   0% /dev/shm
none                   12G   60K   12G   1% /var/run
none                   12G     0   12G   0% /var/lock
/dev/sda1             228M   45M  172M  21% /boot
/dev/sdb              5.0G  139M  4.6G   3% /mnt

大致上這樣就完成啦! 收工~