Sharing

2011年12月13日 星期二

SCST 使用方式


收集了一些還不錯的連結

http://wiki.alpinelinux.org/w/index.php?title=High_performance_SCST_iSCSI_Target_on_Linux_software_Raid&oldid=4999
介紹用 RAID 和 SCST 的搭配使用,不過後半段很清楚的描述從無到有把 iSCSI Target export 出去的流程


Add Target
-add_target -driver
Open Device
-open_dev -handler -attributes   這裡是填 handler 的屬性
Add LUN
-add_lun -driver -target [-group ] -device -attributes
Enable Target
-enable_target -driver
Enable Driver
-set_drv_attr iscsi -attributes enabled=1
Disable Driver
-set_drv_attr iscsi -attributes enabled=0
Disable Target
-disable_target -driver
Remove LUN
-rem_lun -driver -target [-group ]
Close Device
-close_dev -handler
Remove Target
-rem_target -driver


整個流程 Sample

# 加一個 Target
pjack@ubuntu3371:~$ sudo scstadmin -add_target iqn.2011-12.org.pjack:foo1 -driver iscsi

Collecting current configuration: done.


-> Making requested changes.
        -> Creating target 'iqn.2011-12.org.pjack:foo1' for driver 'iscsi': done.
        -> Done.

All done.


# 列出所有的 target 或是列出屬於 iscsi driver 的 Target, 看有沒有出現剛剛新增的
pjack@ubuntu3371:~$ sudo scstadmin -list_target

Collecting current configuration: done.

        Driver Target
        --------------------------------
        iscsi iqn.2011-12.org.pjack:foo1

All done.

pjack@ubuntu3371:~$ sudo scstadmin -list_driver iscsi

Collecting current configuration: done.

        Driver Target
        --------------------------------
        iscsi iqn.2011-12.org.pjack:foo1

All done.



# 看一下這個 Target 的一些屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_tgt_attr iqn.2011-12.org.pjack:foo1 -driver iscsi -nonkey

Collecting current configuration: done.

        Attribute                    Value                                                                       Writable      KEY
        --------------------------------------------------------------------------------------------------------------------------
        HeaderDigest                 None                                                                        Yes           No
        RspTimeout                   90                                                                          Yes           No
        InitialR2T                   No                                                                          Yes           No
        MaxBurstLength               1048576                                                                     Yes           No
        QueuedCommands               32                                                                          Yes           No
        MaxRecvDataSegmentLength     1048576                                                                     Yes           No
        tid                          2                                                                           No            No
        redirect                     < not set="">                                                                   Yes           No
        NopInTimeout                 30                                                                          Yes           No
        rel_tgt_id                   0                                                                           Yes           No
        NopInInterval                30                                                                          Yes           No
        ImmediateData                Yes                                                                         Yes           No
        MaxXmitDataSegmentLength     1048576                                                                     Yes           No
        cpu_mask                     ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff     Yes           No
        enabled                      0                                                                           Yes           No
        per_portal_acl               0                                                                           Yes           No
        addr_method                  PERIPHERAL                                                                  Yes           No
        FirstBurstLength             1048576                                                                     Yes           No
        MaxSessions                  0                                                                           Yes           No
        MaxOutstandingR2T            32                                                                          Yes           No
        DataDigest                   None                                                                        Yes           No
        comment                      < n a="">                                                                       Yes           No
        io_grouping_type             auto                                                                        Yes           No

        Dynamic attributes available
        ----------------------------
        allowed_portal
        OutgoingUser
        IncomingUser

        LUN CREATE attributes available
        -------------------------------
        read_only



# 新增一個 Device
pjack@ubuntu3371:~$ sudo scstadmin -open_dev disk01 -handler vdisk_blockio -attributes filename=/dev/rbd0

Collecting current configuration: done.


-> Making requested changes.
        -> Opening device 'disk01' using handler 'vdisk_blockio': done.
        -> Done.

All done.


# 列出所有的 Device
pjack@ubuntu3371:~$ sudo scstadmin -list_device

Collecting current configuration: done.

        Handler           Device
        ------------------------
        vdisk_nullio     -
        vdisk_fileio     -
        vdisk_blockio    disk01
        vcdrom           -

All done.

# 列出使用 vdisk_blockio handler 的 Device
pjack@ubuntu3371:~$ sudo scstadmin -list_handler vdisk_blockio

Collecting current configuration: done.

        Handler           Device
        ------------------------
        vdisk_blockio    disk01

All done.


# 列出這個 device 的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_dev_attr disk01 -nonkey

Collecting current configuration: done.

        Attribute             Value                                          Writable      KEY
        --------------------------------------------------------------------------------------
        usn                   27cddc71                                       Yes           No
        blocksize             512                                            No            No
        t10_dev_id            27cddc71-disk01                                Yes           No
        rotational            1                                              No            No
        nv_cache              0                                              No            No
        type_string           Direct-access device (e.g., magnetic disk)     No            No
        removable             0                                              No            No
        read_only             0                                              No            No
        resync_size           < n a="">                                      Yes           No
        threads_num           1                                              Yes           No
        size_mb               1024                                           No            No
        write_through         0                                              No            No
        thin_provisioned      0                                              No            No
        handler               vdisk_blockio                                  No            No
        filename              /dev/rbd0                                      Yes           Yes
        dump_prs              < n a="">                                      Yes           No
        exported              < n a="">                                      No            No
        threads_pool_type     per_initiator                                  Yes           No
        type                  0                                              No            No


# 新增一個 LUN
pjack@ubuntu3371:~$ sudo scstadmin -add_lun 0 -driver iscsi -target iqn.2011-12.org.pjack:foo1 -device disk01

Collecting current configuration: done.


-> Making requested changes.
        -> Adding device 'disk01' at LUN 0 to driver/target 'iscsi/iqn.2011-12.org.pjack:foo1': done.
        -> Driver/target is not a fibre channel target, ignoring.
        -> Done.

All done.

# 透過 list_target 再多加 driver 參數, 可以看到這個 target 的 LUN
pjack@ubuntu3371:~$ sudo scstadmin -list_target iqn.2011-12.org.pjack:foo1 -driver iscsi

Collecting current configuration: done.

Driver: iscsi
Target: iqn.2011-12.org.pjack:foo1

Assigned LUNs:

        LUN  Device
        -----------
        0    disk01


All done.

# 看 LUN 的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_lun_attr 0 -driver iscsi -target iqn.2011-12.org.pjack:foo1 -nonkey

Collecting current configuration: done.

        Attribute     Value      Writable      KEY
        ------------------------------------------
        read_only     0          No            No
        device        disk01     No            No

All done.


# Enable Target
pjack@ubuntu3371:~$ sudo scstadmin -enable_target iqn.2011-12.org.pjack:foo1 -driver iscsi

Collecting current configuration: done.


-> Making requested changes.
        -> Enabling driver/target 'iscsi/iqn.2011-12.org.pjack:foo1': done.
        -> Done.

All done.

# 多了一個 rel_tgt_id 的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_tgt_attr iqn.2011-12.org.pjack:foo1 -driver iscsi

Collecting current configuration: done.

        Attribute      Value     Writable      KEY
        ------------------------------------------
        rel_tgt_id     1         Yes           Yes

        Dynamic attributes available
        ----------------------------
        allowed_portal
        OutgoingUser
        IncomingUser

        LUN CREATE attributes available
        -------------------------------
        read_only

All done.

# 把 iscsi enable
pjack@ubuntu3371:~$ sudo scstadmin -set_drv_attr iscsi -attributes enabled=1 -noprompt

Collecting current configuration: done.


-> Making requested changes.
        -> Enabling driver 'iscsi': done.
        -> Done, 0 change(s) made.

All done.

# 從另一台看是否能找到
pjack@ubuntu64-33-7:~$ sudo iscsiadm -m discovery -t st -p 172.16.33.71
172.16.33.71:3260,1 iqn.2011-12.org.pjack:foo1

# Login
pjack@ubuntu64-33-7:~/Git/WiStor/Node/src/wistor/ext/itri/iscsi$ sudo iscsiadm -m node -T iqn.2011-12.org.pjack:foo1 -p 172.16.33.71 -l
Logging in to [iface: default, target: iqn.2011-12.org.pjack:foo1, portal: 172.16.33.71,3260]
Login to [iface: default, target: iqn.2011-12.org.pjack:foo1, portal: 172.16.33.71,3260]: successful


# 回到本機看一下連線 session
pjack@ubuntu3371:~$ sudo scstadmin -list_sessions

Collecting current configuration: done.

Driver/Target: iscsi/iqn.2011-12.org.pjack:foo1

        Session: iqn.1993-08.org.debian:01:4e712cd57f99

        Attribute                    Value                                      Writable      KEY
        -----------------------------------------------------------------------------------------
        write_cmd_count              0                                          Yes           No
        HeaderDigest                 None                                       Yes           No
        sid                          10000023d0200                              Yes           No
        read_cmd_count               118                                        Yes           No
        bidi_io_count_kb             0                                          Yes           No
        commands                     0                                          Yes           No
        InitialR2T                   No                                         Yes           No
        MaxBurstLength               1048576                                    Yes           No
        MaxRecvDataSegmentLength     1048576                                    Yes           No
        active_commands              0                                          Yes           No
        none_cmd_count               7                                          Yes           No
        unknown_cmd_count            0                                          Yes           No
        ImmediateData                Yes                                        Yes           No
        MaxXmitDataSegmentLength     262144                                     Yes           No
        reinstating                  0                                          Yes           No
        FirstBurstLength             262144                                     Yes           No
        read_io_count_kb             354                                        Yes           No
        MaxOutstandingR2T            1                                          Yes           No
        bidi_cmd_count               0                                          Yes           No
        write_io_count_kb            0                                          Yes           No
        DataDigest                   None                                       Yes           No
        initiator_name               iqn.1993-08.org.debian:01:4e712cd57f99     Yes           No
        force_close                                                        Yes           No

# Logout
pjack@ubuntu64-33-7:~/Git/WiStor/Node/src/wistor/ext/itri/iscsi$ sudo iscsiadm -m node -T iqn.2011-12.org.pjack:foo1 -p 172.16.33.71 -u
Logging out of session [sid: 2, target: iqn.2011-12.org.pjack:foo1, portal: 172.16.33.71,3260]
Logout of [sid: 2, target: iqn.2011-12.org.pjack:foo1, portal: 172.16.33.71,3260]: successful

# disable target
pjack@ubuntu3371:~$ sudo scstadmin -disable_target iqn.2011-12.org.pjack:foo1 -driver iscsi -noprompt

Collecting current configuration: done.


-> Making requested changes.
        -> Disabling driver/target 'iscsi/iqn.2011-12.org.pjack:foo1': done.
        -> Done.

All done.

# remove lun
pjack@ubuntu3371:~$ sudo scstadmin -rem_lun 0 -driver iscsi -target iqn.2011-12.org.pjack:foo1 -device disk01 -noprompt

Collecting current configuration: done.


-> Making requested changes.
        -> Removing LUN 0 from driver/target 'iscsi/iqn.2011-12.org.pjack:foo1': done.
        -> Driver/target is not a fibre channel target, ignoring.
        -> Done.

All done.


# close the device
pjack@ubuntu3371:~$ sudo scstadmin -close_dev disk01 -handler vdisk_blockio -noprompt

Collecting current configuration: done.


-> Making requested changes.
        -> Closing device 'disk01' using handler 'vdisk_blockio': done.
        -> Driver/target is not a fibre channel target, ignoring.
        -> Driver/target is not a fibre channel target, ignoring.
        -> Done.

All done.

# remove the target
pjack@ubuntu3371:~$ sudo scstadmin -rem_target iqn.2011-12.org.pjack:foo1 -driver iscsi -noprompt

Collecting current configuration: done.


-> Making requested changes.
        -> Removing virtual target 'iqn.2011-12.org.pjack:foo1' from driver 'iscsi': done.
        -> Driver/target is not a fibre channel target, ignoring.
        -> Done.

All done.



http://iscsi-scst.sourceforge.net/SCST_Gentoo_HOWTO.txt
這篇裡面有不少設定 scstadmin 的範例

這篇大部份不透過 scstadmin 設定

這篇是滿早期的文章,有些已經不適用,包括安裝的 ppa 不能用,其它的部份則是可參考



這應該是把 scst 寫的最詳細的資料,而且有很多專有名詞的解釋及用法,改天應該要好好研究一下。
裡面有提到 /sys/kernel/scst_tgt/ 裡面的目錄結構,如果要查看每一個 target 有那一些連線,就要查看這個目錄,
/sys/kernel/scst_tgt/targets/iscsi//sessions

這是 scstadmin 官方的 README


https://git.ipxe.org/mirror/scst/.git/blob/master:/scstadmin/scstadmin.sysfs/scstadmin
這份有整理好的 scstadmin 指令


General Operations
-write_config < file>
Writes the current configuration to < file>.
-clear_config
Clear all SCST configuration.
Query Operations
-list_handler
List all available handlers
-list_handler [< hndlr>]
List all available device under specific handler
-list_driver
List all available drivers
-list_driver [< driver>]
List all available target under specific driver
-list_target
List all available targets
-list_target [< target>][-driver < driver>]
List all available LUN under specific target
List Attribute Operations
-list_scst_attr
List all attributes for SCST
-list_hnd_attr < hndlr>
List all attributes for a given handler.
-list_dev_attr < device>
List all attributes for a given device.
-list_drv_attr < driver>
List all attributes for a given driver.
-list_lun_attr < lun> -driver < driver> -target < target> [-group < group>]
List all attributes for a driver/target/lun
-list_sessions
List all current initiator sessions


# 看一下有支援那些 handler
pjack@ubuntu3371:~$ sudo scstadmin -list_handler

Collecting current configuration: done.

        Handler
        -------------
        vdisk_fileio
        vdisk_blockio
        vdisk_nullio
        vcdrom

All done.

# 看一下有支援那些 driver
pjack@ubuntu3371:~$ sudo scstadmin -list_driver

Collecting current configuration: done.

        Driver
        -----
        iscsi

All done.

# 看一下有 blockio 支援的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_hnd_attr vdisk_blockio -nonkey

Collecting current configuration: done.

        Attribute       Value                                          Writable      KEY
        --------------------------------------------------------------------------------
        type_string     Direct-access device (e.g., magnetic disk)     No            No
        disk01          < n a="">                                      Yes           No
        trace_level     out_of_mem                                     Yes           No
                        minor                                          Yes           No
                        pid                                            Yes           No
                        line                                           Yes           No
                        function                                       Yes           No
                        special                                        Yes           No
                        mgmt                                           Yes           No
                        mgmt_dbg                                       Yes           No
        type            0                                              No            No

        Possible trace levels:
          (use trace_level="add < level>", none, all or default to set):
                debug, function, line, pid, entryexit,
                buff, mem, sg, out_of_mem, special,
                scsi, mgmt, minor, mgmt_dbg, scsi_serializing,
                retry, recv_bot, send_bot, recv_top, pr,
                send_top, order,

        Device CREATE attributes available
        ----------------------------------
        blocksize
        write_through
        rotational
        nv_cache
        thin_provisioned
        filename
        removable
        read_only

# 看一下有 fileio 支援的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_hnd_attr vdisk_fileio -nonkey

Collecting current configuration: done.

        Attribute       Value                                          Writable      KEY
        --------------------------------------------------------------------------------
        type_string     Direct-access device (e.g., magnetic disk)     No            No
        trace_level     out_of_mem                                     Yes           No
                        minor                                          Yes           No
                        pid                                            Yes           No
                        line                                           Yes           No
                        function                                       Yes           No
                        special                                        Yes           No
                        mgmt                                           Yes           No
                        mgmt_dbg                                       Yes           No
        type            0                                              No            No

        Possible trace levels:
          (use trace_level="add < level>", none, all or default to set):
                debug, function, line, pid, entryexit,
                buff, mem, sg, out_of_mem, special,
                scsi, mgmt, minor, mgmt_dbg, scsi_serializing,
                retry, recv_bot, send_bot, recv_top, pr,
                send_top, order,

        Device CREATE attributes available
        ----------------------------------
        o_direct
        blocksize
        write_through
        rotational
        nv_cache
        thin_provisioned
        filename
        removable
        read_only

All done.

# 看一下有 nullio 支援的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_hnd_attr vdisk_nullio -nonkey

Collecting current configuration: done.

        Attribute       Value                                          Writable      KEY
        --------------------------------------------------------------------------------
        type_string     Direct-access device (e.g., magnetic disk)     No            No
        trace_level     out_of_mem                                     Yes           No
                        minor                                          Yes           No
                        pid                                            Yes           No
                        line                                           Yes           No
                        function                                       Yes           No
                        special                                        Yes           No
                        mgmt                                           Yes           No
                        mgmt_dbg                                       Yes           No
        type            0                                              No            No

        Possible trace levels:
          (use trace_level="add < level>", none, all or default to set):
                debug, function, line, pid, entryexit,
                buff, mem, sg, out_of_mem, special,
                scsi, mgmt, minor, mgmt_dbg, scsi_serializing,
                retry, recv_bot, send_bot, recv_top, pr,
                send_top, order,

        Device CREATE attributes available
        ----------------------------------
        removable
        blocksize
        read_only
        rotational

# 看一下有 cdrom 支援的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_hnd_attr vcdrom -nonkey

Collecting current configuration: done.

        Attribute       Value                Writable      KEY
        ------------------------------------------------------
        type_string     CD-ROM device        No            No
        trace_level     out_of_mem           Yes           No
                        minor                Yes           No
                        pid                  Yes           No
                        line                 Yes           No
                        function             Yes           No
                        special              Yes           No
                        mgmt                 Yes           No
                        mgmt_dbg             Yes           No
        type            5                    No            No

        Possible trace levels:
          (use trace_level="add < level>", none, all or default to set):
                debug, function, line, pid, entryexit,
                buff, mem, sg, out_of_mem, special,
                scsi, mgmt, minor, mgmt_dbg, scsi_serializing,
                retry, recv_bot, send_bot, recv_top, pr,
                send_top, order,

All done.

# 看一下有 iscsi driver 支援的屬性
pjack@ubuntu3371:~$ sudo scstadmin -list_drv_attr iscsi --nonkey

Collecting current configuration: done.

        Attribute            Value          Writable      KEY
        -----------------------------------------------------
        open_state           open           No            No
        isns_entity_name     < not set="">  Yes           No
        version              3.0.0-pre1     No            No
        trace_level          out_of_mem     Yes           No
                             minor          Yes           No
                             pid            Yes           No
                             line           Yes           No
                             function       Yes           No
                             special        Yes           No
                             mgmt           Yes           No
                             mgmt_dbg       Yes           No
                             conn           Yes           No
        iSNSServer           < not set="">  Yes           No
        enabled              0              Yes           No

        Possible trace levels:
          (use trace_level="add < level>", none, all or default to set):
                debug, function, line, pid, entryexit,
                buff, mem, sg, out_of_mem, special,
                scsi, mgmt, minor, mgmt_dbg, scsi_serializing,
                retry, recv_bot, send_bot, recv_top, pr,
                send_top, d_write, conn, conn_dbg, iov,
                pdu, net_page,

        Dynamic attributes available
        ----------------------------
        OutgoingUser
        IncomingUser



scstadmin script 內容


把原本的 IET 改接到 SCST,另外也有提到一些 performance fine tune 的建議,另外也有 /sys/kernel/scst_tgt/ 的目錄結構


http://linux-iscsi.org/index.php/ALUA
對於 ALUA 的解釋

沒有留言: