Sharing

2012年10月2日 星期二

Openstack Folsom - Installation of Cinder with Ceph

Openstack Folsom Release

前幾天 Openstack Folsom 正式 Release
Release Software Site
http://www.openstack.org/software/folsom/
Release Note
http://wiki.openstack.org/ReleaseNotes/Folsom
Architecture
http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/

這次的 Release 多了 Quantum 及 Cinder 兩個 project. Quantum 是將 Open vSwitch 整合進 Openstack, 加強了原本 Openstack 對於網路虛擬化不足的地方. Cinder 則是將原來 nova-volume 拆出來變成一個獨立的 module, 我想原因不外乎背後有太多 Storage 的大廠想要整合進 Openstack, ex: nexenta、netapp... 所以把這塊的 API 獨立出來, 和各家整合才會順利.

PPA of openstack testing
root@ubuntu12:~$ apt-get install -y python-software-properties
root@ubuntu12:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-trunk-testing
root@ubuntu12:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-deps-staging
root@ubuntu12:~$ apt-get update && apt-get -y dist-upgrade


PPA of Ubuntu Cloud
root@ubuntu12:~$ add-apt-repository ppa:ubuntu-cloud-archive/folsom-staging
You are about to add the following PPA to your system:

 More info: https://launchpad.net/~ubuntu-cloud-archive/+archive/folsom-staging
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpdzKHU_/secring.gpg' created
gpg: keyring `/tmp/tmpdzKHU_/pubring.gpg' created
gpg: requesting key 9F68104E from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpdzKHU_/trustdb.gpg: trustdb created
gpg: key 9F68104E: public key "Launchpad PPA for Ubuntu Cloud Archive Team" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

Openstack 的更新速度會比 Ubuntu Cloud 來的快, 以下的流程是以 Opentack 的 PPA 來進行

Installation

這次我先從 Cinder 下手, 並且目標是要把它和 Ceph 整合在一起. 原本想要單獨裝 Cinder 就好, 但研究 Cinder Source Code 後, 發現他目前不支援 noauth 的模式, 所以就一定要裝 Keystone, 所以整體來說重要的 Component 有: Cinder + Keystone + MySQL + CEPH

首先先裝基本的套件, MySQL 及 RabbitMQ

MySQL Installation

root@ubuntu12:~$ apt-get -y install mysql-server python-mysqldb
root@ubuntu12:~$ sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf
root@ubuntu12:~$ service mysql restart
root@ubuntu12:~$ mysql -u root -ppassword
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 36
Server version: 5.5.24-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE cinder;
Query OK, 1 row affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE DATABASE keystone;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)


RabbitMQ Installation

root@ubuntu12:~$ sudo apt-get -y install rabbitmq-server
root@ubuntu12:~$ rabbitmqctl change_password guest password
Changing password for user "guest" ...
...done.

Keystone Installation

root@ubuntu12:~$ apt-get -y install keystone python-keystone python-keystoneclient
root@ubuntu12:~$ dpkg -l | grep keystone
ii  keystone                                        2012.2+git201209252030~precise-0ubuntu1     OpenStack identity service - Daemons
ii  python-keystone                                 2012.2+git201209252030~precise-0ubuntu1     OpenStack identity service - Python library
ii  python-keystoneclient                           1:0.1.3.19+git201210011900~precise-0ubuntu1 Client libary for Openstack Keystone API

# 修改 keyston.conf 的內容, 隨機取一個 admin_token
root@ubuntu12:~$ cat /etc/keystone/keystone.conf                                                              [DEFAULT]
# A "shared secret" between keystone and other openstack services
admin_token = password

# The IP address of the network interface to listen on
bind_host = 0.0.0.0

# The port number which the public service listens on
public_port = 5000

# The port number which the public admin listens on
admin_port = 35357

# The port number which the OpenStack Compute service listens on
compute_port = 8774

# === Logging Options ===
# Print debugging output
verbose = True

# Print more verbose output
# (includes plaintext request logging, potentially including passwords)
debug = True

...

[sql]
# The SQLAlchemy connection string used to connect to the database
# connection = sqlite:////var/lib/keystone/keystone.db
connection = mysql://keystone:password@localhost:3306/keystone

# the timeout before idle sql connections are reaped
idle_timeout = 200
...

root@ubuntu12:~$ service keystone restart
root@ubuntu12:~$ keystone-manage db_sync

# 編輯一個檔, 設定一些等下會用到的區域變數, 並且匯到 .bashrc 之中, 下次再進來就不需要重新設定
root@ubuntu12:~# cat novarc
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL="http://localhost:5000/v2.0/"
export SERVICE_ENDPOINT="http://localhost:35357/v2.0"
export SERVICE_TOKEN=password
root@ubuntu12:~$ source novarc
root@ubuntu12:~$ echo "source novarc">>.bashrc

接下來下載兩個網路上別人提供好的 script, 裡面預設的 token 也是 "password", 如果你要使用別的 token, 請記得要修改 script 的內容
root@ubuntu12:~$ wget https://raw.github.com/EmilienM/openstack-folsom-guide/master/scripts/keystone-data.sh
root@ubuntu12:~$ wget https://raw.github.com/EmilienM/openstack-folsom-guide/master/scripts/keystone-endpoints.sh
root@ubuntu12:~$ chmod a+x *.sh
# 記得把 keystone-endpoints.sh 內的 MASTER 改成你自己的 ip
root@ubuntu12:~$ less keystone-endpoints.sh
....
# other definitions
MASTER="172.17.123.13"
....

root@ubuntu12:~$ ./keystone-data.sh
root@ubuntu12:~$ ./keystone-endpoints.sh
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |    OpenStack Compute Service     |
|      id     | 597fff05550043efb530ab05fa85d818 |
|     name    |               nova               |
|     type    |             compute              |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |     OpenStack Volume Service     |
|      id     | 27e3539fea104d159fcc7ec9766ac8b3 |
|     name    |              cinder              |
|     type    |              volume              |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |     OpenStack Image Service      |
|      id     | b7047156ef81464b8c6754fc7994ecea |
|     name    |              glance              |
|     type    |              image               |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |    OpenStack Storage Service     |
|      id     | 86e08436f65944be8ab0e23657a9d3e2 |
|     name    |              swift               |
|     type    |           object-store           |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |        OpenStack Identity        |
|      id     | 4ec3ab138f2a4bd2800b5a4a5d407ef1 |
|     name    |             keystone             |
|     type    |             identity             |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |      OpenStack EC2 service       |
|      id     | 15bb5a5a0c3d446e8b3bfa293df3b10e |
|     name    |               ec2                |
|     type    |               ec2                |
+-------------+----------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |   OpenStack Networking service   |
|      id     | cacfce49cb2141a1ac48b3e31cef5c01 |
|     name    |             quantum              |
|     type    |             network              |
+-------------+----------------------------------+
+-------------+--------------------------------------------+
|   Property  |                   Value                    |
+-------------+--------------------------------------------+
|   adminurl  | http://172.17.123.13:8774/v2/$(tenant_id)s |
|      id     |      090f849da5a94dcb817aa340b39eb83c      |
| internalurl | http://172.17.123.13:8774/v2/$(tenant_id)s |
|  publicurl  | http://172.17.123.13:8774/v2/$(tenant_id)s |
|    region   |                 RegionOne                  |
|  service_id |      597fff05550043efb530ab05fa85d818      |
+-------------+--------------------------------------------+
+-------------+--------------------------------------------+
|   Property  |                   Value                    |
+-------------+--------------------------------------------+
|   adminurl  | http://172.17.123.13:8776/v1/$(tenant_id)s |
|      id     |      6363131b18974040a3dd1276ddc2c72e      |
| internalurl | http://172.17.123.13:8776/v1/$(tenant_id)s |
|  publicurl  | http://172.17.123.13:8776/v1/$(tenant_id)s |
|    region   |                 RegionOne                  |
|  service_id |      27e3539fea104d159fcc7ec9766ac8b3      |
+-------------+--------------------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
|   adminurl  |   http://172.17.123.13:9292/v2   |
|      id     | 1db94d180acb4d64adab45c3116812cb |
| internalurl |   http://172.17.123.13:9292/v2   |
|  publicurl  |   http://172.17.123.13:9292/v2   |
|    region   |            RegionOne             |
|  service_id | b7047156ef81464b8c6754fc7994ecea |
+-------------+----------------------------------+
+-------------+-------------------------------------------------+
|   Property  |                      Value                      |
+-------------+-------------------------------------------------+
|   adminurl  |           http://172.17.123.13:8080/v1          |
|      id     |         b9006b5f4fc64e4d854c171ea157b7b4        |
| internalurl | http://172.17.123.13:8080/v1/AUTH_$(tenant_id)s |
|  publicurl  | http://172.17.123.13:8080/v1/AUTH_$(tenant_id)s |
|    region   |                    RegionOne                    |
|  service_id |         86e08436f65944be8ab0e23657a9d3e2        |
+-------------+-------------------------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
|   adminurl  | http://172.17.123.13:35357/v2.0  |
|      id     | faf6ca350f4842799b3801d0b7571a59 |
| internalurl |  http://172.17.123.13:5000/v2.0  |
|  publicurl  |  http://172.17.123.13:5000/v2.0  |
|    region   |            RegionOne             |
|  service_id | 4ec3ab138f2a4bd2800b5a4a5d407ef1 |
+-------------+----------------------------------+
+-------------+------------------------------------------+
|   Property  |                  Value                   |
+-------------+------------------------------------------+
|   adminurl  | http://172.17.123.13:8773/services/Admin |
|      id     |     da745295e7ef419682d45516456047c5     |
| internalurl | http://172.17.123.13:8773/services/Cloud |
|  publicurl  | http://172.17.123.13:8773/services/Cloud |
|    region   |                RegionOne                 |
|  service_id |     15bb5a5a0c3d446e8b3bfa293df3b10e     |
+-------------+------------------------------------------+
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
|   adminurl  |    http://172.17.123.13:9696/    |
|      id     | 8b570092f8614cba98fe227de6e65e27 |
| internalurl |    http://172.17.123.13:9696/    |
|  publicurl  |    http://172.17.123.13:9696/    |
|    region   |            RegionOne             |
|  service_id | cacfce49cb2141a1ac48b3e31cef5c01 |
+-------------+----------------------------------+

安裝好之後驗証一下

root@ubuntu12:~$ keystone endpoint-list
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
|                id                |   region  |                    publicurl                    |                   internalurl                   |                  adminurl                  |
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
| 090f849da5a94dcb817aa340b39eb83c | RegionOne |    http://172.17.123.13:8774/v2/$(tenant_id)s   |    http://172.17.123.13:8774/v2/$(tenant_id)s   | http://172.17.123.13:8774/v2/$(tenant_id)s |
| 1db94d180acb4d64adab45c3116812cb | RegionOne |           http://172.17.123.13:9292/v2          |           http://172.17.123.13:9292/v2          |        http://172.17.123.13:9292/v2        |
| 6363131b18974040a3dd1276ddc2c72e | RegionOne |    http://172.17.123.13:8776/v1/$(tenant_id)s   |    http://172.17.123.13:8776/v1/$(tenant_id)s   | http://172.17.123.13:8776/v1/$(tenant_id)s |
| 8b570092f8614cba98fe227de6e65e27 | RegionOne |            http://172.17.123.13:9696/           |            http://172.17.123.13:9696/           |         http://172.17.123.13:9696/         |
| b9006b5f4fc64e4d854c171ea157b7b4 | RegionOne | http://172.17.123.13:8080/v1/AUTH_$(tenant_id)s | http://172.17.123.13:8080/v1/AUTH_$(tenant_id)s |        http://172.17.123.13:8080/v1        |
| da745295e7ef419682d45516456047c5 | RegionOne |     http://172.17.123.13:8773/services/Cloud    |     http://172.17.123.13:8773/services/Cloud    |  http://172.17.123.13:8773/services/Admin  |
| faf6ca350f4842799b3801d0b7571a59 | RegionOne |          http://172.17.123.13:5000/v2.0         |          http://172.17.123.13:5000/v2.0         |      http://172.17.123.13:35357/v2.0       |
+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+--------------------------------------------+
root@ubuntu12:~$ sudo apt-get install -y curl openssl
root@ubuntu12:~$ curl -d '{"auth": {"tenantName": "admin", "passwordCredentials":{"username": "admin", "password": "pass
word"}}}' -H "Content-type: application/json" http://172.17.123.13:35357/v2.0/tokens | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2946    0  2844  100   102  18698    670 --:--:-- --:--:-- --:--:-- 18834
{
    "access": {
        "metadata": {
            "is_admin": 0,
            "roles": [
                "5660593decab401bbc8f13aa8b19dc23",
                "833b1617cb404466ba8546c8194f8ad6",
                "7ec43588e3a14b69ae2278334bee1423"
            ]
        },
        "serviceCatalog": [
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:8774/v2/4b310104ee3345fd988fe16dd2f1f79d",
                        "id": "090f849da5a94dcb817aa340b39eb83c",
                        "internalURL": "http://172.17.123.13:8774/v2/4b310104ee3345fd988fe16dd2f1f79d",
                        "publicURL": "http://172.17.123.13:8774/v2/4b310104ee3345fd988fe16dd2f1f79d",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "nova",
                "type": "compute"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:9696/",
                        "id": "8b570092f8614cba98fe227de6e65e27",
                        "internalURL": "http://172.17.123.13:9696/",
                        "publicURL": "http://172.17.123.13:9696/",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "quantum",
                "type": "network"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:9292/v2",
                        "id": "1db94d180acb4d64adab45c3116812cb",
                        "internalURL": "http://172.17.123.13:9292/v2",
                        "publicURL": "http://172.17.123.13:9292/v2",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "glance",
                "type": "image"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:8776/v1/4b310104ee3345fd988fe16dd2f1f79d",
                        "id": "6363131b18974040a3dd1276ddc2c72e",
                        "internalURL": "http://172.17.123.13:8776/v1/4b310104ee3345fd988fe16dd2f1f79d",
                        "publicURL": "http://172.17.123.13:8776/v1/4b310104ee3345fd988fe16dd2f1f79d",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "cinder",
                "type": "volume"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:8773/services/Admin",
                        "id": "da745295e7ef419682d45516456047c5",
                        "internalURL": "http://172.17.123.13:8773/services/Cloud",
                        "publicURL": "http://172.17.123.13:8773/services/Cloud",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "ec2",
                "type": "ec2"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:8080/v1",
                        "id": "b9006b5f4fc64e4d854c171ea157b7b4",
                        "internalURL": "http://172.17.123.13:8080/v1/AUTH_4b310104ee3345fd988fe16dd2f1f79d",
                        "publicURL": "http://172.17.123.13:8080/v1/AUTH_4b310104ee3345fd988fe16dd2f1f79d",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "swift",
                "type": "object-store"
            },
            {
                "endpoints": [
                    {
                        "adminURL": "http://172.17.123.13:35357/v2.0",
                        "id": "faf6ca350f4842799b3801d0b7571a59",
                        "internalURL": "http://172.17.123.13:5000/v2.0",
                        "publicURL": "http://172.17.123.13:5000/v2.0",
                        "region": "RegionOne"
                    }
                ],
                "endpoints_links": [],
                "name": "keystone",
                "type": "identity"
            }
        ],
        "token": {
            "expires": "2012-10-03T06:55:20Z",
            "id": "5f2797bdf8fd4380bfe919a05b01772e",
            "tenant": {
                "description": null,
                "enabled": true,
                "id": "4b310104ee3345fd988fe16dd2f1f79d",
                "name": "admin"
            }
        },
        "user": {
            "id": "c7f17dfd798242fc9065afd2ea251a6d",
            "name": "admin",
            "roles": [
                {
                    "name": "admin"
                },
                {
                    "name": "KeystoneAdmin"
                },
                {
                    "name": "KeystoneServiceAdmin"
                }
            ],
            "roles_links": [],
            "username": "admin"
        }
    }
}


CEPH Installation

root@ubuntu12:~$ wget -q -O - https://raw.github.com/ceph/ceph/master/keys/release.asc | apt-key add -
OK
# 手動增加一個 ceph.list 在 /etc/apt/sources.list.d 下
root@ubuntu12:/etc/apt/sources.list.d$ cat ceph.list
deb http://ceph.newdream.net/debian/ precise main
deb-src http://ceph.newdream.net/debian/ precise main
root@ubuntu12:~$ apt-get update
root@ubuntu12:~$ apt-get install -y ceph python-ceph
root@ubuntu12:~$ dpkg -l | grep ceph
ii  ceph                                            0.48.2argonaut-1precise                     distributed storage and file system
ii  ceph-common                                     0.48.2argonaut-1precise                     common utilities to mount and interact with a ceph storage cluster
ii  ceph-fs-common                                  0.48.2argonaut-1precise                     common utilities to mount and interact with a ceph file system
ii  ceph-fuse                                       0.48.2argonaut-1precise                     FUSE-based client for the Ceph distributed file system
ii  ceph-mds                                        0.48.2argonaut-1precise                     metadata server for the ceph distributed file system
ii  libcephfs1                                      0.48.2argonaut-1precise                     Ceph distributed file system client library
ii  python-ceph                                     0.48.2argonaut-1precise                     Python libraries for the Ceph distributed filesystem

# 安裝好之後, 就把你的 ceph cluster 的設定檔 copy 到 /etc/ceph 下, 正常就可以使用
# 至於怎麼安裝 ceph cluster 就請到 ceph 的官網去看囉~ 
root@ubuntu12:~$ ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at {wistor-003=172.17.123.92:6789/0,wistor-006=172.17.123.94:6789/0,wistor-007=172.17.123.95:6789/0}, election epoch 10, quorum 0,1,2 wistor-003,wistor-006,wistor-007
   osdmap e24: 23 osds: 23 up, 23 in
    pgmap v2242: 4416 pgs: 4416 active+clean; 8362 MB data, 156 GB used, 19850 GB / 21077 GB avail
   mdsmap e1: 0/0/1 up


如果只是要和一般的 iscsi server 整合, 可以參考官網, 大致上就是把一個 block device (ex:/dev/sdb) 丟到 LVM 去管理, 然後記得要建一個 cinder-volume
http://docs.openstack.org/trunk/openstack-compute/install/apt/content/osfolubuntu-cinder.html

Cinder Installation

root@ubuntu12:~$ apt-get install -y cinder-api cinder-scheduler cinder-volume iscsitarget open-iscsi iscsitarget-dkms python-cinderclient tgt
root@ubuntu12:~$ dpkg -l | grep cinder
ii  cinder-api                                      2012.2+git201209252100~precise-0ubuntu1            Cinder storage service - api server
ii  cinder-common                                   2012.2+git201209252100~precise-0ubuntu1            Cinder starage service - common files
ii  cinder-scheduler                                2012.2+git201209252100~precise-0ubuntu1            Cinder storage service - api server
ii  cinder-volume                                   2012.2+git201209252100~precise-0ubuntu1            Cinder storage service - api server
ii  python-cinder                                   2012.2+git201209252100~precise-0ubuntu1            Cinder python libraries
ii  python-cinderclient                             1:0.2.26+git201209201100~precise-0ubuntu1          python bindings to the OpenStack Volume API

修改 /etc/cinder/api_paste.ini
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = 127.0.0.1
service_port = 5000
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http
# 修改這三行
admin_tenant_name = service
admin_user = cinder
admin_password = password

修改 /etc/cinder/api_paste.ini, 改成使用 mysql 及 RBD driver, 並且把 rabbitmq 的 password 修改一下
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
rabbit_password = password
sql_connection = mysql://cinder:password@localhost:3306/cinder
volume_driver=cinder.volume.driver.RBDDriver


root@ubuntu12:~$ cinder-manage db sync
root@ubuntu12:~$ service cinder-api restart
root@ubuntu12:~$ service cinder-scheduler restart
root@ubuntu12:~$ service cinder-volume restart
root@ubuntu12:~$ cinder create --display_name test 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|      created_at     |      2012-10-02T07:14:34.815546      |
| display_description |                 None                 |
|     display_name    |                 test                 |
|          id         | e7e83b13-761e-40e3-8b4c-415126404e40 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
root@ubuntu12:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| e7e83b13-761e-40e3-8b4c-415126404e40 | available |     test     |  1   |     None    |             |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
root@ubuntu12:~$ rbd list
volume-e7e83b13-761e-40e3-8b4c-415126404e40


Reference

Folsom Announcement
http://lists.openstack.org/pipermail/openstack-announce/2012-September/000035.html
Folsom: How it was made
http://blog.bitergia.com/2012/09/27/how-the-new-release-of-openstack-was-built/

Cinder Installation Document
http://docs.openstack.org/trunk/openstack-compute/install/apt/content/osfolubuntu-cinder.html
https://github.com/EmilienM/openstack-folsom-guide/blob/master/doc/out/pdf/openstack-folsom-guide.pdf
Cinder Developer document
http://docs.openstack.org/developer/cinder/
Cinder Source Code
https://github.com/openstack/cinder.git
https://github.com/openstack/python-cinderclient

The Top 3 New Swift Features in OpenStack Folsom
http://swiftstack.com/blog/2012/09/27/top-three-swift-features-in-openstack-folsom/

2012年10月1日 星期一

Extra Line in Syntax Highlighter

最近發現 blogger 的 Syntax Highlighter 看起來都怪怪的, Line 12 會被拆成 2 行, 一定要用滑鼠縮放顯示成 90% 才會正常, 原以為是 Syntax Highlighter 的問題, 但到官網一看, 又發現不管怎麼縮放都沒問題. 後來又懷疑是 google blogger 的 template 內容出了問題, 但試了半天也不是. 最後在   Syntax Highlighter 的 github issue tracker 看到一模一樣的問題

https://github.com/alexgorbatchev/SyntaxHighlighter/issues/140
http://stackoverflow.com/questions/11804789/extra-lines-using-syntaxhighlighter-for-chrome-only

原來兇手是 Chrome, 如果有朋友遇到一樣的狀況, 就不要再花時間去 Debug 了!


2012年9月25日 星期二

CAP、BASE 理論筆記


這篇對 CAP 理論做了一個初步的解釋, 建議對 CAP 理論完全不清楚的人可以先看這篇
這篇對 CAP 的論文解釋的更深入, 同時也提到延申的 BASE 理論
Eric Brewer 解釋每一種設計在某一個象限都會有部份的漏洞 (Consistency/Availability/Partition Tolerance), 但這些漏洞在經過十二年後都發展出不同的配套措施, 我覺得很值得一看, 因為有些事情如果一直用工程師的角度去看, 會發現根本無解, 而且很嚴重, 但如果是加上行政上的配套措施之後, 似乎也就沒這麼嚴重了, 所以偶而也要跳脫工程師的思維才是。



http://pl.atyp.us/wordpress/index.php/2009/11/availability-and-partition-tolerance/
一個試著用簡單例子說明 CAP 的人, 但我總覺得還是有什麼地方怪怪的
下面的留言還滿有可看性的

留言一:
Quorum is to avoid “split brain syndrome” in which two mutually isolated sets of nodes continue running independently after a network partition. The key observation is that only one such set can contain N/2+1 or more nodes. Therefore, if you can reach at least that many nodes (including yourself) then you’re part of the quorum majority and it’s safe to keep running. If you can’t, then you’re part of the minority and it’s not safe so you shut down. Since the quorum members can see or assume that non-quorum members are no longer alive, they can break locks instead of waiting for operations that require them. It’s a simple and completely reasonable approach in a local environment where you’ve done everything you can to prevent partitions (e.g. redundant networks or even signaling through storage), but in a distributed environment where partitions are inevitable it can mean that all but one site becomes unusable.

留言二:

Thank you Jeff, for perfect and quick elaboration. Now it makes so much more sense to me.
I would like to clarify one more thing. Under the figure you have three bullets. The first bullet describes a scenario where we have an available but not partition-tolerant system. In the description you say that Z would be blocked because X is unreachable. Which I understand. However, I do not understand why this is called available system because when I compare it to the second bullet, and look at both situation as black-box from availability prospective, they appear to me completely similar. In both cases one node is working while the other is blocked. Yes, it is for two different reasons, which I understand, but what I don’t is why one is called available while the other is not. To me it appears that both situations are actually not-available (for two different reasons, though).

回覆:
CP without A – The distributed service will always return correct results, though some or all of the nodes may not respond at all in when there are problems. If the network gets partitioned this will continue to work.
AP without C – Any non-failed node will always respond to requests, though the data may be stale. If the network gets partitioned this will continue to work.
CA without P – The distributed service will return consistent results from all non-failed nodes, all the time. Since there is no “Partition-Tolerance” the distributed systems network must be perfect – never failing. If the network does get partitioned, all bets are off. You’ll lose consistency, availability, or both. Choosing to forgo “Partition-Tolerance” means that your system will not tolerate network partitions.


回覆:
It is impossible to have a “consistent” and “available” distributed system, unless you can guarantee there will be no network partitions. But It doesn’t mean there can’t be “almost CA systems” out there that are available most of the time and are consistent most of the time.


看完之後, 我認為紅色的留言才是正確的, 事實上, 再回頭看 infoq 那邊文章, 會發現 Eric Brewer 已解釋, 

  • CAP prohibits only a tiny part of the design space: perfect availability and consistency in the presence of partitions, which are rare
  • designers need to choose between consistency and availability when partitions are present
  • a period when the program must make a fundamental decision-the partition decision:
    • cancel the operation and thus decrease availability (CP)
    • proceed with the operation and thus risk inconsistency. (AP)

一個分散式的系統, 一般來說就必須滿足 P  (不會因為某些節點的損壞或網路中斷而造成服務完全不可得), 所以剩下的就是考慮你到底要
1. 犧牲某些人, 讓他們得不到服務, 但確保得到服務的人都能拿到一致的資料 (C)
2. 不犧牲任何人, 所有的人都可以得到服務, 但有可能拿到不一致的資料 (A)

至於宣稱同時滿足 CA 的系統, 一但某些節點壞掉(或因為網路的中斷而 unavailable)  就會讓整個系統停頓, 所有的人都得不到服務, 我個人認為 Single Point Failure 就是說明的好例子, 但在我們設計一個擁有 High Availability 的服務系統時, 我們最不想見到的就是整個服務停擺, 所以前面才會說現在的分散式系統大多都會以滿足 P 為前提, 接下來才是考慮要放棄 C 或是放棄 A,  甚至有些 NoSQL 的設計已經可以自由選擇要 CP or AP, 端看使用者的需求.