Sharing

2012年11月21日 星期三

EMC Survey Part II

上一篇大部份是記載有關管理軟體以及 EMC 開發的技術名詞講解, 這一篇則是記錄有關 EMC 硬體產品種類.

產品種類

Celerra

Celerra 是 NAS 的產品, 雖然目前已經停產, 但過去的技術文件都還很有參考價值, 現在新出的 VNX 未必有這麼齊全, 那就可以試著回頭看看 Celerra 的文件, 搞不好就會有收獲
http://en.wikipedia.org/wiki/Celerra

CLARiiON

CLARiiON 是 SAN 的產品, 目前已經停產, 但同樣的, 技術文件也都很有參考價值
http://en.wikipedia.org/wiki/CLARiiON

VNX

VNX 是 2011 年發表的新產品, 同時取代Celerra/CLARiiON, 我個人認為這也代表過去區分 NAS/SAN 產品的年代也正式結束, 未來的產品都同時支援這兩類的功能
http://www.emc.com/storage/vnx/vnx-series.htm

White Paper, 這一份把 VNX 每一個 component 都介紹的很詳細, 而且還附上圖, 值的一看
http://www.emc.com/collateral/hardware/white-papers/h8217-introduction-vnx-wp.pdf


從表中可以看的出來 VNX 的擴充能力很高, 如果是成長型的公司, 可以從最小的型號買起, 之後再 scale-out, 其中 VNX5100 因為缺少 X-Blades (Data Mover), 所以無法提供 File Protocol (NFS/CIFS/MPFS), 不管那一種 size, 搭配的軟體全是 Unisphere,

Slide:
http://www.slideshare.net/mrsmartbusiness/technical-perspective-emc-vnx-series
http://www.slideshare.net/John_Allegro/emc-vnxe-presentation
http://www.slideshare.net/ASBISSK/jg-vnx-series

Blog Article:
這篇介紹了 VNX 買來後要怎麼接線, 其實這也呈現了每個 Component 之間要怎麼溝通, 以及 HA 的架構
http://jpaul.me/?p=3505

VNX 設定 VDM: https://www.youtube.com/watch?v=Mn1YjLa-qPo
VNX 設定 NFS: https://www.youtube.com/watch?v=6vWqevhFs3o
VNX 設定 CIFS: https://www.youtube.com/watch?v=4v8euG5Y85I

VNXe

VNX 的推出是為了成長型的中大型公司, 而 VNXe 的推出則是為了還沒進入成長期的中小企業, 所有的 Component 都整合到一台之內, 所以不像 VNX 有 X-Blades
http://www.emc.com/storage/vnx/vnxe-series.htm

Hardware Spec:
http://www.emc.com/collateral/hardware/specification-sheet/h8515-vnxe-ss.pdf


VNXe CIFS:
在設定 share folder 時, 流程上雖然也是先建 CIFS Server, 但在 host 主機的用詞上就不太一樣, VNX 上是可以選擇使用那一台 Data Mover, 也就是 X-Blade, 但在 VNXe , 因為就只有主機, 所以不需要選擇 host 主機, 只需要設定好 ip/interface 即可
http://www.youtube.com/watch?v=nHI1DgA9ptw

這篇比較了 VNX & VNXe, 也說明了
http://searchstorage.techtarget.com.au/news/2240030675/FULL-DETAILS-EMCs-new-VNX-and-VNXe-range

Symmetrix

Symmetrix 是 storage array, 可以和 Celerra/CLARiiON/VNX 搭配, 提供更大量的儲存空間
http://en.wikipedia.org/wiki/Symmetrix

ATMOS

還沒有時間去看這個系列, 只知道有支援 Object Storage, 為了 Cloud Storage 所推出的產品, 有機會再補上
http://www.emc.com/storage/atmos/atmos.htm#!
http://www.emc.com/collateral/software/data-sheet/h5770-atmos-ds.pdf
http://www.emc.com/collateral/software/data-sheet/h7472-atmos-cloud-delivery-platform-ds.pdf
http://www.emc.com/collateral/software/specification-sheet/h5853-atmos-stor-hrdw-ss.pdf
http://www.emc.com/collateral/software/white-papers/h9505-emc-atmos-archit-wp.pdf


2012年11月15日 星期四

Write sparse image to block device


因為要把 image 倒回 block device, 最常用的方式就是直接 dd 回去, dd 真的是大家的好朋友, 但如果 image 的內容大部份都是 0, 有沒有加速的方式? 上網找了一下, 有兩個工具可以做到 sparse writing

ddpt

http://pkgs.org/download/ddpt
http://manpages.ubuntu.com/manpages/precise/man8/ddpt.8.html

使用方式和 dd 幾乎一樣

ddpt if=sparse_image of=/dev/sdx oflag=sparse

dd_rescue

http://pkgs.org/download/dd_rescue
http://manpages.ubuntu.com/manpages/precise/en/man1/dd_rescue.1.html

使用方式比較不一樣, 不過顯示的介面比較 User Friendly
dd_rescue -a sparse_image /dev/sdx

另外要注意在 CentOS 上有另外一個 package 叫 ddrescue, 少了一個底線, 功能有點類似, 但只支援檔案, 不支援 block device.千萬不要裝錯了.

補充一下, 看 sparse_file 實際佔的空間的方法
ls -lks sparse-file

2012年11月13日 星期二

EMC Survey Part I

最近在 Survey EMC 的 Feature List,發現有好多名詞看不太懂,他們真的是一家很強的廠商,自行研發各種技術,一些專有名詞也可以都可以在他們網站上找到文件


Introduction

這三篇系列文章介紹了 EMC 的幾個 Feature
http://storagezilla.typepad.com/storagezilla/2010/05/the-clariion-storage-pool.html
http://storagezilla.typepad.com/storagezilla/2010/05/storage-services-for-clariion-storage-pool-luns.html
http://storagezilla.typepad.com/storagezilla/2010/05/fast-cache-for-emc-unified-storage.html



http://gestaltit.com/featured/top/gestalt/emc-unified-platform-storage-tiering/


FLARE LUN and MetaLUN (Traditional)

簡中版的簡短說明



新增說明文字


Think LUN and Thick LUN (Virtual Provisioning)

英文版詳細說明
http://www.emc.com/collateral/hardware/white-papers/h5512-emc-clariion-virtual-provisioning-wp.pdf

裡面也有傳統的 LUN 和 Think/Thick LUN 適合的使用情況


有個表格清楚的列出來不同之處
https://community.emc.com/docs/DOC-19009

功能傳統FLARE LUN厚LUN精簡LUN
性能最好最好非性能優先業務
硬盤數量限制16池硬盤數量限制(整個存儲硬盤數減去5塊跳馬盤)池硬盤數量限制(120或整個存儲硬盤數減去5塊跳馬盤)
RAID類型全部5,6,1/05,6,1/0
讀取高速緩存(讀緩存)
寫高速緩存(寫緩存)
展開(LUN擴展)通過的MetaLUN的方式
收縮(LUN壓縮)
自旋向下(磁盤省電模式)
數據壓縮(數據壓縮)
自動級(自動存儲分層)
保留LUN池(保留LUN池)
寫意圖日誌(的MirrorView寫日誌)
克隆專用LUN(克隆私有LUN)
捕捉/克隆(快照/克隆鏡像)
MirrorView(CLARiiON的遠程複製軟件)
SAN複製(CLARiiON的遠程複製軟件)

FAST (Full Automated Storage Tiering)

EMC 文件
http://www.emc.com/collateral/hardware/white-papers/h8234-fast-clariion-wp.pdf

RAID Group, Storage Pool and Storage Group


簡中版
https://community.emc.com/message/615219

這篇有寫到整個 Storage Pool 的設計理念
http://virtualeverything.wordpress.com/2011/03/05/emc-storage-pool-deep-dive-design-considerations-caveats/

不過也有人認為 Storage Pool 有一些缺點

http://blog.virtualtacit.com/post/3077935618/emc-storage-pools-the-good-the-bad-the-use-case

Software -- UniSphere

Overview

http://www.emc.com/storage/vnx/unisphere.htm

DataSheet

http://www.emc.com/collateral/software/data-sheet/h7303-unisphere-ds.pdf

White Paper

http://www.emc.com/collateral/hardware/white-papers/h8179-unisphere-vnxe-wp.pdf
http://www.emc.com/collateral/software/white-papers/h8017-unisphere-element-manager.pdf
http://www.emc.com/collateral/software/white-papers/h8157-unisphere-storage-pool-oracle-wp.pdf
http://www.emc.com/collateral/hardware/white-papers/h1215-celerra-mgr-web-ldv.pdf

Slide

http://www.slideshare.net/xigua_521/emc-vnx-unisphere

Other

http://www.emc.com/collateral/analyst-reports/11327-ar-vnx-vs-netapp-fas-usability-study.pdf
http://www.ndm.net/emcvnx/pdf/vnx/300-012-899-vnx-security-configuration-bLock.pdf
http://www.thulinaround.com/2011/08/01/configuring-ldap-authentication-for-unisphere-on-the-vnx/
http://www.boche.net/blog/index.php/2011/02/14/vsphere-integration-with-emc-unisphere/
http://www.cisco.com/en/US/docs/unified_computing/ucs/UCS_CVDs/ucsvspex_hyperv.html

Software -- Symmetrix Management Console

Introduction

http://www.emc.com/collateral/hardware/data-sheet/h6194-storage-management-solution.pdf

Data Sheet

http://www.emc.com/collateral/software/data-sheet/h2815-smc-ease-of-use-ds-ldv.pdf
http://www.emc.com/collateral/software/data-sheet/c1144-symmetrix-management-console.pdf

Install Guide

http://zh.scribd.com/doc/84913911/SMC-V7-3-1

Blogger Article

http://gestaltit.com/all/tech/storage/devang/emc-symmetrix-management-console-smc-symmetrix-vmax-systems/

Software -- Solution Enabler

Document

http://zh.scribd.com/doc/75901574/Solutions-Enabler-7-3
http://zh.scribd.com/doc/100022721/EMC-Solutions-Enabler-Symmetrix-Migration-CLI-V7-4

Software -- ProSphere

White Paper

http://www.emc.com/collateral/software/white-papers/h8886-prosphere-overview.pdf

Blog Article

https://community.emc.com/docs/DOC-15851
http://virtualgeek.typepad.com/virtual_geek/2012/03/emc-prosphere-15-play-learn-try.html
http://managedview.emc.com/2012/01/managing-the-exploding-growth-in-storage/#more-202
http://managedview.emc.com/2012/03/watch-this-space-2/



其它進階應用文件

Storage System Fundamentals for Performance and Availability

這份還滿值得看一下, 因為把所有的 component 都講了一下, 不會太深也不會太淺
http://www.emc.com/collateral/hardware/white-papers/h1049-emc-clariion-fibre-chnl-wp-ldv.pdf




Virtual Provisioning Best Practice

http://www.emc.com/collateral/hardware/white-papers/h10688-z-os-virt-prov-best-practices-wp.pdf

Best Practices for Performance and Availability

http://www.emc.com/collateral/hardware/white-papers/h5773-clariion-best-practices-performance-availability-wp.pdf

EMC Enginuity 5876 for Mainframe Environments

有寫到一些 Migration 的東西
http://taiwan.emc.com/collateral/hardware/white-papers/h10686-enginuity5876-mainframe-env-wp.pdf


Disk-Drive Spin Down Technology

http://www.emc.com/collateral/hardware/white-papers/h6632-clariion-disk-drive-spin-down-wp.pdf

Unified Platform Storage Device Techonology

http://www.emc.com/collateral/hardware/white-papers/h4208-an-intro-emc-clariion-hard0drive-tech-wp.pdf

Global Hot Spares and Proactive Hot Sparing

http://www.emc.com/collateral/software/white-papers/c1069-clariion-global-hot-spares-ldv.pdf

Asymmetric Active/Active Feature

http://www.emc.com/collateral/hardware/white-papers/h2890-emc-clariion-asymm-active-wp.pdf

Best Practices for EMC VPLEX Technical Notes

http://www.vclouds.nl/wp-content/uploads/2012/04/h7139-implementation-planning-vplex-tn.pdf

EMC CLARiiON CX4 Series Featuring UltraFlex Technology

EMC TechTalk

VMware vSphere

http://www.emc.com/collateral/hardware/white-papers/h6337-introduction-using-celerra-vmware-vsphere-wp.pdf

HyperV

http://www.emc.com/collateral/hardware/white-papers/h6182-using-clariion-microsoft-hyper-v-wp.pdf

2012年10月30日 星期二

健保相關規定及省錢方法


最近因為小草莓要出生了, 再加上老爸要退休了, 所以要重新安排阿媽的健保要掛在何處, 就找了一點資料,
文章從這篇開始, 裡面有舉了一些例子來說明如何省錢
http://www.moneynet.com.tw/e_news.php?id=1508

超過65歲的國民有補助

這個連結有列出來各個單位的補助方案, 基本上都有排富條款

http://www.nhi.gov.tw/webdata/webdata.aspx?menu=18&menu_id=682&webdata_id=2393&WD_ID=745

以台北市為例
(一)年滿65歲老人或55歲原住民,且設籍並實際居住臺北市滿1年者。
(二)經稅捐稽徵機關核定之最近1年綜合所得總額合計未達申報標準或綜合所得稅率未達20%者。老人由納稅義務人申報為受扶養人而有上開情事者,亦同。
(三)未獲政府機關健保自付額之全額補助者。]

更詳細一點可以看這裡
http://www.bosa.tcg.gov.tw/i/i0300.asp?fix_code=0425002&l1_code=04&l2_code=25

每人每月至多補助749元(中央健康保險局第六類保險對象自付額),低於749元者核實補助。

基本上這個方案會自動補助, 不用另外提出申請, 除非曾經因為不符規定而停止補助的人, 當又符合規定之後, 必須要重新申請.

<健保局和社會局的說法不同, 一個說會自動重新生效, 一個說一定要提出申請, 我最後是採信社會局的說, 因為真的在處理補助的單位是他們, 我就還是送了文件去申請>

眷屬的保險費, 最多算到三人, 超過的就免費

http://dohlaw.doh.gov.tw/Chi/FLAW/FLAWDAT0201.asp?lsid=FL014028

第十八條
第一類至第三類被保險人及其眷屬之保險費,依被保險人之投保金額及保
險費率計算之;保險費率,以百分之六為上限。
前項眷屬之保險費,由被保險人繳納;超過三口者,以三口計。

如果總共有三個眷屬以上(不包含自己), 那就算三個, 下面是保險負擔金額表, 最多就是列到三人
http://www.nhi.gov.tw/webdata/webdata.aspx?menu=1&menu_id=5&webdata_id=2389&WD_ID=


眷屬依附在薪資較低者

這一點大家應該都知道, 但另外要注意的事情是,
1. 直系血親是不包含姻親, 也就是自己的父母親是不能依附在老婆或先生的健保內,
2. 配偶如果有工作的話也不能依附在自己底下
3. 奶奶不能依附在孫子底下, 除非奶奶所有的子女都已經退休且都依附成別人的眷屬
    <老實說, 我不太清楚這是那一個法條規定的, 這是打去健保局問到的>

第二條
二、眷屬:
(一)被保險人之配偶,且無職業者。
(二)被保險人之直系血親尊親屬,且無職業者。
(三)被保險人二親等內直系血親卑親屬未滿二十歲且無職業,或年滿二
十歲無謀生能力或仍在學就讀且無職業者。

父母用第六類人口投保

如果你的健保費多於 $749 元, 那建議父母或奶奶都改向區公所投保, 而不要依附在你底下
http://www.nhi.gov.tw/webdata/webdata.aspx?menu=1&menu_id=5&webdata_id=3328&WD_ID=
http://www.nhi.gov.tw/webdata/webdata.aspx?menu=18&menu_id=678&webdata_id=3436&WD_ID=722


相關連結:
保險費 DIY 計算
http://www.nhi.gov.tw/webdata/webdata.aspx?menu=18&menu_id=679&webdata_id=3444&WD_ID=679

2012年10月28日 星期日

Openstack Folsom - Boot from Volume (Rados Block Device)


這個是 Folsom 的隱藏新功能, 之前要做這件事都要做偷天換日的行為把 volume 的內容換掉, 但現在不必了, 這個功能已經整合進去 Folsom. 設定及操作都是參考以下這篇

http://ceph.com/docs/master/rbd/rbd-openstack/

Cinder Configuration

主要的安裝及設定請參考 openstack-folsom-installation-of-cinder

/etc/cinder/cinder.conf

這邊因為 cinder 要和 glance 拿到 image template, 所以要在設定檔內加入 glance host 的 ip
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
sql_connection = mysql://cinder:password@localhost:3306/cinder
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
volume_driver=cinder.volume.driver.RBDDriver
rabbit_host=rabbitmq
rabbit_password = password
my_ip = 172.17.123.12
glance_host = 172.17.123.16

Glance Configuration

主要的安裝及設定請參考 openstack-folsom-installation-of-glance

/etc/glance/glance-api.conf

需要 glance 把 rbd url 傳送出去, 所以把這個隱藏選項打開
show_image_direct_url = True

Upload Image

如果要直接從 Block Device 開機, 那原來的 image 的格式目前必須要是 raw image, 而不能用 qcow2. 所以我們先把 qcow2 轉成 raw image, 然後上傳到 glance
root@glance:~$ wget http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
root@glance:~$ kvm-img convert -f qcow2 -O raw precise-server-cloudimg-amd64-disk1.img precise-server-cloudimg-amd64-disk1.raw
root@glance:~$ glance image-create --name Ubuntu-Precise-Raw --is-public true --container-format bare --disk-format raw < ./precise-server-cloudimg-amd64-disk1.raw
root@glance:~$ glance image-list
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| ID                                   | Name                | Disk Format | Container Format | Size       | Status |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| cad779fc-c851-4581-ac4d-474c3773bf89 | Ubuntu-Precise-Raw  | raw         | bare             | 2147483648 | active |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+

Create Volume from Image Template

接下來從 cinder 生成一個新的 volume, 但多加一個參數指定從 Image Template 產生
root@cinder:~# cinder create --image-id cad779fc-c851-4581-ac4d-474c3773bf89 10
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|      created_at     |      2012-10-29T03:12:59.504616      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|          id         | 4e8527a9-eb01-44f1-8fed-fc831c4134f4 |
|       image_id      | cad779fc-c851-4581-ac4d-474c3773bf89 |
|       metadata      |                  {}                  |
|         size        |                  10                  |
|     snapshot_id     |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

root@cinder:~# cinder list
+--------------------------------------+-----------+----------------+------+-------------+--------------------------------------+
|                  ID                  |   Status  |  Display Name  | Size | Volume Type |             Attached to              |
+--------------------------------------+-----------+----------------+------+-------------+--------------------------------------+
| 4e8527a9-eb01-44f1-8fed-fc831c4134f4 | available |      None      |  10  |     None    |                                      |
+--------------------------------------+-----------+----------------+------+-------------+--------------------------------------+

root@cinder:~# rbd info volume-4e8527a9-eb01-44f1-8fed-fc831c4134f4
rbd image 'volume-4e8527a9-eb01-44f1-8fed-fc831c4134f4':
        size 10240 MB in 1280 objects
        order 23 (8192 KB objects)
        block_name_prefix: rbd_data.2f8e5262f5ff
        format: 2
        features: layering
        parent: images/cad779fc-c851-4581-ac4d-474c3773bf89@snap
        overlap: 2048 MB

觀察 /var/log/cinder/cinder-volume.log
# 拿到 image location
2012-10-29 11:12:59 DEBUG cinder.volume.manager [req-eed17d93-f2da-479b-b04c-4418ca4948b3 fafd0583de8a4a1b93b924a6b2cb7e
b5 eefa301a6a424e7da3d582649ad0e59e] image_location: rbd://77e083f7-de88-4f9e-b654-8ce6949a3039/images/cad779fc-c851-458
1-ac4d-474c3773bf89/snap create_volume /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:151
2012-10-29 11:12:59 DEBUG cinder.utils [req-eed17d93-f2da-479b-b04c-4418ca4948b3 fafd0583de8a4a1b93b924a6b2cb7eb5 eefa30
1a6a424e7da3d582649ad0e59e] Running cmd (subprocess): ceph fsid execute /usr/lib/python2.7/dist-packages/cinder/utils.py
:163

# 檢查一下 image 的 snapshot 
2012-10-29 11:12:59 DEBUG cinder.utils [req-eed17d93-f2da-479b-b04c-4418ca4948b3 fafd0583de8a4a1b93b924a6b2cb7eb5 eefa30
1a6a424e7da3d582649ad0e59e] Running cmd (subprocess): rbd info --pool images --image cad779fc-c851-4581-ac4d-474c3773bf8
9 --snap snap execute /usr/lib/python2.7/dist-packages/cinder/utils.py:163

# 使用 ceph clone 功能 (COW)
2012-10-29 11:13:00 DEBUG cinder.utils [req-eed17d93-f2da-479b-b04c-4418ca4948b3 fafd0583de8a4a1b93b924a6b2cb7eb5 eefa30
1a6a424e7da3d582649ad0e59e] Running cmd (subprocess): rbd clone --pool images --image cad779fc-c851-4581-ac4d-474c3773bf
89 --snap snap --dest-pool rbd --dest volume-4e8527a9-eb01-44f1-8fed-fc831c4134f4 execute /usr/lib/python2.7/dist-packag
es/cinder/utils.py:163

# 最後 resize 大小
2012-10-29 11:13:00 DEBUG cinder.utils [req-eed17d93-f2da-479b-b04c-4418ca4948b3 
fafd0583de8a4a1b93b924a6b2cb7eb5 eefa301a6a424e7da3d582649ad0e59e] Running cmd (subprocess): rbd resize --pool rbd --image volume-4e8527a9-eb01-44f1-8fed-fc831c4134f4 --size 10240 execute /usr/lib/python2.7/dist-packages/cinder/utils.py:163

Create VM

選擇剛剛上傳的 Raw Image

選擇 "Boot from volume", 然後選擇剛剛利用 Cinder 指令從 Image Template 做出來的  Volume


觀察 computer node 上面的 VM, 發現他的 block device 直接從 rbd protocol 連到 Ceph.
root@nova:~$ virsh list
 Id Name                 State
----------------------------------
  1 instance-00000023    running
  4 instance-0000002b    running

root@nova:~$ virsh domblklist 4
Target     Source
------------------------------------------------
vda        rbd/volume-4e8527a9-eb01-44f1-8fed-fc831c4134f4

補充

如果遇到 Glance 出問題, 有可能是介接的部份出問題, 但問題不大, 主要是要把 unicode 轉 str

/usr/lib/python2.7/dist-packages/glance/store/rbd.py

with rados.Rados(conffile=self.conf_file, rados_id=self.user) as conn:
            with conn.open_ioctx(self.pool) as ioctx:
                if loc.snapshot:
                    # 修改這行
                    with rbd.Image(ioctx, str(loc.image)) as image:
                        try:
                            # 修改這行
                            image.unprotect_snap(str(loc.snapshot))
                        except rbd.ImageBusy:
                            log_msg = _("snapshot %s@%s could not be "
                                        "unprotected because it is in use")
                            LOG.error(log_msg % (loc.image, loc.snapshot))
                            raise exception.InUseByStore()
                        # 修改這行   
                        image.remove_snap(str(loc.snapshot))
                try:
                    # 修改這行
                    rbd.RBD().remove(ioctx, str(loc.image))
                except rbd.ImageNotFound:
                    raise exception.NotFound(
                        _('RBD image %s does not exist') % loc.image)
                except rbd.ImageBusy:
                    log_msg = _("image %s could not be removed"
                                "because it is in use")
                    LOG.error(log_msg % loc.image)
                    raise exception.InUseByStore()



2012年10月24日 星期三

vsftp server with virtual account


第一次架 ftp server 就看這兩篇, 很快就可以架好
https://help.ubuntu.com/12.04/serverguide/ftp-server.html
http://manpages.ubuntu.com/manpages/precise/en/man5/vsftpd.conf.5.html

如果想要用 virtual user 來搭配 ftp server, 而不是用 server 上的帳號, 那就看這篇

http://sigerr.org/linux/setup-vsftpd-custom-multiple-directories-users-accounts-ubuntu-step-by-step

Package Installation

總共要裝三個東西, 一個是 vsftpd, 一個是 PAM Pluggable Authentication Modules), 用來建立虛擬帳號以及認証, 最後是 apache 裡的一個小工具, 在建立帳號時會使用到
root@ubuntu:~$ apt-get install vsftpd libpam-pwdfile apache2-utils

Configuration

/etc/pam.d/vsftpd-virtual

建立一個認証的機制, 我們將密碼存在 /nfsroot/ftp/ftpd.passwd 之中
# Customized login using htpasswd file
auth required pam_pwdfile.so pwdfile /nfsroot/ftp/ftpd.passwd
account required pam_permit.so

/etc/vsftpd.conf

前面幾項都是原來的設定檔有的, 我就沒有拿掉, 因為我只是要開放一個 ftp 站供人下載東西, 而不支援上傳, 所以我沒有把 write_enable 打開.
listen=YES
# 改成 NO
anonymous_enable=NO
# 改成 YES
local_enable=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES
# 改成 YES
chroot_local_user=YES
secure_chroot_dir=/var/run/vsftpd/empty
# 使用另一個 PAM 設定檔
pam_service_name=vsftpd-virtual
rsa_cert_file=/etc/ssl/private/vsftpd.pem
# 讓所有的虛擬帳號都改用 guest 登入
guest_enable=YES
# 讓所有的虛擬帳號都有各自的目錄, 互相不影響
user_sub_token=$USER
local_root=/nfsroot/ftp/$USER
# 讓檔案擁有者顯示成 ftp
hide_ids=YES
# 使用目錄設定的權限
virtual_use_local_privs=YES

Register User

以下是創造虛擬帳號的方法, 因為在 vsftp.conf 之中我們把 local_root 設定在 /nfsroot/ftp/$USER, 所以我們要主動幫他把目錄建出來, 擁有者改成 ftp, 因為預設的 guest 帳號是 ftp, 權限要改成不能寫入, 不然會無法登入
root@ubuntu:~$ touch /nfsroot/ftp/ftpd.passwd
root@ubuntu:~$ htpasswd -bd /nfsroot/ftp/ftpd.passwd <username> <password>
root@ubuntu:~$ mkdir /nfsroot/ftp/<username>
root@ubuntu:~$ chown ftp:nogroup /nfsroot/ftp/<username>
root@ubuntu:~$ chmod -w /nfsroot/ftp/<username>

以上都做完之後, 就記得要重啟 vsftpd, 就完成囉~

其它雷同的介紹, 但有不同的設定, 可以看看

http://www.onaxer.com/2010/12/01/virtual-users-and-directories-in-vsftpd/
http://howto.gumph.org/content/setup-virtual-users-and-directories-in-vsftpd/




2012/11/13 補充:

Writable Root Folder

如果虛擬帳號的 root folder 不是唯讀的, 就無法登入, 你會看到以下的訊息

500 OOPS: vsftpd: refusing to run with writable root inside chroot()

後來在網路上找了一些文章, 才發現原來這是 vsftpd 2.3.5 才出現的設計,
原來是沒有這個限制的, 也很多人抱怨這個功能很難用, 必須要多建立一個可寫入的子目錄,
才能讓使用者上傳檔案到 ftp server, 所以就有一些解法
http://www.benscobie.com/fixing-500-oops-vsftpd-refusing-to-run-with-writable-root-inside-chroot/#comment-2051

解法1:升級到 vsftpd 3.0.0, 設定檔內加上 allow_writeable_chroot=YES

解法2:改成使用 vsftpd 2.3.2

解法3:修改 2.3.5 的 code, 把這個限制解除掉, 網路上有人已經做了這件事

root@ubuntu:~$ apt-get install python-software-properties
root@ubuntu:~$ add-apt-repository ppa:thefrontiergroup/vsftpd
root@ubuntu:~$ apt-get update
root@ubuntu:~$ apt-get install vsftpd

以下我採取解法3, 在這種設定下, 如果你想要新增一個 read-only 的帳號, 簡單的作法就是照樣把 root folder 的 write permission 去掉, 如果想要讓使用者可以上傳檔案, root folder 就要記得加上 write permission

/etc/vsftpd.conf

listen=YES
# 改成 NO
anonymous_enable=NO
# 改成 YES
local_enable=YES
# 改成 YES
write_enable=YES
dirmessage_enable=YES
use_localtime=YES
xferlog_enable=YES
connect_from_port_20=YES
# 改成 YES
chroot_local_user=YES
secure_chroot_dir=/var/run/vsftpd/empty
# 使用另一個 PAM 設定檔
pam_service_name=vsftpd-virtual
rsa_cert_file=/etc/ssl/private/vsftpd.pem
# 讓所有的虛擬帳號都改用 guest 登入
guest_enable=YES
# 讓所有的虛擬帳號都有各自的目錄, 互相不影響
user_sub_token=$USER
local_root=/nfsroot/ftp/$USER
# 讓檔案擁有者顯示成 ftp
hide_ids=YES
# 使用目錄設定的權限
virtual_use_local_privs=YES
allow_writeable_chroot=YES



2012年10月21日 星期日

Git filter-branch 的使用


最近因為幫忙重新整理 git repository, 所以有機會接觸 git filter-branch, 過去因為 policy 沒有定義好, 所以大家都將 binary 檔也加進 repository, 造成整個 repository 很肥大, 要 checkout 時也必須要等很久, 所以我的任務就是把這些 binary 檔拔掉, 改成用 md5 or sha1 檔案代替, 如果真的需要用到時, 再去另外一個 http server 下載.
http://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html

假設整個 repository 有 1000 個版本, 那filter-branch 執行的過程, 其實就是依序把這些版本 checkout, 然後執行你指定的指令來改變檔案的內容, 或是甚至新增/刪除檔案, 之後再把修改過的檔案 commit 進去新的 repository. 所以執行完之後, 整個 repository 中每一筆項目的 SHA ID 都會改變, 是個很暴力的作法, 但也因為很暴力, 所以基本上你想要做什麼事應該都可以達成.

Filter

filter-branch 提供了許多不同 filter, 可以幫助你在正確的時間點執行指令

--env-filter

幫助你修改 author name 或是 author e-mail, 可以參考以下網址提供的 script
https://help.github.com/articles/changing-author-info

--tree-filter

幫助你在每一個版本去修改檔案的內容, 新增/刪除檔案, 最常使用的應該也是這個, 以我自己的例子來說, 我會寫一個像以下的 script 把大檔案轉成 sha1 檔
#!/bin/bash
function transform {

    file=$1
    sha1_name=$file.sha1

    if [ -f /tmp/git/$file ]; then
        rm -f $file
        cp /tmp/git/$sha1_name $sha1_name
    else
        sha1sum $file > $sha1_name
        mv $file /tmp/git
        cp $sha1_name /tmp/git
    fi

}

pushd ./bigfile
for file in *.tgz *.zip; do
    if [ ! -f $file ]; then
        continue;
    fi

    transform $file
popd

--index-filter

這個其實是 --tree-filter 的快速版本, 如果你沒有要改變檔案的內容, 只是單純改變 repository 的 history, 那可以使用這個, 因為他不會真的 checkout 檔案, 速度上快很多, 下面兩個網址都有示範, 最常用的功能就是把某一個特定的檔案永久的從 repository 中刪除.
https://help.github.com/articles/remove-sensitive-data
http://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository

--msg-filter

這個是用來改變 commit message 的內容, 原來的內容會介由 standard input 輸入, 而你輸出到 standard output 的內容就會作為新的 commit message.
# 舉例來說, 因為 cat 就不會改變內容, 所以 commit message 完全不變
$ git filter-branch --msg-filter cat
# tac 會把內容全部反過來
$ git filter-branch --msg-filter tac

--tag-name-filter

用來改變 tag name, 如果該版本有加上 tag, 那當 checkout 這個版本時, 就會乎叫你指定的指令, 原來的內容會介由 standard input 輸入, 而你輸出到 standard output 的內容就會作為新的 tag name

--subdirectory-filter

把某一個 folder 下的 commit 獨立出來變成一個新的 repository, 如果當你的專案越來越大時, 你可能會想把某個資料夾獨立出來變成一個新專案, 那這個功能就很好用
http://gugod.org/2012/07/split-a-git-repository/

Other Command

--prune-empty

如果有些 filter 產生了空的 commit, 那會主動清除掉

-f --force

強制把暫存區的東西清掉

2012年10月11日 星期四

Openstack Folsom - Installation of Horizon

Before Installation


Add repository

root@horizon:~$ apt-get install -y python-software-properties
root@horizon:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-trunk-testing
root@horizon:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-deps-staging
root@horizon:~$ apt-get update && apt-get -y dist-upgrade


Hostname Setting

最簡單的方式是在 /etc/hosts 內設定會用到的 hostname

172.17.123.12   rabbitmq
172.17.123.12   mysql
172.17.123.12   cinder
172.17.123.13   keystone
172.17.123.14   swift-proxy
172.17.123.16   glance
172.17.123.17   nova
172.17.123.18   horizon

Environment Variable Setting

編輯一個設定檔: novarc, 設定一些等下會用到的區域變數, 並且匯到 .bashrc 之中, 下次再進來就不需要重新設定
root@glance:~$ cat novarc
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL="http://keystone:5000/v2.0/"
export SERVICE_ENDPOINT="http://keystone:35357/v2.0"
export SERVICE_TOKEN=password
root@horizon:~$ source novarc
root@horizon:~$ echo "source novarc">>.bashrc


Horizon Installation

Install Package


root@horizon:~$ apt-get install -y memcached libapache2-mod-wsgi openstack-dashboard

Configuration


設定好 keystone 的 ip address
/etc/openstack-dashboard/local_settings.py
...
OPENSTACK_HOST = "keystone"
OPENSTACK_KEYSTONE_URL = "http://%s:5000/v2.0" % OPENSTACK_HOST
OPENSTACK_KEYSTONE_DEFAULT_ROLE = "Member"
...

dashboard 在這版多了一個 ubuntu_theme.py 的設定, 可以套用自己設計的 theme, 不過預設的 theme 看起來怪怪的, 所以看不習慣的人可以把 /etc/openstack-dashboard/ubuntu_theme.py 砍掉, 會讓整個 UI 順眼不少.


另外要提醒的是, 官網上有提到可以設定 SWIFT_ENABLED 以及 QUANTUM_ENABLED, 但這兩個設定其實已經不能用了, dashboard 會直接詢問 keystone 有提供那些服務, 所以如果是用 script 安裝 keystone, 那可能會遇到 dashboard 上面出現 "Network" 設定, 但其實沒有安裝 Quantum, 所以記得把 keystone 內的 quantum endpoint service 砍掉, 然後 restart apache2.
https://answers.launchpad.net/horizon/+question/210437

Login

設定完成後就登入 http://horizon's ip/horizon, 帳號密碼是看當初在 keystone 的設定, 我這邊是用 admin/password



Create VM

選擇用之前上傳的 Ubuntu-Precise

塞入 mykey

啟動之後還可以使用 vnc 界面操作, 如果無法順利看到 vnc 的人, 可能是因為 nova-novncproxy 沒有順利啟動, 可以看一下是否遇到和我同樣的問題, 我補充在上一篇的最後面
Openstack Folsom - Installation of Nova



2012年10月9日 星期二

Openstack Folsom - Installation of Nova

Before Installation

Add repository

root@nova:~$ apt-get install -y python-software-properties
root@nova:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-trunk-testing
root@nova:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-deps-staging
root@nova:~$ apt-get update && apt-get -y dist-upgrade

Hostname Setting

最簡單的方式是在 /etc/hosts 內設定會用到的 hostname
172.17.123.12   rabbitmq
172.17.123.12   mysql
172.17.123.12   cinder
172.17.123.13   keystone
172.17.123.14   swift-proxy
172.17.123.16   glance
172.17.123.17   nova

MySQL Setting

在 MySQL 內加入一個新的 database, 如果 MySQL 和 Glance 在同一台 Server 上, 記得也要設定從 localhost 登入的權限及密碼
mysql> CREATE DATABASE nova;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL ON nova.* TO 'nova'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

在 Nova 這邊記得也要裝 mysql-client, 然後可以測試看看能不能連線成功
root@nova:~$ apt-get install mysql-client python-mysqldb
root@nova:~$ mysql -h 172.17.123.12 -u nova -ppassword
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 236
Server version: 5.5.24-0ubuntu0.12.04.1 (Ubuntu)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Environment Variable Setting

編輯一個設定檔: novarc, 設定一些等下會用到的區域變數, 並且匯到 .bashrc 之中, 下次再進來就不需要重新設定
root@glance:~$ cat novarc
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL="http://keystone:5000/v2.0/"
export SERVICE_ENDPOINT="http://keystone:35357/v2.0"
export SERVICE_TOKEN=password
root@glance:~$ source novarc
root@glance:~$ echo "source novarc">>.bashrc

Keystone Setting

把 endpoint 的網址設定好
root@nova:~$ apt-get install python-keystoneclient
root@nova:~$ keystone endpoint-list
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+--------------------------------------+
|                id                |   region  |                   publicurl                   |                  internalurl                  |               adminurl               |
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+--------------------------------------+
| 580b71d126804c5197b91c79fd74a330 | RegionOne |           http://keystone:5000/v2.0           |           http://keystone:5000/v2.0           |      http://keystone:35357/v2.0      |
| 5ef55e38e5c54477bd659d4185d0a776 | RegionOne |             http://glance:9292/v2             |             http://glance:9292/v2             |        http://glance:9292/v2         |
| 6c788747593d475f831b6ff128bde995 | RegionOne |      http://cinder:8776/v1/$(tenant_id)s      |      http://cinder:8776/v1/$(tenant_id)s      | http://cinder:8776/v1/$(tenant_id)s  |
| 95e16e71a8f04ac68ae401df5284ce3e | RegionOne | http://swift-proxy:8080/v1/AUTH_$(tenant_id)s | http://swift-proxy:8080/v1/AUTH_$(tenant_id)s |      http://swift-proxy:8080/v1      |
| c9659fab79454ee38bd926a2b78fa351 | RegionOne |       http://nova:8774/v2/$(tenant_id)s       |       http://nova:8774/v2/$(tenant_id)s       |  http://nova:8774/v2/$(tenant_id)s   |
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+--------------------------------------+

CEPH Installation

Optional, 如果 Cinder 的設定是 LVM 就不必安裝這個, 想要看深入一點介紹的可以看這個連結
http://www.sebastien-han.fr/blog/2012/06/10/introducing-ceph-to-openstack/
root@nova:~$ wget -q -O - https://raw.github.com/ceph/ceph/master/keys/release.asc | apt-key add -
OK
# 手動增加一個 ceph.list 在 /etc/apt/sources.list.d 下
root@nova:/etc/apt/sources.list.d$ cat ceph.list
deb http://ceph.newdream.net/debian/ precise main
deb-src http://ceph.newdream.net/debian/ precise main
root@nova:~$ apt-get update
root@nova:~$ apt-get install -y ceph python-ceph
root@nova:~$ dpkg -l | grep ceph
ii  ceph                                            0.48.2argonaut-1precise                     distributed storage and file system
ii  ceph-common                                     0.48.2argonaut-1precise                     common utilities to mount and interact with a ceph storage cluster
ii  ceph-fs-common                                  0.48.2argonaut-1precise                     common utilities to mount and interact with a ceph file system
ii  ceph-fuse                                       0.48.2argonaut-1precise                     FUSE-based client for the Ceph distributed file system
ii  ceph-mds                                        0.48.2argonaut-1precise                     metadata server for the ceph distributed file system
ii  libcephfs1                                      0.48.2argonaut-1precise                     Ceph distributed file system client library
ii  python-ceph                                     0.48.2argonaut-1precise                     Python libraries for the Ceph distributed filesystem

# 安裝好之後, 就把你的 ceph cluster 的設定檔 copy 到 /etc/ceph 下, 正常就可以使用
# 至於怎麼安裝 ceph cluster 就請到 ceph 的官網去看囉~ 
root@nova:~$ ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at {wistor-003=172.17.123.92:6789/0,wistor-006=172.17.123.94:6789/0,wistor-007=172.17.123.95:6789/0}, election epoch 10, quorum 0,1,2 wistor-003,wistor-006,wistor-007
   osdmap e24: 23 osds: 23 up, 23 in
    pgmap v2242: 4416 pgs: 4416 active+clean; 8362 MB data, 156 GB used, 19850 GB / 21077 GB avail
   mdsmap e1: 0/0/1 up

Nova Installation

Nova Package

因為是要和 Cinder 整合, 所以沒有安裝 nova-volume
root@nova:~$ apt-get install nova-compute nova-api nova-ajax-console-proxy nova-cert nova-consoleauth nova-doc nova-scheduler nova-network nova-novncproxy novnc python-novnc
root@nova:~$ ii  nova-ajax-console-proxy          2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - AJAX console proxy - transitional package
ii  nova-api                         2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - API frontend
ii  nova-cert                        2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - certificate management
ii  nova-common                      2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - common files
ii  nova-compute                     2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - compute node
ii  nova-compute-kvm                 2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - compute node (KVM)
ii  nova-consoleauth                 2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - Console Authenticator
ii  nova-doc                         2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - documentation
ii  nova-network                     2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - Network manager
ii  nova-novncproxy                  2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - NoVNC proxy
ii  nova-scheduler                   2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute - virtual machine scheduler
ii  novnc                            2012.2~20120906+dfsg-0ubuntu2~precise              HTML5 VNC client
ii  python-nova                      2012.2+git201210091907~precise-0ubuntu1            OpenStack Compute Python libraries
ii  python-novaclient                1:2.9.0.10+git201210101300~precise-0ubuntu1        client library for OpenStack Compute API
ii  python-novnc                     2012.2~20120906+dfsg-0ubuntu2~precise              HTML5 VNC client - libraries

/etc/nova/nova.conf

內容是設定成和 Keystone, Cinder(Ceph), Glance(Ceph), 整合在一起, 中間其實用不到 iscsi server.
[DEFAULT]
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
ec2_private_dns_show_ip=True

# LOGS/STATE
verbose=True
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova

# AUTHENTICATION
auth_strategy=keystone
keystone_ec2_url=http://keystone:5000/v2.0/ec2tokens

# VOLUMES
#volume_driver=nova.volume.driver.ISCSIDriver
#volume_group=nova-volumes
#volume_name_template=volume-%08x
#iscsi_helper=tgtadm
volume_driver=nova.volume.driver.RBDDriver
volume_api_class=nova.volume.cinder.API
volumes_path=/var/lib/nova/volumes

# DATABASE
sql_connection=mysql://nova:password@mysql/nova

# COMPUTE
libvirt_type=kvm
compute_driver=libvirt.LibvirtDriver
instance_name_template=instance-%08x
api_paste_config=/etc/nova/api-paste.ini
libvirt_use_virtio_for_bridges=True

# RABBITMQ
rabbit_host=rabbitmq
rabbit_password=password

# GLANCE
image_service=nova.image.glance.GlanceImageService
glance_api_servers=glance:9292

# NETWORK
network_manager=nova.network.manager.FlatDHCPManager
force_dhcp_release=True
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
my_ip=172.17.123.17
public_interface=br100
vlan_interface=eth0
flat_network_bridge=br100
flat_interface=eth0
fixed_range=192.168.100.0/27

# NOVNC CONSOLE
novnc_enable=true
novncproxy_base_url=http://172.17.123.17:6080/vnc_auto.html
vncserver_proxyclient_address=127.0.0.1
vncserver_listen=0.0.0.0

/etc/nova/api.paste.ini

要設定 keystone 的相關資訊
...
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
auth_host = keystone
auth_port = 35357
auth_protocol = http
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
admin_tenant_name = service
admin_user = nova
admin_password = password
signing_dirname = /tmp/keystone-signing-nova

/etc/nova/nova_restart.sh

另外準備一個 nova_restart.sh, 方便等下使用
#!/bin/bash
for a in nova-network nova-compute nova-api nova-scheduler; do sudo service $a stop; done
for a in nova-consoleauth nova-cert novnc libvirt-bin; do sudo service $a stop; done
for a in nova-network nova-compute nova-api nova-scheduler; do sudo service $a start; done
for a in nova-consoleauth nova-cert novnc libvirt-bin; do sudo service $a start; done

啟動 nova 之後, 要記得成生成 database, 完成後就可以把服務啟動
root@nova:~$ chown -R nova:nova *
root@nova:~$ nova-manage db sync
root@nova:~$ /etc/nova/nova_restart.sh
root@nova:~$ nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-cert        nova                                 nova             enabled    :-)   2012-10-09 02:38:29
nova-scheduler   nova                                 nova             enabled    :-)   2012-10-09 02:38:29
nova-consoleauth nova                                 nova             enabled    :-)   2012-10-09 02:38:29
nova-compute     nova                                 nova             enabled    :-)   2012-10-09 02:38:31
nova-network     nova                                 nova             enabled    :-)   2012-10-09 02:38:30

root@nova:~$ ps aux | grep nova | grep python
nova      3611  0.3  0.7 209044 58800 ?        S    15:05   0:07 /usr/bin/python /usr/bin/nova-network --config-file=/etc/nova/nova.conf
nova      3623  0.4  0.8 1367636 67448 ?       Sl   15:05   0:10 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf
nova      3634  0.0  0.7 136864 58780 ?        S    15:05   0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      3645  0.2  0.7 276524 61796 ?        S    15:05   0:05 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf
nova      3656  0.2  0.6 202344 52512 ?        S    15:05   0:04 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf
nova      3667  0.2  0.6 202216 52248 ?        S    15:05   0:05 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf
nova      3755  0.0  0.2  95772 22824 ?        S    15:05   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      4164  0.0  0.9 250300 79596 ?        S    15:05   0:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      4171  0.0  0.6 135852 54420 ?        S    15:05   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      4173  0.0  0.6 137112 55960 ?        S    15:05   0:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nova      7092  0.0  0.2 122972 24216 ?        S    15:13   0:00 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
nova      7137  0.0  0.3 137492 25832 ?        S    15:13   0:00 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf

Test and Verification

驗証一下和所有的 component 是否正確連結, 首先驗証一下 Cinder
root@nova:~$ nova volume-create --display-name test 1
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| created_at          | 2012-10-09T08:35:06.518633           |
| display_description | None                                 |
| display_name        | test                                 |
| id                  | 728832a1-32d8-44f3-ba1a-8944adbeca11 |
| metadata            | {}                                   |
| size                | 1                                    |
| snapshot_id         | None                                 |
| status              | creating                             |
| volume_type         | None                                 |
+---------------------+--------------------------------------+

root@nova:~$ nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 728832a1-32d8-44f3-ba1a-8944adbeca11 | available | test         | 1    | None        |             |
+--------------------------------------+-----------+--------------+------+-------------+-------------+

root@nova:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 728832a1-32d8-44f3-ba1a-8944adbeca11 | available |     test     |  1   |     None    |             |
+--------------------------------------+-----------+--------------+------+-------------+-------------+

root@nova:~$ nova volume-delete test

再來驗証一下和 Glance 的連結是否正確
root@nova:~$ nova image-list
+--------------------------------------+---------------------+--------+--------+
| ID                                   | Name                | Status | Server |
+--------------------------------------+---------------------+--------+--------+
| fdc49609-6047-426c-a382-75928c0deb17 | Ubuntu-Precise      | ACTIVE |        |
| ad46b050-a03e-4d31-bc60-84f81806853b | tty-linux           | ACTIVE |        |
| e504fcf2-fdbd-4d15-be1c-49e24dd28458 | tty-linux-kernel    | ACTIVE |        |
| 5897d530-b625-4b7c-91eb-56313cf2741c | tty-linux-ramdisk   | ACTIVE |        |
+--------------------------------------+---------------------+--------+--------+

root@nova:~$ wget -c https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img -O stackimages/cirros.img

root@nova:~$ glance add name=cirros-0.3.0-x86_64 disk_format=qcow2 container_format=bare < cirros.img
Added new image with ID: 1e4a8f0c-235f-46ce-9aef-fc7fa143f141

root@nova:~$ glance image-list
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+
| 1e4a8f0c-235f-46ce-9aef-fc7fa143f141 | cirros-0.3.0-x86_64 | qcow2       | bare             | 9761280   | active |
| 5897d530-b625-4b7c-91eb-56313cf2741c | tty-linux-ramdisk   | ari         | ari              | 96629     | active |
| ad46b050-a03e-4d31-bc60-84f81806853b | tty-linux           | ami         | ami              | 25165824  | active |
| e504fcf2-fdbd-4d15-be1c-49e24dd28458 | tty-linux-kernel    | aki         | aki              | 4404752   | active |
| fdc49609-6047-426c-a382-75928c0deb17 | Ubuntu-Precise      | qcow2       | ovf              | 232718336 | active |
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+

root@nova:~$ nova image-list
+--------------------------------------+---------------------+--------+--------+
| ID                                   | Name                | Status | Server |
+--------------------------------------+---------------------+--------+--------+
| fdc49609-6047-426c-a382-75928c0deb17 | Ubuntu-Precise      | ACTIVE |        |
| 1e4a8f0c-235f-46ce-9aef-fc7fa143f141 | cirros-0.3.0-x86_64 | ACTIVE |        |
| ad46b050-a03e-4d31-bc60-84f81806853b | tty-linux           | ACTIVE |        |
| e504fcf2-fdbd-4d15-be1c-49e24dd28458 | tty-linux-kernel    | ACTIVE |        |
| 5897d530-b625-4b7c-91eb-56313cf2741c | tty-linux-ramdisk   | ACTIVE |        |
+--------------------------------------+---------------------+--------+--------+

Network Creation

要先成兩個網段, 一個是對內的 private ip, 一個是對外的floating ip
root@nova:~$ nova-manage network create private --multi_host=T --fixed_range_v4=192.168.100.0/27 --bridge=br100 --bridge_interface=eth0 --num_networks=1 --network_size=32
root@nova:~$ nova-manage floating create --ip_range=172.17.123.192/28
root@nova:~$ nova network-list
+--------------------------------------+---------+------------------+
| ID                                   | Label   | Cidr             |
+--------------------------------------+---------+------------------+
| 471a3258-1f30-458d-8476-262521597fbf | private | 192.168.100.0/27 |
+--------------------------------------+---------+------------------+

root@nova:~$ nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp         | 22        | 22      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

root@nova:~$ nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range  | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp        | -1        | -1      | 0.0.0.0/0 |              |
+-------------+-----------+---------+-----------+--------------+

Add Keypair

root@nova:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
8b:12:db:df:24:9e:31:05:da:8d:ed:7e:37:46:5f:9b root@nova
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|        .        |
|       o =       |
|    . . S +      |
|     + . +    . .|
|    o o = o  . .+|
|     . o O  . +E.|
|        + o. o . |
+-----------------+
root@nova:~$ nova keypair-add --pub_key ~/.ssh/id_rsa.pub mykey
root@nova:~$ nova keypair-list
+-------+-------------------------------------------------+
| Name  | Fingerprint                                     |
+-------+-------------------------------------------------+
| mykey | 8b:12:db:df:24:9e:31:05:da:8d:ed:7e:37:46:5f:9b |
+-------+-------------------------------------------------+

Boot a Virtual Machine

我們用 cirros-0.3.0-x86_64 來測試是否能成功新增一個 Virtual Machine
root@nova:~$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
| 1  | m1.tiny   | 512       | 0    | 0         |      | 1     | 1.0         | True      | {}          |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      | {}          |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      | {}          |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      | {}          |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      | {}          |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+-------------+
root@nova:~$ nova boot --flavor 1 --image 1e4a8f0c-235f-46ce-9aef-fc7fa143f141 --key_name mykey --security_group default vm1
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-SRV-ATTR:host                | nova                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname | nova                                                     |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000004                                        |
| OS-EXT-STS:power_state              | 0                                                        |
| OS-EXT-STS:task_state               | scheduling                                               |
| OS-EXT-STS:vm_state                 | building                                                 |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| adminPass                           | qmmhTYWn5N8K                                             |
| config_drive                        |                                                          |
| created                             | 2012-10-09T08:06:53Z                                     |
| flavor                              | m1.tiny                                                  |
| hostId                              | 09574c18e8c0a491179c061b91f64d31726f3d0c19ea4cee36ee0cc7 |
| id                                  | 5c983f6f-9d94-4f97-a6fb-1bf4a3aaa487                     |
| image                               | cirros-0.3.0-x86_64                                      |
| key_name                            | mykey                                                    |
| metadata                            | {}                                                       |
| name                                | vm1                                                      |
| progress                            | 0                                                        |
| security_groups                     | [{u'name': u'default'}]                                  |
| status                              | BUILD                                                    |
| tenant_id                           | eefa301a6a424e7da3d582649ad0e59e                         |
| updated                             | 2012-10-09T08:06:54Z                                     |
| user_id                             | fafd0583de8a4a1b93b924a6b2cb7eb5                         |
+-------------------------------------+----------------------------------------------------------+

root@nova:~$ nova list
+--------------------------------------+------+--------+-----------------------+
| ID                                   | Name | Status | Networks              |
+--------------------------------------+------+--------+-----------------------+
| 5c983f6f-9d94-4f97-a6fb-1bf4a3aaa487 | vm1  | ACTIVE | private=192.168.100.2 |
+--------------------------------------+------+--------+-----------------------+

# 網路有正確的接到 br100
root@nova:~$ brctl show
bridge name     bridge id               STP enabled     interfaces
br100           8000.00505682d12a       no              eth0
virbr0          8000.000000000000       yes

# libvirt 內也有看到這個 VM
root@nova:~$ virsh list
 Id Name                 State
----------------------------------
  2 instance-00000004    running

# 可以透過 vnc 去看
root@nova:~$ virsh vncdisplay 2
:0

# ssh 也沒問題
root@nova:~$ ssh cirros@192.168.100.2
The authenticity of host '192.168.100.2 (192.168.100.2)' can't be established.
RSA key fingerprint is 36:4c:6f:9c:40:a7:9f:07:13:6a:28:67:e2:1d:08:1c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.2' (RSA) to the list of known hosts.

# 連外也沒問題
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=52 time=10.089 ms
64 bytes from 8.8.8.8: seq=1 ttl=52 time=8.558 ms
64 bytes from 8.8.8.8: seq=2 ttl=52 time=11.982 ms
64 bytes from 8.8.8.8: seq=3 ttl=52 time=11.889 ms


# 驗証完後就砍掉吧!
root@nova:~$ nova delete 5c983f6f-9d94-4f97-a6fb-1bf4a3aaa487

Attach Volume

這裡我要測試透過 Cinder 產生的 volume 是否能 attach 到 vm 上
root@nova:~$ nova list
+--------------------------------------+------+--------+-----------------------+
| ID                                   | Name | Status | Networks              |
+--------------------------------------+------+--------+-----------------------+
| 321b2521-b144-4ec4-88ac-1916ae9d8427 | vm1  | ACTIVE | private=192.168.100.2 |
+--------------------------------------+------+--------+-----------------------+

root@nova:~$ nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 6051f6a4-c507-4d39-91f7-be7214b8d326 | available | test         | 30   | None        |             |
+--------------------------------------+-----------+--------------+------+-------------+-------------+

root@nova:~$ nova volume-attach 321b2521-b144-4ec4-88ac-1916ae9d8427 6051f6a4-c507-4d39-91f7-be7214b8d326 auto

+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 6051f6a4-c507-4d39-91f7-be7214b8d326 |
| serverId | 321b2521-b144-4ec4-88ac-1916ae9d8427 |
| volumeId | 6051f6a4-c507-4d39-91f7-be7214b8d326 |
+----------+--------------------------------------+

root@nova:~$ nova volume-list
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| 6051f6a4-c507-4d39-91f7-be7214b8d326 | in-use | test         | 30   | None        | 321b2521-b144-4ec4-88ac-1916ae9d8427 |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+

root@nova:~$ virsh list
 Id Name                 State
----------------------------------
  4 instance-00000006    running

root@nova:~$ virsh domblklist instance-00000006
Target     Source
------------------------------------------------
vda        /var/lib/nova/instances/instance-00000006/disk
vdb        rbd/volume-6051f6a4-c507-4d39-91f7-be7214b8d326.

root@nova:~$ ssh cirros@192.168.100.2
$ sudo fdisk -l

Disk /dev/vda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065    83875364    41929650   83  Linux

Disk /dev/vdb: 32.2 GB, 32212254720 bytes
16 heads, 63 sectors/track, 62415 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/vdb doesn't contain a valid partition table

看起來是成功了, 再多看一下這個 VM 的 xml 設定, 透過以下這個指令 virsh dumpxml instance-00000006 可以看到他新增了一個 disk, 是透過 rbd protocol
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/instance-00000006/disk'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='rbd' name='rbd/volume-6051f6a4-c507-4d39-91f7-be7214b8d326'/>
      <target dev='vdb' bus='virtio'/>
      <serial>6051f6a4-c507-4d39-91f7-be7214b8d326
      <alias name='virtio-disk1'/>       
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>     
    </disk> 
...  


Create Ubuntu(Precise) Virtual Machine

Ubuntu 的 image 可以到這邊下載
http://uec-images.ubuntu.com/precise/current/
裡面分成兩種, 一種是把 kernel image & machine image 分開的, 另一種則是用 qcow2 包起來

拆開的: precise-server-cloudimg-amd64-root.tar.gz
包起來的: precise-server-cloudimg-amd64-disk1.img
Openstack Folsom - Installation of Glance with Ceph 內有紀錄怎麼第二種的上傳方式, 至於第一種就要解壓縮後, 分別上傳 aki & ami, 然後連結起來. 這邊假設已經上傳好了

root@nova:~$ nova image-list
+--------------------------------------+---------------------+--------+--------+
| ID                                   | Name                | Status | Server |
+--------------------------------------+---------------------+--------+--------+
| fdc49609-6047-426c-a382-75928c0deb17 | Ubuntu-Precise      | ACTIVE |        |
| 1e4a8f0c-235f-46ce-9aef-fc7fa143f141 | cirros-0.3.0-x86_64 | ACTIVE |        |
| ad46b050-a03e-4d31-bc60-84f81806853b | tty-linux           | ACTIVE |        |
| e504fcf2-fdbd-4d15-be1c-49e24dd28458 | tty-linux-kernel    | ACTIVE |        |
| 5897d530-b625-4b7c-91eb-56313cf2741c | tty-linux-ramdisk   | ACTIVE |        |
+--------------------------------------+---------------------+--------+--------+
root@nova:~$ nova boot --flavor 2 --image Ubuntu-Precise --key_name mykey --security_group default vm
+-------------------------------------+----------------------------------------------------------+
| Property                            | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-SRV-ATTR:host                | nova                                                     |
| OS-EXT-SRV-ATTR:hypervisor_hostname | nova                                                     |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000000c                                        |
| OS-EXT-STS:power_state              | 0                                                        |
| OS-EXT-STS:task_state               | scheduling                                               |
| OS-EXT-STS:vm_state                 | building                                                 |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| adminPass                           | 5q4sWjHENyvP                                             |
| config_drive                        |                                                          |
| created                             | 2012-10-10T02:52:40Z                                     |
| flavor                              | m1.small                                                 |
| hostId                              | 09574c18e8c0a491179c061b91f64d31726f3d0c19ea4cee36ee0cc7 |
| id                                  | 019e2db9-cabe-4711-9b95-ceaefd97f22e                     |
| image                               | Ubuntu-Precise                                           |
| key_name                            | mykey                                                    |
| metadata                            | {}                                                       |
| name                                | vm                                                       |
| progress                            | 0                                                        |
| security_groups                     | [{u'name': u'default'}]                                  |
| status                              | BUILD                                                    |
| tenant_id                           | eefa301a6a424e7da3d582649ad0e59e                         |
| updated                             | 2012-10-10T02:52:40Z                                     |
| user_id                             | fafd0583de8a4a1b93b924a6b2cb7eb5                         |
+-------------------------------------+----------------------------------------------------------+
root@nova:~$ nova list
+--------------------------------------+------+--------+-----------------------+
| ID                                   | Name | Status | Networks              |
+--------------------------------------+------+--------+-----------------------+
| 019e2db9-cabe-4711-9b95-ceaefd97f22e | vm   | ACTIVE | private=192.168.100.4 |
+--------------------------------------+------+--------+-----------------------+

root@nova:~$ ssh ubuntu@192.168.100.4
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-31-virtual x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Wed Oct 10 03:17:49 UTC 2012

  System load:  0.0               Processes:           60
  Usage of /:   3.3% of 19.67GB   Users logged in:     0
  Memory usage: 2%                IP address for eth0: 192.168.100.4
  Swap usage:   0%

  Graph this data and manage this system at https://landscape.canonical.com/

0 packages can be updated.
0 updates are security updates.

Get cloud support with Ubuntu Advantage Cloud Guest
  http://www.ubuntu.com/business/services/cloud
Last login: Wed Oct 10 02:55:30 2012 from 192.168.100.3
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@vm:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=52 time=9.96 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=52 time=7.63 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=52 time=5.18 ms


註1:

在執行 nova 指令時, 會不斷的叫你輸入 keyring 的 password, 目前也有人把他提報成一個 bug.
https://bugs.launchpad.net/python-novaclient/+bug/1020238
http://wiki.openstack.org/KeyringSupport
如果不想要一直輸入 password, 可以在 .bashrc 內加上一行
alias nova='nova --no-cache'

註2:

在裝完 horizon 之後, 發現 vnc 起不來, 才發現 vncproxy 啟動失敗
在 /var/log/upstart/nova-novncproxy.log 內
Traceback (most recent call last):
  File "/usr/bin/nova-novncproxy", line 29, in 
    import websockify
在網路上找了一下發現是 known issue, 只需要去下載最新的 package, 就會解掉這個問題
https://bugs.launchpad.net/ubuntu/+source/websockify/+bug/1060374

root@nova:~$ wget https://launchpad.net/ubuntu/+archive/primary/+files/websockify_0.2~20121002-0ubuntu1_amd64.deb
root@nova:~$ dpkg -i websockify_0.2~20121002-0ubuntu1_amd64.deb
root@nova:~$ ./nova_restart.sh