Sharing

2013年6月28日 星期五

Network in VirtualBox


因為最近工作的平台又換成 Windows, 所以開始使用 VirtualBox
但發現 VirtualBox 的網路設定名詞和 KVM/Vmware 有些不太一樣
過去都把 VM 藏在 Host 背後, 所以就選 NAT, 但沒想到他的 NAT 是 isolated.
每台 VM 之間是不互通的, 如果要互通要選 Host-Only Networking

個人覺得這篇寫的最清楚, 又有圖
https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1

官網
http://www.virtualbox.org/manual/ch06.html
http://www.virtualbox.org/manual/ch09.html#changenat


2013年6月25日 星期二

Static vs Dynamic Scoping

最近在看 Puppet, 裡面提到 3.0 和 2.7 版的差異
其中有一項是要將 Dynamic Scoping 換成 Static Scoping.
http://docs.puppetlabs.com/guides/scope_and_puppet.html
http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html
看起來寫的很詳細,但我就是看不懂=.=
只好另外找說明

直接用程式碼說明,最快速
http://msujaws.wordpress.com/2011/05/03/static-vs-dynamic-scoping/


Static Scoping
從 Definition 找 Scope
const int b = 5;
int foo()
{
   int a = b + 5;
   return a;
}
 
int bar()
{
   int b = 2;
   return foo();
}
 
int main()
{
   foo(); // returns 10
   bar(); // returns 10
   return 0;
}

Dynamic scoping
從 Call Stack 找 Scope
const int b = 5;
int foo()
{
   int a = b + 5;
   return a;
}
 
int bar()
{
   int b = 2;
   return foo();
}
 
int main()
{
   foo(); // returns 10
   bar(); // returns 7
   return 0;
}



稍微學術一點,另外 Lixical Scoping 等同於 Static Scoping
http://c2.com/cgi/wiki?DynamicScoping
http://c2.com/cgi/wiki?LexicalScoping

最完整的說明,也有從不同 Programming Language 來說明
http://en.wikipedia.org/wiki/Scope_(computer_science)

順便複習一下 Python 的 Scoping 原則
LEGB rule (Local, Enclosing, Global, Built-in)


DenyHosts: Remove / Delete an IP address


原來 Ubuntu 裡面有一個 service 叫 DenyHost, 會主動更新 /etc/hosts.deny
相關文章如下

http://www.cyberciti.biz/faq/linux-unix-delete-remove-ip-address-that-denyhosts-blocked/
http://denyhosts.sourceforge.net/faq.html

cron-apt


今天為了找一個 dpkg 被 lock 的問題,意外發現系統有裝 cron-apt
平時如果想要自動做 update/upgrade package, 就可以使用這個,

apt-get install cron-tab

安裝好之後,會發現在 /etc/cron.d 下多了一個 cron-apt 檔案, 內容就是每天 4 點鐘自動執行 cron-apt

#
# Regular cron jobs for the cron-apt package
#
# Every night at 4 o'clock.
0 4     * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt
# Every hour.
# 0 *   * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt /etc/cron-apt/config2
# Every five minutes.
# */5 * * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt /etc/cron-apt/config2

另外可以看一下 /etc/cron-apt/action.d, 就知道 cron-apt 做了那些事
預設是只做 update repository list 以及 download new packages

/etc/cron-apt/action.d$ ls
0-update  3-download

/etc/cron-apt/action.d$ cat 0-update
update -o quiet=2

/etc/cron-apt/action.d$ cat 3-download
autoclean -y
dist-upgrade -d -y -o APT::Get::Show-Upgraded=true

相關的 log 可以看 /var/log/cron-apt/log

CRON-APT RUN [/etc/cron-apt/config]: Tue Jun 25 04:00:01 BST 2013
CRON-APT SLEEP: 2274, Tue Jun 25 04:37:55 BST 2013
CRON-APT ACTION: 0-update
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 update -o quiet=2
CRON-APT ACTION: 3-download
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 autoclean -y
Reading package lists...
Building dependency tree...
Reading state information...
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -d -y -o APT::Get::Show-Upgraded=true
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  ruby-rgen
The following packages will be upgraded:
  cpan-libnet-http-perl facter libpython2.7 libvirt0 nodejs puppet
  puppet-common puppetmaster puppetmaster-common python python-dev
  python-django python-minimal python2.7 python2.7-dev python2.7-minimal
  uwsgi-core uwsgi-plugin-carbon uwsgi-plugin-cgi uwsgi-plugin-corerouter
  uwsgi-plugin-http uwsgi-plugin-psgi uwsgi-plugin-python
  uwsgi-plugin-rack-ruby1.8 uwsgi-plugin-rack-ruby1.9.1
25 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/49.3 MB of archives.
After this operation, 5977 kB of additional disk space will be used.
Download complete and in download only mode


Reference:

http://www.the-art-of-web.com/system/cron-apt-wheezy/#.UckR2hoW1Xo
http://www.debianadmin.com/automatic-update-of-packages-using-cron-apt.html
http://www.techrepublic.com/article/automatically-update-your-ubuntu-system-with-cron-apt/6310660

2013年6月13日 星期四

Problem: How to manage python package from apt-get?

目前 pip&distibute 解決了大部份  python package 的 installation 時遇到的 dependency 問題.
http://pythonhosted.org/distribute/
http://guide.python-distribute.org/


但如果遇到和外界的 c library 有 dependency 時, ex: pyzmq --> libzmq3  或是 pymysqldb --> libmysqlclient, 就無法在 python 的世界裡解決了, 必須要跳脫出來. 我想這也是為什麼 Ubuntu 在 python package 之外經常要再包一層看似沒做什麼事情的 .deb 檔.  ex: python-zmq, python-crypto, 透過 .deb 檔可以讓 python package 和其它 c library 建立起關係. 並且有時候可以做更多的事情, 比方說在 /etc/ 下建立 .conf 檔案, 在 /etc/init.d/ 下建立這樣一來 python package 和 OS 的關係


但問題又來了,  precise 內建的 python-crypto 是 2.4 版, 如果我要裝 2.6 版要怎麼做?

$ pip install pycrypto==2.6 
$ dpkg -l | grep python-crypto
ii  python-crypto                                       2.4.1-1ubuntu0.1                                    cryptographic algorithms and protocols for Python
$ pip freeze | grep pycrypto
pycrypto==2.6

用 pip 直接安裝是最快速的方式, 但會讓 apt/dpkg 內的資訊不一致, 這在 management 以及 deployment 有時會造成麻煩
目前還沒有找到直接的解法, 先紀錄下來, 未來有找到解法時再更新

相關 material:

http://www.g-loaded.eu/2008/01/28/how-to-extract-rpm-or-deb-packages/

http://www.pyinstaller.org/

http://stackoverflow.com/questions/4700178/should-i-bundle-c-libraries-with-my-python-application

http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

http://developer.ubuntu.com/packaging/html/
http://developer.ubuntu.com/packaging/html/python-packaging.html
http://askubuntu.com/questions/90764/how-do-i-create-a-deb-package-for-a-single-python-script
https://wiki.ubuntu.com/PackagingGuideDeprecated/Python









2013年6月10日 星期一

File::Find in Perl

最近在寫一個小程式要處理檔案, 所以需要工具把一個 folder 下的檔案列出來,整理一下有試過的方法

readdir

http://perlmeme.org/faqs/file_io/directory_listing.html
這種方法的缺點是,

  • 會列出  "." 和 ".." 
  • 要自己區分 folder & file
  • 沒有辦法加 Filter
  • 沒有按照字母順序

#!/usr/bin/perl
use strict;
use warnings;

my $directory = '/tmp';

opendir (DIR, $directory) or die $!;

while (my $file = readdir(DIR)) {
    print "$file\n";
}

closedir(DIR);

Glob

http://perldoc.perl.org/functions/glob.html

語法很簡單, 而且回傳有按照字母順序,也可以做簡單的 filter, 不過遇到複雜一點的 case, 可能就不夠用

@filelist = glob "/usr/lib/perl/*.pl";

File::Find

這個方案就強大的多, 要做什麼事幾乎都做的到,因為 filter function 是自己寫,
也有 find2perl 這個工具幫你產生 code. 但壞處就是什麼都要自己刻

use File::Find;
find(\&wanted, @directories_to_search);
sub wanted { ... }

File::Find::Rule

http://search.cpan.org/~rclamp/File-Find-Rule-0.33/lib/File/Find/Rule.pm
這個是 File::Find 的改良版,他提供不少小工具來幫助你做 filter, 幾乎和 Linux 裡 find 功能差不多,不過他仍然有一些限制
  • 沒有按照字母順序,必須自己用 sort 排序
  • modified, accessed, changed 這三個理論上應該要回傳天數,但在這邊只回傳 Boolean 值

實作 find -type f -name "*.pl" -mtime +3 -maxdepth 1
my $rule =  File::Find::Rule->new;
$rule->file;
$rule->maxdepth(1);
$rule->name("*.pl");
$rule->exec( sub { -M $_ > 3});
my @fid_files = sort { $a cmp $b } $rule->in($curr_path);

File Operation in Perl

再復習一下 File 的一些基本操作
https://metacpan.org/module/perlfunc#Alphabetical-Listing-of-Perl-Functions

Find in Bash Shell

也復習一下 Find 的操作

http://www.gnu.org/software/findutils/manual/html_mono/find.html

-mtime n

    Numeric arguments can be specified as

       +n     for greater than n,

       -n     for less than n,

       n      for exactly n.


# find the files modified in 30 days.
$ find . -iname "*.c" -mtime -30 -type f

# find the files modified over 30 days.
$ find . -iname "*.c" -mtime +30 -type f

# find the files modified between 20-30 days ago.
$ find . -mtime -30 -mtime +20 -type f

# find the files modified in 30 days, but the based time is 00:00:00 today, not current time.
$ find . -daystart -mtime 0 -type f

# find the files modified between 2012-01-01 and 2013-01-01
$ touch --date "2012-01-01" /tmp/start
$ touch --date "2013-01-01" /tmp/end
$ find . -type f -newer /tmp/start -not -newer /tmp/end


http://www.cyberciti.biz/faq/linux-unix-osxfind-files-by-date/

http://blog.miniasp.com/post/2010/08/27/Linux-find-command-tips-and-notice.aspx

http://www.electrictoolbox.com/using-find-to-locate-files-modified-in-the-last-24-hours/




Linux Process Introduction and Simple Monitoring

簡介
http://careers.directi.com/display/tu/Understanding+Processes+in+Linux

最需要馬上知道的應該是這個 Process State

the state of a process:
D uninterruptible sleep (usually IO)
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent.

# list all tasks that is not in running state
$ ps -eN r  
# list all tasks that is in running state
$ ps -e r



快速列出目前有那些 process 是 Dead 狀態, 也可以很簡易的改成別的

$ top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
top - 03:06:19 up 12 days, 22:30,  2 users,  load average: 0.00, 0.04, 0.05
Tasks: 120 total,   1 running, 118 sleeping,   1 stopped,   0 zombie
Cpu(s):  0.2%us,  0.4%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1985624k total,  1682512k used,   303112k free,   211976k buffers
Swap:   407548k total,     2836k used,   404712k free,   708468k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                   
Total status D: 

Load average in top
http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

Other
https://blogs.oracle.com/ksplice/entry/disown_zombie_children_and_the
https://idea.popcount.org/2012-12-11-linux-process-states/
http://www.jollen.org/blog/2008/07/process_state_wait_queue.html

2013年6月6日 星期四

[feedly] MySQL HA 的選擇…

DevOp 工具: twill - a simple scripting language for Web browsing

由於需要 monitor 一些 Service 的狀況, 所以必須寫一些 script 去測試網站能不能登入之類的, 所以需要模擬 browser 的行為。今天看到同事用這個工具來做這件事,當我發現他是用 Python 寫的工具,馬上試用了一下,覺得還滿好用的,簡單的操作應該都沒問題。而且他也承襲了 Python 的風格 => Interactive Prompt, 立即寫立即測試, 不然等所有的 code 寫完再測.

官網: http://twill.idyll.org/

以下先來示範一下如何登入 yahoo

>>> import twill.shell
>>> twill.shell.main()

 -= Welcome to twill! =-

current page:  *empty page*

>> help

Undocumented commands:
======================
add_auth             fa           info             save_html           title
add_extra_header     find         load_cookies     setglobal           url
agent                follow       notfind          setlocal
back                 formaction   redirect_error   show
clear_cookies        formclear    redirect_output  show_cookies
clear_extra_headers  formfile     reload           show_extra_headers
code                 formvalue    reset_browser    showforms
config               fv           reset_error      showhistory
debug                get_browser  reset_output     showlinks
echo                 getinput     run              sleep
exit                 getpassword  runfile          submit
extend_with          go           save_cookies     tidy_ok


>> go https://login.yahoo.com/config/login
==> at https://login.yahoo.com/config/login
current page: https://login.yahoo.com/config/login

>> showforms
Form name=login_form (#2)
## ## __Name__________________ __Type___ __ID________ __Value__________________
29    login                    text      username
30    passwd                   password  passwd
31    .persistent              checkbox  persistent   [] of ['y']
32 1  .save                    submi ... .save

>> setlocal login pjack1981
current page: https://login.yahoo.com/config/login
>> setlocal passwd xxxxxxxx
current page: https://login.yahoo.com/config/login
>> formvalue 2 login $login
current page: https://login.yahoo.com/config/login
>> formvalue 2 passwd $passwd
current page: https://login.yahoo.com/config/login

>> showforms
Form name=login_form (#2)
## ## __Name__________________ __Type___ __ID________ __Value__________________
29    login                    text      username     pjack1981
30    passwd                   password  passwd       xxxxxxxx
31    .persistent              checkbox  persistent   [] of ['y']
32 1  .save                    submi ... .save

>> submit
Note: submit is using submit button: name=".save", value=""

Following HTTP-EQUIV=REFRESH to http://my.yahoo.com
current page: http://my.yahoo.com
>> code 200
current page: http://my.yahoo.com

>> code 20

ERROR: code is 200 != 20

current page: http://my.yahoo.com


所以你可以先用 interactive mode 測一下你要的功能, 接下來就把你剛剛輸入的指令集合起來, 這樣就變成一個很簡單的測試 script 囉!

$ cat test.twill
setlocal login pjack1981
setlocal passwd xxxxxxxx

go https://login.yahoo.com/config/login
formvalue 2 login $login
formvalue 2 passwd $passwd
submit

code 200

$ twill-sh test.twill
>> EXECUTING FILE test.twill
AT LINE: test.twill:1
AT LINE: test.twill:2
AT LINE: test.twill:4
==> at https://login.yahoo.com/config/login
AT LINE: test.twill:5
AT LINE: test.twill:6
AT LINE: test.twill:7
Note: submit is using submit button: name=".save", value=""

Following HTTP-EQUIV=REFRESH to http://my.yahoo.com
AT LINE: test.twill:9
--
1 of 1 files SUCCEEDED.

其它 Example

http://stackoverflow.com/questions/2688408/how-can-i-put-all-twill-commands-together-into-one-piece-of-code-in-a-py-file

補充:
其它 testing tool for web browsing.
https://pypi.python.org/pypi/zope.testbrowser/4.0.2

2013年6月2日 星期日

DevOp 的好朋友 Multiple connection manager & screen

Gnome Connection ManagerByobu 聯合起來一起用, 之前也寫了一篇介紹 GCM
利用 GCM 可以快速的開啟每個 Node 的連結
利用 Byobu 可以在每一點同時做很多事, 又不必開新的 Tab, 而且 Byobu 特性是即使不小心斷線, 下次再連進去也可以接上原有的 session, 不必擔心正在跑的 program 被中斷, 甚至從不同機器連進去也可以看到同樣的畫面. (偶而還被我拿來當 Share Screen, 和遠端的同事接力 Debug)

所以兩個結合起來真是太贊了!


[轉] Google 帶你了解20多年來地球的變化