Sharing

2013年6月28日 星期五

Network in VirtualBox


因為最近工作的平台又換成 Windows, 所以開始使用 VirtualBox
但發現 VirtualBox 的網路設定名詞和 KVM/Vmware 有些不太一樣
過去都把 VM 藏在 Host 背後, 所以就選 NAT, 但沒想到他的 NAT 是 isolated.
每台 VM 之間是不互通的, 如果要互通要選 Host-Only Networking

個人覺得這篇寫的最清楚, 又有圖
https://blogs.oracle.com/fatbloke/entry/networking_in_virtualbox1

官網
http://www.virtualbox.org/manual/ch06.html
http://www.virtualbox.org/manual/ch09.html#changenat


2013年6月25日 星期二

Static vs Dynamic Scoping

最近在看 Puppet, 裡面提到 3.0 和 2.7 版的差異
其中有一項是要將 Dynamic Scoping 換成 Static Scoping.
http://docs.puppetlabs.com/guides/scope_and_puppet.html
http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html
看起來寫的很詳細,但我就是看不懂=.=
只好另外找說明

直接用程式碼說明,最快速
http://msujaws.wordpress.com/2011/05/03/static-vs-dynamic-scoping/


Static Scoping
從 Definition 找 Scope
const int b = 5;
int foo()
{
   int a = b + 5;
   return a;
}
 
int bar()
{
   int b = 2;
   return foo();
}
 
int main()
{
   foo(); // returns 10
   bar(); // returns 10
   return 0;
}

Dynamic scoping
從 Call Stack 找 Scope
const int b = 5;
int foo()
{
   int a = b + 5;
   return a;
}
 
int bar()
{
   int b = 2;
   return foo();
}
 
int main()
{
   foo(); // returns 10
   bar(); // returns 7
   return 0;
}



稍微學術一點,另外 Lixical Scoping 等同於 Static Scoping
http://c2.com/cgi/wiki?DynamicScoping
http://c2.com/cgi/wiki?LexicalScoping

最完整的說明,也有從不同 Programming Language 來說明
http://en.wikipedia.org/wiki/Scope_(computer_science)

順便複習一下 Python 的 Scoping 原則
LEGB rule (Local, Enclosing, Global, Built-in)


DenyHosts: Remove / Delete an IP address


原來 Ubuntu 裡面有一個 service 叫 DenyHost, 會主動更新 /etc/hosts.deny
相關文章如下

http://www.cyberciti.biz/faq/linux-unix-delete-remove-ip-address-that-denyhosts-blocked/
http://denyhosts.sourceforge.net/faq.html

cron-apt


今天為了找一個 dpkg 被 lock 的問題,意外發現系統有裝 cron-apt
平時如果想要自動做 update/upgrade package, 就可以使用這個,

apt-get install cron-tab

安裝好之後,會發現在 /etc/cron.d 下多了一個 cron-apt 檔案, 內容就是每天 4 點鐘自動執行 cron-apt

#
# Regular cron jobs for the cron-apt package
#
# Every night at 4 o'clock.
0 4     * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt
# Every hour.
# 0 *   * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt /etc/cron-apt/config2
# Every five minutes.
# */5 * * * *   root    test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt /etc/cron-apt/config2

另外可以看一下 /etc/cron-apt/action.d, 就知道 cron-apt 做了那些事
預設是只做 update repository list 以及 download new packages

/etc/cron-apt/action.d$ ls
0-update  3-download

/etc/cron-apt/action.d$ cat 0-update
update -o quiet=2

/etc/cron-apt/action.d$ cat 3-download
autoclean -y
dist-upgrade -d -y -o APT::Get::Show-Upgraded=true

相關的 log 可以看 /var/log/cron-apt/log

CRON-APT RUN [/etc/cron-apt/config]: Tue Jun 25 04:00:01 BST 2013
CRON-APT SLEEP: 2274, Tue Jun 25 04:37:55 BST 2013
CRON-APT ACTION: 0-update
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 update -o quiet=2
CRON-APT ACTION: 3-download
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 autoclean -y
Reading package lists...
Building dependency tree...
Reading state information...
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -d -y -o APT::Get::Show-Upgraded=true
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  ruby-rgen
The following packages will be upgraded:
  cpan-libnet-http-perl facter libpython2.7 libvirt0 nodejs puppet
  puppet-common puppetmaster puppetmaster-common python python-dev
  python-django python-minimal python2.7 python2.7-dev python2.7-minimal
  uwsgi-core uwsgi-plugin-carbon uwsgi-plugin-cgi uwsgi-plugin-corerouter
  uwsgi-plugin-http uwsgi-plugin-psgi uwsgi-plugin-python
  uwsgi-plugin-rack-ruby1.8 uwsgi-plugin-rack-ruby1.9.1
25 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/49.3 MB of archives.
After this operation, 5977 kB of additional disk space will be used.
Download complete and in download only mode


Reference:

http://www.the-art-of-web.com/system/cron-apt-wheezy/#.UckR2hoW1Xo
http://www.debianadmin.com/automatic-update-of-packages-using-cron-apt.html
http://www.techrepublic.com/article/automatically-update-your-ubuntu-system-with-cron-apt/6310660

2013年6月13日 星期四

Problem: How to manage python package from apt-get?

目前 pip&distibute 解決了大部份  python package 的 installation 時遇到的 dependency 問題.
http://pythonhosted.org/distribute/
http://guide.python-distribute.org/


但如果遇到和外界的 c library 有 dependency 時, ex: pyzmq --> libzmq3  或是 pymysqldb --> libmysqlclient, 就無法在 python 的世界裡解決了, 必須要跳脫出來. 我想這也是為什麼 Ubuntu 在 python package 之外經常要再包一層看似沒做什麼事情的 .deb 檔.  ex: python-zmq, python-crypto, 透過 .deb 檔可以讓 python package 和其它 c library 建立起關係. 並且有時候可以做更多的事情, 比方說在 /etc/ 下建立 .conf 檔案, 在 /etc/init.d/ 下建立這樣一來 python package 和 OS 的關係


但問題又來了,  precise 內建的 python-crypto 是 2.4 版, 如果我要裝 2.6 版要怎麼做?

$ pip install pycrypto==2.6 
$ dpkg -l | grep python-crypto
ii  python-crypto                                       2.4.1-1ubuntu0.1                                    cryptographic algorithms and protocols for Python
$ pip freeze | grep pycrypto
pycrypto==2.6

用 pip 直接安裝是最快速的方式, 但會讓 apt/dpkg 內的資訊不一致, 這在 management 以及 deployment 有時會造成麻煩
目前還沒有找到直接的解法, 先紀錄下來, 未來有找到解法時再更新

相關 material:

http://www.g-loaded.eu/2008/01/28/how-to-extract-rpm-or-deb-packages/

http://www.pyinstaller.org/

http://stackoverflow.com/questions/4700178/should-i-bundle-c-libraries-with-my-python-application

http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

http://developer.ubuntu.com/packaging/html/
http://developer.ubuntu.com/packaging/html/python-packaging.html
http://askubuntu.com/questions/90764/how-do-i-create-a-deb-package-for-a-single-python-script
https://wiki.ubuntu.com/PackagingGuideDeprecated/Python









2013年6月10日 星期一

File::Find in Perl

最近在寫一個小程式要處理檔案, 所以需要工具把一個 folder 下的檔案列出來,整理一下有試過的方法

readdir

http://perlmeme.org/faqs/file_io/directory_listing.html
這種方法的缺點是,

  • 會列出  "." 和 ".." 
  • 要自己區分 folder & file
  • 沒有辦法加 Filter
  • 沒有按照字母順序

#!/usr/bin/perl
use strict;
use warnings;

my $directory = '/tmp';

opendir (DIR, $directory) or die $!;

while (my $file = readdir(DIR)) {
    print "$file\n";
}

closedir(DIR);

Glob

http://perldoc.perl.org/functions/glob.html

語法很簡單, 而且回傳有按照字母順序,也可以做簡單的 filter, 不過遇到複雜一點的 case, 可能就不夠用

@filelist = glob "/usr/lib/perl/*.pl";

File::Find

這個方案就強大的多, 要做什麼事幾乎都做的到,因為 filter function 是自己寫,
也有 find2perl 這個工具幫你產生 code. 但壞處就是什麼都要自己刻

use File::Find;
find(\&wanted, @directories_to_search);
sub wanted { ... }

File::Find::Rule

http://search.cpan.org/~rclamp/File-Find-Rule-0.33/lib/File/Find/Rule.pm
這個是 File::Find 的改良版,他提供不少小工具來幫助你做 filter, 幾乎和 Linux 裡 find 功能差不多,不過他仍然有一些限制
  • 沒有按照字母順序,必須自己用 sort 排序
  • modified, accessed, changed 這三個理論上應該要回傳天數,但在這邊只回傳 Boolean 值

實作 find -type f -name "*.pl" -mtime +3 -maxdepth 1
my $rule =  File::Find::Rule->new;
$rule->file;
$rule->maxdepth(1);
$rule->name("*.pl");
$rule->exec( sub { -M $_ > 3});
my @fid_files = sort { $a cmp $b } $rule->in($curr_path);

File Operation in Perl

再復習一下 File 的一些基本操作
https://metacpan.org/module/perlfunc#Alphabetical-Listing-of-Perl-Functions

Find in Bash Shell

也復習一下 Find 的操作

http://www.gnu.org/software/findutils/manual/html_mono/find.html

-mtime n

    Numeric arguments can be specified as

       +n     for greater than n,

       -n     for less than n,

       n      for exactly n.


# find the files modified in 30 days.
$ find . -iname "*.c" -mtime -30 -type f

# find the files modified over 30 days.
$ find . -iname "*.c" -mtime +30 -type f

# find the files modified between 20-30 days ago.
$ find . -mtime -30 -mtime +20 -type f

# find the files modified in 30 days, but the based time is 00:00:00 today, not current time.
$ find . -daystart -mtime 0 -type f

# find the files modified between 2012-01-01 and 2013-01-01
$ touch --date "2012-01-01" /tmp/start
$ touch --date "2013-01-01" /tmp/end
$ find . -type f -newer /tmp/start -not -newer /tmp/end


http://www.cyberciti.biz/faq/linux-unix-osxfind-files-by-date/

http://blog.miniasp.com/post/2010/08/27/Linux-find-command-tips-and-notice.aspx

http://www.electrictoolbox.com/using-find-to-locate-files-modified-in-the-last-24-hours/




Linux Process Introduction and Simple Monitoring

簡介
http://careers.directi.com/display/tu/Understanding+Processes+in+Linux

最需要馬上知道的應該是這個 Process State

the state of a process:
D uninterruptible sleep (usually IO)
R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
T stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent.

# list all tasks that is not in running state
$ ps -eN r  
# list all tasks that is in running state
$ ps -e r



快速列出目前有那些 process 是 Dead 狀態, 也可以很簡易的改成別的

$ top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
top - 03:06:19 up 12 days, 22:30,  2 users,  load average: 0.00, 0.04, 0.05
Tasks: 120 total,   1 running, 118 sleeping,   1 stopped,   0 zombie
Cpu(s):  0.2%us,  0.4%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1985624k total,  1682512k used,   303112k free,   211976k buffers
Swap:   407548k total,     2836k used,   404712k free,   708468k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                   
Total status D: 

Load average in top
http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

Other
https://blogs.oracle.com/ksplice/entry/disown_zombie_children_and_the
https://idea.popcount.org/2012-12-11-linux-process-states/
http://www.jollen.org/blog/2008/07/process_state_wait_queue.html

2013年6月6日 星期四

[feedly] MySQL HA 的選擇…

DevOp 工具: twill - a simple scripting language for Web browsing

由於需要 monitor 一些 Service 的狀況, 所以必須寫一些 script 去測試網站能不能登入之類的, 所以需要模擬 browser 的行為。今天看到同事用這個工具來做這件事,當我發現他是用 Python 寫的工具,馬上試用了一下,覺得還滿好用的,簡單的操作應該都沒問題。而且他也承襲了 Python 的風格 => Interactive Prompt, 立即寫立即測試, 不然等所有的 code 寫完再測.

官網: http://twill.idyll.org/

以下先來示範一下如何登入 yahoo

>>> import twill.shell
>>> twill.shell.main()

 -= Welcome to twill! =-

current page:  *empty page*

>> help

Undocumented commands:
======================
add_auth             fa           info             save_html           title
add_extra_header     find         load_cookies     setglobal           url
agent                follow       notfind          setlocal
back                 formaction   redirect_error   show
clear_cookies        formclear    redirect_output  show_cookies
clear_extra_headers  formfile     reload           show_extra_headers
code                 formvalue    reset_browser    showforms
config               fv           reset_error      showhistory
debug                get_browser  reset_output     showlinks
echo                 getinput     run              sleep
exit                 getpassword  runfile          submit
extend_with          go           save_cookies     tidy_ok


>> go https://login.yahoo.com/config/login
==> at https://login.yahoo.com/config/login
current page: https://login.yahoo.com/config/login

>> showforms
Form name=login_form (#2)
## ## __Name__________________ __Type___ __ID________ __Value__________________
29    login                    text      username
30    passwd                   password  passwd
31    .persistent              checkbox  persistent   [] of ['y']
32 1  .save                    submi ... .save

>> setlocal login pjack1981
current page: https://login.yahoo.com/config/login
>> setlocal passwd xxxxxxxx
current page: https://login.yahoo.com/config/login
>> formvalue 2 login $login
current page: https://login.yahoo.com/config/login
>> formvalue 2 passwd $passwd
current page: https://login.yahoo.com/config/login

>> showforms
Form name=login_form (#2)
## ## __Name__________________ __Type___ __ID________ __Value__________________
29    login                    text      username     pjack1981
30    passwd                   password  passwd       xxxxxxxx
31    .persistent              checkbox  persistent   [] of ['y']
32 1  .save                    submi ... .save

>> submit
Note: submit is using submit button: name=".save", value=""

Following HTTP-EQUIV=REFRESH to http://my.yahoo.com
current page: http://my.yahoo.com
>> code 200
current page: http://my.yahoo.com

>> code 20

ERROR: code is 200 != 20

current page: http://my.yahoo.com


所以你可以先用 interactive mode 測一下你要的功能, 接下來就把你剛剛輸入的指令集合起來, 這樣就變成一個很簡單的測試 script 囉!

$ cat test.twill
setlocal login pjack1981
setlocal passwd xxxxxxxx

go https://login.yahoo.com/config/login
formvalue 2 login $login
formvalue 2 passwd $passwd
submit

code 200

$ twill-sh test.twill
>> EXECUTING FILE test.twill
AT LINE: test.twill:1
AT LINE: test.twill:2
AT LINE: test.twill:4
==> at https://login.yahoo.com/config/login
AT LINE: test.twill:5
AT LINE: test.twill:6
AT LINE: test.twill:7
Note: submit is using submit button: name=".save", value=""

Following HTTP-EQUIV=REFRESH to http://my.yahoo.com
AT LINE: test.twill:9
--
1 of 1 files SUCCEEDED.

其它 Example

http://stackoverflow.com/questions/2688408/how-can-i-put-all-twill-commands-together-into-one-piece-of-code-in-a-py-file

補充:
其它 testing tool for web browsing.
https://pypi.python.org/pypi/zope.testbrowser/4.0.2

2013年6月2日 星期日

DevOp 的好朋友 Multiple connection manager & screen

把 Gnome Connection Manager 和 Byobu 聯合起來一起用, 之前也寫了一篇介紹 GCM
利用 GCM 可以快速的開啟每個 Node 的連結
利用 Byobu 可以在每一點同時做很多事, 又不必開新的 Tab, 而且 Byobu 特性是即使不小心斷線, 下次再連進去也可以接上原有的 session, 不必擔心正在跑的 program 被中斷, 甚至從不同機器連進去也可以看到同樣的畫面. (偶而還被我拿來當 Share Screen, 和遠端的同事接力 Debug)

所以兩個結合起來真是太贊了!


[轉] Google 帶你了解20多年來地球的變化