Tuesday, 7 October 2014

linux tmpfs

tmpfs is a special file system used in Linux. Not like other traditional disk/partition based file systems, it store’s the file in memory. So it is much quicker than normal file system.
Then how can we use tmpfs? Well, we will just mount it as below:
#mount tmpfs –t tmpfs /var/tmp
Usually we can just add a size parameter to set the maximane size as
mount tmpfs -t tmpfs /var/tmp -o size=100m
we can do some test to write a 50M file there, then we can check the used memory increased, then we delete the file and the memory is freed
root@mysql:~# free -m
             total       used       free     shared    buffers     cached
Mem:          7985        290       7694          0         17        115
-/+ buffers/cache:        156       7828
Swap:         8187          0       8187
root@mysql:~# dd if=/dev/zero of=/var/tmp/zero.txt count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 0.198847 s, 257 MB/s
root@mysql:~# free -m
             total       used       free     shared    buffers     cached
Mem:          7985        339       7646         49         17        164
-/+ buffers/cache:        156       7828
Swap:         8187          0       8187
root@mysql:~# rm /var/tmp/zero.txt
root@mysql:~# free -m
             total       used       free     shared    buffers     cached
Mem:          7985        290       7694          0         17        115
-/+ buffers/cache:        157       7827

Swap:         8187          0       8187

Monday, 1 September 2014

how to filter the table status output from mysql

show table status is a handy command to get the table status, but you can't filter the fields you want as normal select statement. Here is the alternative way to get the output
SELECT table_name,Engine,Version,Row_format,table_rows,Avg_row_length,
  Data_length,Max_data_length,Index_length,Data_free,Auto_increment,
  Create_time,Update_time,Check_time,table_collation,Checksum,
  Create_options,table_comment
FROM information_schema.tables
WHERE Table_Schema='MyTableName';
you can replace 'MyTableName' with the table you want.

Saturday, 23 August 2014

The differnce between TCP and UDP

Notes to the blog:
http://www.javacodegeeks.com/2014/07/9-differences-between-tcp-and-udp-protocol-java-network-interview-question.html

This question what is the difference between TCP and UDP is often asked during the interviews. It is also important for sysadmin and program to understand for dairy work. Then we will explain what is the major differences we can see between the two protocols.

Connection-oriented vs Connectionless:

this is the essential differnce between TCP and UDP. The differnce also explains the other different points in the blog. when we say TCP is connection-oriented, it means the protocol itself controls and manages the connection session. It needs so-call 3-ways handshake and 4-ways termination. TCP also has many control packages such as SYN,ACK to for flow control. Meanwhile, UDP does not provide a 'connection' session for communication parties.

Reliability:

TCP is reliable as the application can assume TCP provides a reliable communication while UDP does not provide any  delivery guaruntee

Ordering:

TCP also guarantees order of message as it can get the sequence number from all the packages. while UDP does not care about the package sequence.

Data Boundary:

TCP does not preserve data boundary, UDP does.In TCP, data is sent as a byte stream, and no distinguishing indications are transmitted to signal message (segment) boundaries. On UDP, Packets are sent individually and are checked for integrity only if they arrived. Packets have definite boundaries which are honoured upon receipt, meaning a read operation at the receiver socket will yield an entire message as it was originally sent.

Speed:

In one word, TCP is slow and UDP is fast.

Heavy weight vs Light weight:

Because of the overhead mentioned above, Transmission control protocol is considered as heavy weight as compared to light weight UDP protocol. TCP header is longer than UDP headers.

Congestion or Flow control:

TCP does Flow Control. TCP requires three packets to set up a socket connection, before any user data can be sent. TCP handles reliability and congestion control. On the other hand, UDP does not have an option for flow control.

Different Applications:


because TCP and UDP provides different behaviour or characters, different application use either TCP or UDP as their transmission protocal. for example, HTTP and FTP, they use TCP while SIP, TFTP use UDP.

Wednesday, 20 August 2014

some tips about MySQL query Optimization

1. use the index: for every SQL, use the explain statement to verify the execution plan.
2. dont use index for computing substatement
3. fine the search critieria as much as possible
4. create hash index on fields for min/max/count
5. do manual calculation on some nequalities

Thursday, 7 August 2014

Python STL - OS

OS module.
os is a very important standard library (STL) in python for system admin,  sysadmin can use it to interact with OS.
Common os functions
#list the directory’s content
>os.listdir(“pathname”)
 #call shell script or external binary
>os.system(“commands”)

>>> to be continue 

Tuesday, 5 August 2014

very good summary about linux system administrator interview questions

this is I prepared for the interview for Linux admin/System Engineer Technical questions:

1)      how to install Digital Certificates on apache/nginx
2)      Apache MPM types
3)      Linux symlink vs hardlink
4)      Linux run level (RHEL/Ubuntu)
5)      Linux static library link vs dynamic library link
6)      How to install software from source code and the meaning of each steps
7)      Files in /boot and their function
8)      The content in /etc/grub.conf
9)      Linux booting procedure
10)   MYSQL database creation and deletion
11)   MySQL primary key, unique key, foreign key
12)   MySQL b+ tree and hash index
13)   MySQL index advantage and disadvantage, how to create keys
14)   Select statement, left join, right join, inner join
15)   How to shutdown a linux box
16)   DHCP requests procedure
17)   What is a proxy and reverse proxy
18)   ASymmetric and Symmetric encryption method
19)   How to use ssh key to login to the system without a password (what’s the advantage)
20)   Some remote login security rules
21)   How to execute shell commands in MySQL
22)   Name some options of MySQLadmin
23)   What GUI tools you have used for MySQL admin
24)   How to generate a service for Linux
25)   RAID 0, 1, 5
26)   Name the method to copy a file from a linux server to another server
27)   Rsync working mode. How to setup an rsync environment
28)   Linux – how to enter the single user mode
29)   How kill works in linux, name a few common Signals in LINUX
30)   Describe the memory layout of a procedure
31)   Ext2 special file permission and ACL
32)   HDD layout? How about SSD
33)   What is the function of VFS
34)   List common Linux environment variables
35)   How does CDN work
36)   How to forbid the root user to login directly
37)   How to grant/revoke permissions to MySQL user
38)   How to change mysql user’s password
39)   OSI mode and TCPIP mode
40)   What is VLAN and it’s function
41)   Common HTTP method
42)   Common HTTP response code
43)   How to setup MySQL replication
44)   What is file’s magic number
45)   Linux file types
46)   Linux super lock content
47)   Linux inode content
48)   How ping works
49)   How tracert works
50)   DNS server type, records type
51)   How DNS query works
52)   TCP establishment and termination method
53)   TCP flags
54)   What is TCP half close
55)   TCP 2MSL function
56)   List the TCP timers
57)   The format of a standard URL
58)   How can you design an Web Server
59)   Web Server’s keepalive advantage and disadvantage
60)   The responsibility of a system administrator
61)   The difference between procedure and thread
62)   Procedure life cycle
63)   The priority of the process, nice value and RT
64)   How to track down a process
65)   How does strace/ltrace work
66)   Linux cache/buffer
67)   /etc/passwd, /etc/shadow difference
68)   How to lock a user
69)   How to add a disk to linux system and use it in production environment
70)   How to create file system (for example ext4)
71)   Explain the output of top command and the meaning of each field
72)   The difference between router and switch, what is layer 3 switch
73)   how to make the OS secure
74)   myISAM vs InnoDB
75)   what is ACID
76)   what is the isolation level
77)   explain the backup and restore for MySQL
78)   how to create user in mysql
79)   mysql log explaination
80)   explain what is PKI
81)   SSL establishment procedure
82)   List the file system Linux supports
83)   List the linux disk blocks
84)   What is init process and it’s function
85)   Explain the disk block types in linux
86)   LVS working method and common scheduler
87)   Linux should at least contains which file partitions
88)   How to add modules in Apache
89)   How to integrate Apache and Tomcat
90)   What’s the function of firewall
91)   NTFS vs FAT
92)   Hsrp vs vrrp
93)   Mysqldump common options
94)   Mysqlbinlog
95)   Common redundant method for network
96)   Apache’s tuning
97)   Windows build-in group and users
98)   File system common commands
99)   Yum configuration file and repo.d sample
100) The process status and flag
101) The baseline for linux performance (4 sub system)
102) Linux performance checking tools
103) iptables tables and chains
104) collion domain and broadcast domain
105) NAT types
106) IP private sections
107) how OSPF is working
108) swap related commands. How to setup swap partitions on the fly
109) which software Apache rely on?

110) How Apache and Nginx configure a virtual host

Wednesday, 25 June 2014

ss (socket statistics) introduction

Almost every sys-admin know the powerful tool called ‘netstat’ but it is time to say bye-bye to netstat as we have a more powerful and concise tool called ‘ss’.

1. ss introduction

ss is the abbreviation for socket statistics. It is part of the iproute tool. If you can’t use ss, you need to install iproute such as
       yum install iproute iproute-doc
while netstat is part of net-tools which has not been updated since 2001
when we compare the performance between ss and netstat, we can get the conclusion that ss is much faster than netstat, that is because ss uses TCP’s tcp_diag to get the statistics from linux kernel.

See when a linux handle 10000 TCP connections, netstat took over 1 second to get the output and only 0.01 second to get the ss status

[root@X003 bin]# time netstat -ant > /dev/null

real    0m1.334s
user    0m0.230s
sys     0m1.101s
[root@X003 bin]# time ss -s
Total: 10439 (kernel 10442)
TCP:   11073 (estab 9203, closed 1, orphaned 800, synrecv 0, timewait 0/0), ports 10007

Transport Total     IP        IPv6
*         10442     -         -
RAW       0         0         0
UDP       8         5         3
TCP       11072     10008     1064
INET      11080     10013     1067
FRAG      0         0         0


real    0m0.014s
user    0m0.000s
sys     0m0.012s
ss 

2. Basic usage:

1. ss -s #Get basic statistics information

[root@X003 bin]# ss -s
Total: 180 (kernel 197)
TCP:   15 (estab 3, closed 1, orphaned 0, synrecv 0, timewait 0/0), ports 7

Transport Total     IP        IPv6
*         197       -         -
RAW       0         0         0
UDP       8         5         3
TCP       14        8         6
INET      22        13        9
FRAG      0         0         0
2. ss -l #only list the listening port

[root@X003 bin]# ss -l
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
LISTEN     0      128                    :::sunrpc                  :::*
LISTEN     0      128                     *:sunrpc                   *:*
LISTEN     0      128                    :::http                    :::*
LISTEN     0      128                    :::ssh                     :::*
LISTEN     0      128                     *:ssh                      *:*
LISTEN     0      128             127.0.0.1:ipp                      *:*
LISTEN     0      128                   ::1:ipp                     :::*
LISTEN     0      100                   ::1:smtp                    :::*
LISTEN     0      100             127.0.0.1:smtp                     *:*
LISTEN     0      128                    :::55878                   :::*
LISTEN     0      128                     *:38186                    *:*
3. ss -t #only list the tcp status

[root@X003 bin]# ss -t
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418
4. ss -u #only list the udp status
5. ss -p #list the program associated with port

[root@X003 bin]# ss -p
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433    users:(("sshd",1789,3))
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417    users:(("sshd",1734,3))
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418    users:(("sshd",1759,3))
6. ss -n #don’t convert the number to the hostname or known service name

[root@X003 bin]# ss -n
State      Recv-Q Send-Q        Local Address:Port          Peer Address:Port
ESTAB      0      52           192.168.56.113:22            192.168.56.1:65433
ESTAB      0      0            192.168.56.113:22            192.168.56.1:65417
ESTAB      0      0            192.168.56.113:22            192.168.56.1:65418
7. ss -e #show the extensive/detailed information.

[root@X003 bin]# ss -e
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433    timer:(on,354ms,0) ino:74064 sk:ffff88002c8d5440
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417    timer:(keepalive,76min,0) ino:73782 sk:ffff8800299e4d80
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418    timer:(keepalive,76min,0) ino:73915 sk:ffff8800299e4700

Tuesday, 24 June 2014

Powerful DNS query tool - DIG

When we talk about the DNS query tool, we usually refer to nslookup. Now in this blog, we are going to talk about a more powerful tool called dig.

1.dig. Command without any parameters

root@ip-172-31-7-84:~# dig

; <<>> DiG 9.9.5-3-Ubuntu <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25338
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;.                              IN      NS

;; ANSWER SECTION:
.                       19664   IN      NS      d.root-servers.net.
.                       19664   IN      NS      e.root-servers.net.
.                       19664   IN      NS      f.root-servers.net.
.                       19664   IN      NS      g.root-servers.net.
.                       19664   IN      NS      h.root-servers.net.
.                       19664   IN      NS      i.root-servers.net.
.                       19664   IN      NS      j.root-servers.net.
.                       19664   IN      NS      k.root-servers.net.
.                       19664   IN      NS      l.root-servers.net.
.                       19664   IN      NS      m.root-servers.net.
.                       19664   IN      NS      a.root-servers.net.
.                       19664   IN      NS      b.root-servers.net.
.                       19664   IN      NS      c.root-servers.net.

;; Query time: 6 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Jun 25 15:27:35 EST 2014
;; MSG SIZE  rcvd: 239

2.basic command: dig @DNS domain-name record-type

root@ip-172-31-7-84:~# dig @8.8.8.8 rms.nsw.gov.au AAAA

; <<>> DiG 9.9.5-3-Ubuntu <<>> @8.8.8.8 rms.nsw.gov.au AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23205
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;rms.nsw.gov.au.                        IN      AAAA

;; AUTHORITY SECTION:
rms.nsw.gov.au.         1799    IN      SOA     dns2.rta.nsw.gov.au. root.dns2.rta.nsw.gov.au. 2013120601 7200 3600 3600000 7200

;; Query time: 289 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Jun 25 15:29:14 EST 2014
;; MSG SIZE  rcvd: 93

3.common options: Here is the most common options we will need to keep in mind.
-t [type]: by default, it is A record(address) but you can set MX for query

root@ip-172-31-7-84:~# dig -t MX rms.nsw.gov.au

; <<>> DiG 9.9.5-3-Ubuntu <<>> -t MX rms.nsw.gov.au
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41964
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;rms.nsw.gov.au.                        IN      MX

;; ANSWER SECTION:
rms.nsw.gov.au.         7200    IN      MX      30 mx22out.rta.nsw.gov.au.
rms.nsw.gov.au.         7200    IN      MX      40 waterways-b1.maritime.nsw.gov.au.
rms.nsw.gov.au.         7200    IN      MX      10 cluster4.us.messagelabs.com.
rms.nsw.gov.au.         7200    IN      MX      20 cluster4a.us.messagelabs.com.
rms.nsw.gov.au.         7200    IN      MX      30 mx11out.rta.nsw.gov.au.
rms.nsw.gov.au.         7200    IN      MX      30 mx12out.rta.nsw.gov.au.
rms.nsw.gov.au.         7200    IN      MX      30 mx21out.rta.nsw.gov.au.

;; Query time: 14 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Jun 25 15:31:10 EST 2014
;; MSG SIZE  rcvd: 250
-q [domain]: -q can be omitted, but it makes more clear if you add the –q option.

root@ip-172-31-7-84:~# dig -q www.google.com

; <<>> DiG 9.9.5-3-Ubuntu <<>> -q www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60498
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         166     IN      A       74.125.237.180
www.google.com.         166     IN      A       74.125.237.176
www.google.com.         166     IN      A       74.125.237.177
www.google.com.         166     IN      A       74.125.237.178
www.google.com.         166     IN      A       74.125.237.179

;; Query time: 6 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Jun 25 15:33:07 EST 2014
;; MSG SIZE  rcvd: 123
-x [IP]: reverse query. Get the hostname for the IP.

root@ip-172-31-7-84:~# dig -x 193.0.14.129

; <<>> DiG 9.9.5-3-Ubuntu <<>> -x 193.0.14.129
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19459
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;129.14.0.193.in-addr.arpa.     IN      PTR

;; ANSWER SECTION:
129.14.0.193.in-addr.arpa. 21600 IN     PTR     k.root-servers.net.

;; Query time: 21 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Jun 25 15:34:54 EST 2014
;; MSG SIZE  rcvd: 86
+tcp: use tcp for query.

root@ip-172-31-7-84:~# dig +tcp www.myrta.com

; <<>> DiG 9.9.5-3-Ubuntu <<>> +tcp www.myrta.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27172
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.myrta.com.                 IN      A

;; ANSWER SECTION:
www.myrta.com.          30      IN      A       163.189.7.175

;; Query time: 43 msec
;; SERVER: 172.31.0.2#53(172.31.0.2)
;; WHEN: Wed Jun 25 15:38:26 EST 2014
;; MSG SIZE  rcvd: 58

+trace: shows the trace path of the recursive path

root@ip-172-31-7-84:~# dig +trace www.myetoll.com
; <<>> DiG 9.9.5-3-Ubuntu <<>> +trace www.myetoll.com
;; global options: +cmd
.                       18934   IN      NS      l.root-servers.net.
.                       18934   IN      NS      m.root-servers.net.
.                       18934   IN      NS      a.root-servers.net.
.                       18934   IN      NS      b.root-servers.net.
.                       18934   IN      NS      c.root-servers.net.
.                       18934   IN      NS      d.root-servers.net.
.                       18934   IN      NS      e.root-servers.net.
.                       18934   IN      NS      f.root-servers.net.
.                       18934   IN      NS      g.root-servers.net.
.                       18934   IN      NS      h.root-servers.net.
.                       18934   IN      NS      i.root-servers.net.
.                       18934   IN      NS      j.root-servers.net.
.                       18934   IN      NS      k.root-servers.net.
;; Received 239 bytes from 172.31.0.2#53(172.31.0.2) in 14 ms

com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20140701000000 20140623230000 40926 . lDy4cb2Mb6tyMYXfCNX1bcEwE2Rg6OkcsuSv+R81yFHEG5/luQvypHHr nt6apJYVf30t748Dtu5X3H56IXXlgftqgY93AbuAjugAz3gf8YiQtRW5 wQcKjtzq2luW1YIVYtujntwfC4om+sVxtCJbKT88cqQUwyTvKr8D6SxS QtA=
;; Received 739 bytes from 192.36.148.17#53(i.root-servers.net) in 237 ms

myetoll.com.            172800  IN      NS      dns1.rta.nsw.gov.au.
myetoll.com.            172800  IN      NS      dns2.rta.nsw.gov.au.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0QFMDQRCSRU0651QLVA1JQB21IF7UR NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20140630044832 20140623033832 56657 com. Co2ZGs6Fo2h2zlCv4wQS4E28nhjNxNQ/InfoiMq52MUH4/IJ9rh1Ot4C vuEmO3J1U4VVYUedu1EyFtBe6hGpajEfsntCYFA2EFbi2PgzAHAw8Ljr NCC24OnMBfcpbM6ew5UFd/157h6Tv/a+ilystE7+goWZi7q6Y6VNPWle B60=
RFJREQ8DI8OGS8V03LLBHDLV4J3EIPHK.com. 86400 IN NSEC3 1 1 0 - RFJVBCOQJUN2E7A6HEMG5MM79DAEKVMV NS DS RRSIG
RFJREQ8DI8OGS8V03LLBHDLV4J3EIPHK.com. 86400 IN RRSIG NSEC3 8 2 86400 20140701043512 20140624032512 56657 com. McX9ZVW1HVTK4ZkKSOOAj9pkHb8VzqKIbRNdcMWN8OJ75GaL/9II1bNx pXodneZpd+qtRe3TVhweJlv30zXEritWsN507cQDDsDoIgu+aNpgCzO7 louvilFNU88pOPPx4Wk1oTS9BNpNV/rcw0GWF/Yb/Y2Q/XTiyNl5NLe0 LiM=
;; Received 581 bytes from 192.12.94.30#53(e.gtld-servers.net) in 376 ms

www.myetoll.com.        7200    IN      NS      dns22.rta.nsw.gov.au.
www.myetoll.com.        7200    IN      NS      dns11.rta.nsw.gov.au.
;; Received 98 bytes from 163.189.23.22#53(dns2.rta.nsw.gov.au) in 26 ms

;; Received 12 bytes from 163.189.217.1#53(dns22.rta.nsw.gov.au) in 9 ms
+short: only show the result of the query.

root@ip-172-31-7-84:~# dig +short www.rms.nsw.gov.au
163.189.7.150

Monday, 16 June 2014

MySQL replication introduction

Database plays the essential role in the information system such as websites. To avoid the SPF of MySQL, in production system, we use master-slave replication to synchronise the data and MySQL-proxy to enhance the IO performance.

Advantages of MySQL replication

  • Avoid of SPF, when the master server is down, we can use slave server to provide the service
  • We can use master server to handle the write request and slaves to handle the read IO.

 Here is how MySQL replication works

  • Master server write the data changes to Binary log
  • Slave servers IO Thread read from master binary log and put to Relay log.
  • Slave server SQL thread read the changes from relay log and implement the change to Slave database.

 

Test Environment:

1. Master server 192.168.139.100
2. Slave server 192.168.139.200


To simplify the illustration,  we suppose both master and slave are from scratch without any data.

1. Set the binary log and server-id:
Master  - my.cnf
[mysqld]
log-bin=mysql-bin            //the binary log must be enabled
server-id=100                    //unique ID in MySQL cluster
Slave – my.cnf
[mysqld]
log-bin=mysql-bin            //the binary log must be enabled
server-id=200                    //unique ID in MySQL cluste


system restart may be required if the parameter changed.

2. Create the user dedicated for replication.
It is not neccessary but strongly recommended that a specified user will be created for replication
create user 'repl'@'192.168.139.200' identified by 'mysql';
grant replication slave on *.* to repl@'192.168.139.200' identified by
'mysql';

3. Get master status in master server

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 |      120 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

4. In the slave server, set the master server and synchronise parameters

mysql>change master to
master_host='192.168.139.100',
master_user='repl',
master_password='mysql',
master_log_file='mysql-bin.000001',
master_log_pos=120;

5. Start the mysql slave
6. Check the status

mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.139.100
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 120
Relay_Log_File: X002-relay-bin.000001
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
------ omitted ------


Now we can test by create database/table and insert data into the master server and confirm we can get in the slave server.

Sunday, 15 June 2014

Windows common command tool

nslookup - common DNS query tool
logoff - log off the user session
lusrmgr.msc - manage local users and groups
services.msc - manage services
notepad - open notepad
net start [service] - start a service
net stop  [service] - stop a service
compmgmt.msc - local computer management tool
devmgmt.msc - device management tool
regedt32/regeit.exe - register editor
mem.exe - view memory tool
wimmsd - view system information
mstsc - open remote desktop session
diskmgmt.msc - disk management tool
ntbackup - build-in backup tool
taskmgr - task manager
eventvwr - event viewer
calc - calculator
certmgr.msc - Certificates management tool
iisreset - IIS stop/start tool
secpol.msc - local security policy management tool
netstat - common netstat tool
ping - common ping tool
tracert - common traceroute tool
winver - check the windows version

Wednesday, 11 June 2014

Using Python boto to store the content to AWS S3



Recently I have a requirement to store some plain text files to AWS S3. As my primary program is Python, I used boto to fulfil the task. Here are the basic steps

Install boto package into python environment.

If the environment has the internet access ability, boto can be easily setup by ‘easy_install’. Otherwise you have to download the source file from github and manually install it.


easy_install boto
git clone https://github.com/boto/boto

Setup key configuration

When accessing S3, you need to use authentication (security keys, X509 certificates etc), in python/boto, security keys is the easiest way (maybe not the most applicable way), security keys is a pair of key_id and access_key generated by AWS to identify the user. See more about IAM.

Usually when you generated the IAM user, AWS allows you to download the key csv files to your local machine. There are two ways using keys in Python/boto. One is code it into your program as
aws_access_key = “aaaaaaaaa”
aws_secret_key=”sssssssssssssssss”
from boto.s3.connection import S3Connection
conn = connect_s3 (aws_access_key, aws_secret_key)
or you put your security key into /etc/boto and use connect_s3 without any parameters.
from boto.s3.connection import S3Connection
conn = connect_s3 ()
I used the second way as it is more flexible (easily to change the user) and more secure  (you can specify the file permission)
[Credentials]

aws_access_key_id = *******************
aws_secret_access_key = ********************

Program flow:

import boto
bucketname = ‘rafaxubucket’                    #here is the bucket
filename = ‘recports/myfile’                       #here is the filename including path
conn = boto.connect_s3()
b = conn.get_bucket(bucketname)
key = b.new_key(filename)
key.set_contents_from_string(reportContent)
key.set_canned_acl('public-read')

Explaination:                

To store the file, we need to get the bucket, then the filename is called key in S3, the key can contain ‘/’ which means the path actually. Then you can dump your content to the key (write the content to the file as compared in the file IO) then set the acl. After that you can view the content via browser as the content is already on S3 as a web service.

Tuesday, 10 June 2014

Advanced TCP topics - 0

Recently I read two interesting blogs from a Chinese engineer
http://coolshell.cn/articles/11564.html
http://coolshell.cn/articles/11609.html

He introduced some of the TCP advanced topics as:

TCP advanced technology:

1. TCP package format.
Key fields.
Sequence Number,
Acknowledgement Number,
Window,
TCP Flag

2. TCP status transition
    3-wayss handshake, 4-ways termination

3. Sequence number: how the sequence number is generated

4. TCP retransmission (Fast retransmit, SACK, etc)

5. TCP RTT (KP, JK)

6. TCP Sliding Window

7. TCP congestion handling (slow start, fast recovery etc )

I will write 7 blogs as the study notes to the articles in this week.

Monday, 9 June 2014

MySQL show commands

MySQL command prompt provides lots of show commands to view the database objects and status. Here is a short summary of show commands:


show tables or show tables from database_name
Explaination: show all tables in current database
show databases;
Explaination: show databases in current server
show processlist;
Explaination: show all running processes associated with the user, if the user has been granted with process privilege, he can check all processes in the server.
show table status;
Explanation: show table detailed status.
show columns from table_name from database_name; 
show columns from database_name.table_name;
Explanation: show table columns = desc [tablename]
show grants for user_name@localhost;
Explanation: show the users privileges;
show index from table_name;
Explanation: show index for the table;
show status;
Explanation: show database very detailed status such as threads numbers, lock numbers, uptime, etc. 
show variables;
Explanation: show variables and the values
show privileges;
Explanation: show different privileges database supports. 
show create database database_name;
Explanation: show create database command syntax
show create table table_name;
Explanation: show create table command syntax 
show engies;
Explanation: show available data engines and the default one.
show innodb status;
Explanation: show innoDB store engine status
show logs;
Explanation: show BDB store engine log status
show warnings;
Explanation: show the last commands warning messages
show errors;
Explanation: show the last commands error messages