File Magic Number

In many OSes, different type of files have different starting bytes. The typical starting numbers are called magic number. Usually the command ‘file’ checks the magic number to decide what’s the file type

Examples of ‘file’ command:

[ec2-user@ip-172-31-16-100 webchecker]$ file dnstest.py
dnstest.py: a /usr/bin/python script text executable
[ec2-user@ip-172-31-16-100 webchecker]$ file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
[ec2-user@ip-172-31-16-100 webchecker]$ file hostname.conf
hostname.conf: ASCII text

when we write the script, either with python, perl or bash. We actually set the magic number for the file. The script may start with a "shebang" (#!, 23 21) followed by the path to an interpreter,

we may use readelf command to get the magic number

[ec2-user@ip-172-31-16-100 webchecker]$ readelf -h /usr/bin/python
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x400620
  Start of program headers:          64 (bytes into file)
  Start of section headers:          7048 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         31

  Section header string table index: 30
  

Git Commit Golden Rules

commit is an essential activity in Git usage. here are some important rules for commit

only commit one thing in a single commit

don’t mix things together in one single commit. Always commit just one function, one bug-fix for the issue.

write good commit message

commit message is like the comment to the program. Make sure you commit the change with a suitable commit message for further reference.

commit a completed work.

Never commit something that is half-done. If you need to save your current work temporarily in something like a clipboard, you can use Git's "Stash" feature

only commit tested work

Related to the point above, you shouldn't commit code that you think is working. Test it well - and before you commit it to the repository.

Amazon AWS introduction – EBS

EBS (Elastic block store) is like a virtual hard drive you can use with your Amazon EC2 instances.

The advantage of using EBS is:

•  Keep you data separated from your computer instances
•  Can be attached to your instances only when needed
•  When EC2 instances failed, the data on the EBS won’t be lost

The EBS varies from 1G to 1T, it is created on an Amazon AZ (Availability Zone), replicated over AZ to prevent data lost. One EBS can only be attached to one EC2 instance at a time. It is very similar to a hard drive and any other storage device. In practice, you can add multiple EBS to EC2 instance that can stripe IO and increase READ/WRITE performance especially for data-intensive applications such as database.

How to use EBS

In AWS console, go to EC2 Dashboard – ELASTIC BLOCK STORE – Volumes.

Click “Create Volume” and assign the size of storage you want., the AZ and type.

The you will need to attach the EBS volume to the EC2 instance by right click the EBS and select attach volume.

Then you can create file system using fdisk and LVM tools as it is a normal local disk or SAN disk

Advanced linux keyborad Skills

Sysadmins use terminal or virtual terminal to control the Linux Systems. It is very usually for the system admin to master some terminal skills.

Ctrl-a	Move the cursor to the beginning of the line
Ctrl-e	Move the cursor to the end of the line
Ctrl-f	Move forward the cursor one character ahead
Ctrl-b	Move backward the cursor one character behind
Alt-f	Move forward the cursor one word ahead
Alt-b	Move backward the cursor one word behind
Ctrl-d	Clear the screen and keep the command
Alt-l	Change all the letters lower case from cursor to end of the word
Alt-u	Change all the letters upper case from cursor to end of the word
Ctrl-k	cut the command line from cursor to end of the line
Ctrl-u	cut the command line from cursor to beginning of the line
Alt-d	Cut the word from cursor to end of word
Alt-Backspace	Cut the word from cursor to beginning of word
Ctrl-y	Paste
Alt-?	Same as [tab][tab], list all possible command
history command	List all the history command
Ctrl-r	Search in the history command
!!	Execute the last command
!number	Execute the number command

Practice is the most efficient way to master the commands

Program compile and link in linux.

Program compile and link in linux.

Most of system software provides the source code and we can use compile to build it into the executable binaries. This blog will introduce what is inside the compilation process.

Compile tools

On linux platform, gcc is the most wide used tool for c, c++ and even java compiler.

You will use

#which gcc
to determine if gcc has been installed

Single file compilation

here is a single c program:

#include <stdio.h> 
void main(void) 
{ 
  printf("hello world!"); 
} 

now we can compile the program and run it

[root@X001 cprogram]# ls
hello.c
[root@X001 cprogram]# gcc hello.c
[root@X001 cprogram]# ls
a.out  hello.c
[root@X001 cprogram]# ./a.out
hello world!

or you may use 'gcc -o outputfile sourcefile' to assign an output executable file name

multiple files compilation

if we have more than one single source file, which is the most case in application development. source code A will refer to other files, when one of the files is changed,  do we need to re-compile all of the related files?
the answer is no.
we have two files

File:a.c 
#include <stdio.h> 
int main () 
{ 
    printf("this is from first file\n");
    method(); 
} 

File:b.c 
#include <stdio.h> 
void method(void) 
{ 
    printf("this is from second file:\n"); 
}

now we will do
1. compile the individual source files into object files
2. link the object files into binary

[root@X001 cprogram]# ll
total 8
-rw-r--r--. 1 root root 95 Apr 14 14:11 a.c
-rw-r--r--. 1 root root 87 Apr 14 14:05 b.c
======compile the files here ===========
[root@X001 cprogram]# gcc -c a.c b.c
[root@X001 cprogram]# ll
total 16
-rw-r--r--. 1 root root   95 Apr 14 14:11 a.c
-rw-r--r--. 1 root root 1568 Apr 14 14:11 a.o
-rw-r--r--. 1 root root   87 Apr 14 14:05 b.c
-rw-r--r--. 1 root root 1504 Apr 14 14:11 b.o
=======link the files here ==============
[root@X001 cprogram]# gcc -o result a.o b.o

[root@X001 cprogram]# ./result
this is from first file
this is from second file

if we change the print statement of the b.c to "this is the updated file" .
we only need re-compile the b.o file and then regenerate the executable binary. 

link library

none of the commercial software is developed from scratch. We will use some of common software component called library.  for example, you may use mathmatics library to get the sin value for PI. 
 

Linux file types

Linux supports multiple file types such as normal files, directory, socket files etc. we can use command ‘ll’ or ‘li –al ’ to list the file and check the file type by the first character of the output.

In general, linux system has the below file types:

-     normal files
d     directory file
l     link (symlink) file
s     socket file
p     pipe file
b     block file
c     character file

normal file. (marked as ‘-’)

Most of the files in linux system are normal files, it includes normal text file, library files, zip files, executable binaries

Directory: (marked as ‘d’)

Directory is a special kind of the file, it contains can contain other kind of files including directories.

Block device file(marked as ‘b’)

Block files are in /dev/ directory, they are some kind of presuedo files. The file can be visited randomly such as disk, usd stick.

Character device file (marked as ‘c’)

Character files are in /dev/ directory as well, the file has to be visited by sequence. It contains device such as mouse, keyboard, serial ports, console tty.

Both character and block files are called device file.

Socket file (marked as ‘s’)

Socket file is used for process communication. Note, it is not the network socket file, it is the unix socket file.

Link file (marked as ‘l’)

The file is linked file (symlink) and created by ‘ln -s’ command. 

Linux filesystem Introduction

In general linux file system has 3 parts

• Superblock: it records the meta information for the linux filesystem. It contains the inode/iblock, amount, usage, free capacity and other file system information. The superblock information can be viewed by tune2fs
• Inode. Record the file attributes. Every file will have a inode
• Block: record the file content

Every file will use one inode and some blocks, the data will be allocated to the first block of the file, if it is over the block, then it will use the second block. Block can’t be shared between files

Inode contains the below information (most of information can be seen by stat file)

1.       permission

2.       ownership

3.       file size

4.       ctime, atime, mtime

5.       ACL

6.       file pointer

inode 3-layer index The inode use the 3 level table to contain the inodes mapping