Linux filesystem Introduction

In general linux file system has 3 parts

• Superblock: it records the meta information for the linux filesystem. It contains the inode/iblock, amount, usage, free capacity and other file system information. The superblock information can be viewed by tune2fs
• Inode. Record the file attributes. Every file will have a inode
• Block: record the file content

Every file will use one inode and some blocks, the data will be allocated to the first block of the file, if it is over the block, then it will use the second block. Block can’t be shared between files

Inode contains the below information (most of information can be seen by stat file)

1.       permission

2.       ownership

3.       file size

4.       ctime, atime, mtime

5.       ACL

6.       file pointer

inode 3-layer index The inode use the 3 level table to contain the inodes mapping

how to set the initial password for MYSQL

After installed MySQL in your machine, you may meet the below error when you first login

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

This is because the password hasn’t been set for the root user

To resolve the problem.

1.       stop mysql server

service mysql stop

2.       start mysql using safe_mode

/usr/bin/mysqld_safe --user=mysql --skip-grant-tables --skip-networking &

3.       login to mysql without password

mysql -u root

4.       update mysql user password

Update user SET Password=PASSWORD('111111') where USER='root';

5.       start mysql as normal and enjoy your mysql trip

Django Web Design Step by Step – install and setup environment

the demo source code can be downloaded in github.
git@github.com:aaaaatoz/djangoweb.git

Download and Install Django

Django is the most popular Python-based web framework (https://www.djangoproject.com/), in this serial blogs, we are going to setup a website using Django step by step

Django can be downloaded from the official website, the current version is 1.6 but in the blog, we will use 1.5.5 for demonstration.

When Django is downloaded, unzip the file and you will get the files as below:

Then we will use the traditional python command to install the django infrastructure

#python setup.py install

After a few mins, you will get the Django framework installed. To test if it is ready just type

[root@hadoop1 Django-1.5.5]# which django-admin.py

/usr/bin/django-admin.py

Setup a Project

Setup a Project and Application for your website

In Django 1.5, project is the framework and application is the web container. We will navigate to a proper location and create a project as

#django-admin.py startproject djangoweb

You will find a directory called djangoweb has been created. In side the directory, there is a directory with the same name (djangoweb) and a manage.py file

Setup an Application

Now, make sure you are in the djangoweb (project not the subdirectory) directory and create an application called mysite by the below commands:

#django-admin.py startapp mysite

You will see a dictory called mysite has been created. Now You djangoweb project should look like:

Run Django Web Framework

We will explain the files and their function in the next a few blogs but now let’s enjoy Django by the command

#python manage.py runserver 0.0.0.0:8080

There should be no error and the web server is ready to serve the request.

Try using web browser to access it and you will see an Django Welcome page.

Apache configuration – virtual host

Virtual host is widely used in web servers. Probably most of the public facing web servers use the technology. It provides the ability to host multiple hosts in a single Apache HTTP (or any other web servers) instance.

There are two types of virtual host:

• IP-based virtual host: use different IP address to provides different content to the user
• Name-based virtual host: use different host name to provide different content to the user

Obviously, name-based virtual host is much more important as IP address is limited and involves lots of infrastructure configuration.

Name-based virtual host

Name-based virtualhost is based on a field called HOST in http request.

Here is part of my http request to the website
 

Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control:max-age=0
Connection:keep-alive
Host:fujtest:90
If-Modified-Since:Fri, 23 Sep 2011 00:52:33 GMT

The web server then severs the request based on the HOST field.

It is strongly recommended to use different configuration file rather than the main httpd.conf to enable the virtual host setting for a large website. It will make the main section httpd.conf more concise and readable.

Here is a very basic configuration for the name-based virtual host configuration.

In main httpd.conf

Include conf/extra/httpd-vhosts.conf    # to include the virtual host configuration.

In the conf/extra/httpd-vhosts.conf, we first need to enable name virtual host function by

NameVirtualHost [FQDN or IP]:[PORT]
Then define the virtualhost section
<VirtualHost ip:port>
       ServerName
       DocumentRoot
       etc
</VirtualHost>

Here is a simple sample configuration.

NameVirtualHost 192.168.179.150:90
<VirtualHost 192.168.179.150:90>
     DocumentRoot "/usr/local/apache2/htdocs/fujtest1"
     ServerName fujtest1
     ErrorLog "logs/fujtest1_error.log"
     CustomLog "logs/fujtest1_access.log" common
</VirtualHost>

<VirtualHost 192.168.179.150:90>
    DocumentRoot "/usr/local/apache2/htdocs/fujtest2"
    ServerName fujtest2
    ErrorLog "logs/fujtest2_error.log"
    CustomLog "logs/fujtest2_access.log" common
</VirtualHost>

You may need to check the permission options if virtual host is not in your main document ROOT section

Apache HTTP server configuration – Core Part

For the Apache HTTP configuration, usually there are 3 parts. The core part, the container part and the extension part. In this blog, we are going to talk about the core part briefly.

The core part is for the Apache main modules, it applies for the whole Apache configuration.  There are some common configuration items as below:

ServerName: specify the default global server name for the http server.  You may use FQDN or ip for this section.

Sample config: ServerName www.example.com:80         #specify my host is www.example.com

Listen: specify the listening port for the http server. By default, you may define it as 80 but it can be any other port.

Sample config: Listen 80 #specify the http server is listen on port 80

ServerRoot: it is used to define the server’s root. Usually it is defined when the source files are configured by --prefix=[location].  If it is a binary installation. Usually use /etc/apache2.

Sample:  ServerRoot "/usr/local/apache2" #specify my root path is /usr/local/apache2

DocumentRoot: it is used to define the default html root location. It could be overwritten in the virtual host settings.

Sample: DocumentRoot "/usr/local/apache2/htdocs" #specify my root path is /usr/local/apache2

ServerAdmin: used to specify an WebAdmin’s email address . when there is a problem, you may send the email to report the issue.

Sample: ServerAdmin rafa.xu.au@gmail.com

ScriptAlias and Alias: as used to map the url to a specify directory in the machine. ScriptAlias will let the server recoginse the files in the directory as cgi scripts while Alias is only as normal directory.

Sample:

    ScriptAlias /cgi-bin/ "/usr/local/apache2/cgi-bin/"

    Alias /alias/ "/var/tmp/alias/"

User and Group: used to define the ownership for subprocesses. For security reason’s the user and group should be dedicated for httpd processes as an non-login account.

User apache

Group apache

Loadmodule: the command is used to load the modules dynamically.

Sample: LoadModule mime_module modules/mod_mime.so     #it will load mime_module into Apache.

ErrorDocument:   specify the error code with friendly page or scripts

Sample: ErrorDocument 404 /404.html #make a specify 404 page for the user.

Python Function – lambda

Lambda is a single line anonymous function used in python.  The format is like below:

lambda parameters: expression

sample code

>>> (lambda x: x*x)(3)

9

paramters are where you put your parameter list and expression is the function expression you want to return.

Some points:

• You don’t need to use () for the parameter list.
• No explicit return statement
• Lambda expression can be used as a function by (lambda) (parameters)

Here is another example

>>> Condition = True

>>> ObjectString = "my_email_address gmail.com"

>>> processFunc = Condition and (lambda s: "@".join(s.split())) or (lambda s: s)

>>> processFunc(ObjectString)

'my_email_address@gmail.com'

processFunc is actually a function with a Condition.

 

In real word, lambda is widely used with the build-in functions such as map, filter and reduce.  

Get the square of each element in the list

mylist = [1,3,5,7,9,0]
map( lambda x: x*x , mylist)  
or it can be expressed in list comprehension
[ x*x for x in mylist]
 

Get all the element if the element is bigger than 0

mylist = [1,-2,3,-4,5,-6,7,-8]
Filter((lambda x: x> 0), mylist)

Dictionary sort

sorted(dict.items(), key=lambda e:e[1], reverse=True) – sort key by value
sorted(dict.items(), key=lambda e:e[0], reverse=True) – sort key by key

integrate Django with Ngnix

In production system, it is very common that we will use Nginx as the reverse proxy and Python/Django as the backend application server.

here is the simple sample Nginx setting to integrate Django.

The Nginx's HTTP section

http {
     include mime.types;
     default_type application/octet-stream;

     sendfile on;

     keepalive_timeout 65;

     upstream djangoserver {
         server 127.0.0.1:8080;
     }

    server {
    listen 80;
    server_name localhost;

    location / {
       include fastcgi.conf;
       fastcgi_pass djangoserver;
    }
}

• define upstream djangoserver, it can be a group of server for load balance. IP and port should be matched with the Django Application server
• include the fastcgi.conf in the server/location sections.
• pass the request to the application server : fastcgi_pass djangoserver;

we can use default fastcgi setting if no performance consideration but we need to change two lines. otherwise you will get blank urls when passing to Django.




fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;


#fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param HTTPS $https if_not_empty;

fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;

on the Django side, you will start the django with runfcgi option. sample start script as below:
python manage.py runfcgi method=prefork host=0.0.0.0 port=8080