Python Function – lambda

Lambda is a single line anonymous function used in python.  The format is like below:

lambda parameters: expression

sample code

>>> (lambda x: x*x)(3)

9

paramters are where you put your parameter list and expression is the function expression you want to return.

Some points:

• You don’t need to use () for the parameter list.
• No explicit return statement
• Lambda expression can be used as a function by (lambda) (parameters)

Here is another example

>>> Condition = True

>>> ObjectString = "my_email_address gmail.com"

>>> processFunc = Condition and (lambda s: "@".join(s.split())) or (lambda s: s)

>>> processFunc(ObjectString)

'my_email_address@gmail.com'

processFunc is actually a function with a Condition.

 

In real word, lambda is widely used with the build-in functions such as map, filter and reduce.  

Get the square of each element in the list

mylist = [1,3,5,7,9,0]
map( lambda x: x*x , mylist)  
or it can be expressed in list comprehension
[ x*x for x in mylist]
 

Get all the element if the element is bigger than 0

mylist = [1,-2,3,-4,5,-6,7,-8]
Filter((lambda x: x> 0), mylist)

Dictionary sort

sorted(dict.items(), key=lambda e:e[1], reverse=True) – sort key by value
sorted(dict.items(), key=lambda e:e[0], reverse=True) – sort key by key

integrate Django with Ngnix

In production system, it is very common that we will use Nginx as the reverse proxy and Python/Django as the backend application server.

here is the simple sample Nginx setting to integrate Django.

The Nginx's HTTP section

http {
     include mime.types;
     default_type application/octet-stream;

     sendfile on;

     keepalive_timeout 65;

     upstream djangoserver {
         server 127.0.0.1:8080;
     }

    server {
    listen 80;
    server_name localhost;

    location / {
       include fastcgi.conf;
       fastcgi_pass djangoserver;
    }
}

• define upstream djangoserver, it can be a group of server for load balance. IP and port should be matched with the Django Application server
• include the fastcgi.conf in the server/location sections.
• pass the request to the application server : fastcgi_pass djangoserver;

we can use default fastcgi setting if no performance consideration but we need to change two lines. otherwise you will get blank urls when passing to Django.




fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;


#fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param HTTPS $https if_not_empty;

fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;

fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;

on the Django side, you will start the django with runfcgi option. sample start script as below:
python manage.py runfcgi method=prefork host=0.0.0.0 port=8080

Regular Expression

this is the study note to
http://qntm.org/files/re/re.html

1.       literal characters, which just represent themselves. Such as ‘c’, ‘ab’

2.       the dot  --- any single character

3.       Any meta character can be escaped using a backslash, \. This turns it back into a literal.

4.       A character class is a collection of characters in square brackets. This means, "find any one of these characters". eg [0-9] means any digital, [a-z] means any lower case characters. [^a] means any characters except a

5.       Special means

a.       \d – digital           \D – non-digital

b.      \w - [0-9A-Za-z_]: find a word character.  \W – non-word character

c.       \s find a space(tab, space, carriage return or line feed) character. \S non-space character.

6.       Multipliers use braces to put a multiplier after a literal or a character class.

a{3} means "find an a followed by an a followed by an a".

a{3,5} means "find aaaaa or aaaa or aaa".

? means none or once

* means any times (none, once, or more than once)

+ means one or more than once

Alternation with group: alternation is devided by |, eg: ‘cat|dog’ means cat or dog. You may use () to put your matching pattern there for example: (Mon|Tues|Wednes|Thurs|Fri|Satur|Sun)day means the day’s

\b means the word boundary

^ and $ from "start-of-line" and "end-of-line"

Advanced topics about Regular Expression

Capturing : used referring for () group. (\w+) had a (\w+)  (\w+)

And we can use back-references to refer to the captured pattern such as regular expression ([abc])\1 means "find aa or bb or cc". ([abc])\1\1 means "find aaa or bbb or ccc"

 

MySQL data files

 

The MySQL database stores the data in disks as other databases.  The default datastore will be determined by datadir variable

>show variables like "%datadir%";

For example my output is /var/lib/mysql/

Check the directories in

mysql> system ls -ld /var/lib/mysql/*/

drwx------. 2 mysql mysql 4096 Mar 21 22:09 /var/lib/mysql/mydatabase/

drwx--x--x. 2 mysql mysql 4096 Dec 29 10:33 /var/lib/mysql/mysql/

drwx------. 2 mysql mysql 4096 Dec 29 10:33 /var/lib/mysql/performance_schema/

drwx------. 2 mysql mysql 4096 Jan  1 21:08 /var/lib/mysql/test/

at the moment we got 5 databases on my databases(note file information_schema does not have a real data file storage)

mysql> show databases;

| information_schema      |

| mydatabase                    |

| mysql                             |

| performance_schema    |

| test                                 |

data file in  MyISAM

When I create a table using MyISAM engine.

mysql> create table mytable(id int ,name varchar(20)) engine=myisam;

Query OK, 0 rows affected (0.18 sec)

You will see there are three files related with the table

-rw-rw----. 1 mysql mysql 8586 Mar 25 14:38 mytable.frm

-rw-rw----. 1 mysql mysql    0 Mar 25 14:38 mytable.MYD

-rw-rw----. 1 mysql mysql 1024 Mar 25 14:38 mytable.MYI

• *.frm file contains the table related meta files including the table’s definition information
• *.MYD file contains the data for the table.
• *.MYI file contains the index information for the table.

data file in innodb

When we create a table using innodb engine, there is a option called innodb_file_per_table which will decide if every table has single files or shared file. Please use the command

> show variables like ‘%per_table%’;

To verify the setting

To change the settings,  innodb_file_per_table=0 should be in my.cnf’s [mysqld] section then restart the database.

• innodb_file_per_table – on : each table will create a single innodb file for individual table
• innodb_file_per_table – off: tables will share the innodb files

when innodb_file_per_table is on

mysql> create table intable(id int ,name varchar(20)) engine=innodb;

Query OK, 0 rows affected (0.26 sec)

You may see two files for the table

-rw-rw----. 1 mysql mysql  8586 Mar 25 14:43 intable.frm (the meta file)

-rw-rw----. 1 mysql mysql 98304 Mar 25 14:43 intable.ibd (the data file)

when innodb_file_per_table is off.

All the table will share the same file for example ibdata1(datafile) and ib_logfile0 (logfile) but you should still be able to see the meta file

mysql> create table intablesinglefile1(id int ,name varchar(20)) engine=innodb;

Query OK, 0 rows affected (0.19 sec)

mysql> create table intablesinglefile2(id int ,name varchar(20)) engine=innodb;

Query OK, 0 rows affected (0.08 sec)

-rw-rw----. 1 mysql mysql  8586 Mar 25 15:04 intablesinglefile1.frm

-rw-rw----. 1 mysql mysql  8586 Mar 25 15:04 intablesinglefile2.frm

Nginx process structure

Usually Nginx is working in multiple thread mode, there will be a master process, it will prefork multiple children processes called worker processes. 

• Master is not worked for web client connection and session management. It only manages the children processes.
• Worker process handles the requests from clients (usually http/https) and servers them. The number of worker processes are decided by configuration

Configuration Syntax

user userid;

worker_processes  [number];   #define how many worker processes

events{

worker_connections [number];                #define how many connection one process can server

}

Sample configuration

user  nginx;

worker_processes  10;

the processes output

root      5031     1  0 13:41 ?        00:00:00 nginx: master process ./nginx

nginx     5032  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5033  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5034  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5035  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5036  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5037  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5038  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5039  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5040  5031  0 13:41 ?        00:00:00 nginx: worker process

nginx     5041  5031  0 13:41 ?        00:00:00 nginx: worker process

process 5031 is the master(parent) process

processes 5032-5041 are the worker (children) process

hardware and core configuration

low traffic website	medium traffic website	high traffic website
CPU : 2 cores, memory 2GB, visits ~ 1/s	CPU : 4 cores, memory 4GB, visits ~ 50/s	CPU : 8 cores, memory 12GB, visits ~ 1000/s
worker_processor 2; worker_priority -4; worker_cpu_affinity 01 10 events{      worker_connections 128; }	worker_processor 4; worker_priority 0; worker_cpu_affinity 0001 0010 0100 1000 events{      worker_connections 1024; }	worker_processor 8; worker_priority 0; events{      worker_connections 8192; }