Month: September 2010

Dùng lệnh linux để phân tích log file của apache (linux command apache Log file analyzer)

Dùng lệnh linux để phân tích log file của apache (linux command apache Log file analyzer)
How to view log files?

Less can easily handle huge files.

Without line wrapping
% less -S access_log

Long lines wrapped
% less access_log

How to view compressed log files?
To save space log files are usually compressed.
Gzip (GNU zip) compressed file (.gz)
% gzip -dc access_log.gz | less

Bzip2 compressed file (.bz2)
% bzip2 -dc access_log.bz2 | less

How many lines (hits) there are in the log file?
Uncompressed log file
% wc -l access_log 33894

Gzip compressed log file
% gzip -dc access_log.gz | wc -l 33894

How many page views?
Uncompressed log file
% egrep ‘(.gif |.jpg |.png )’ access_log 2569

Compressed log file
% gzip -dc access_log.gz | egrep -vc ‘(.gif |.jpg |.png )’ 2569

How many hits today?
% grep -c `date ‘+%d/%b/%Y’` access_log 2569

How many unique visitors today?
% grep `date ‘+%d/%b/%Y’` access_log | cut -d" " -f1 | sort -u | wc -l 1196

How many hits in a particular day?
Uncompressed log file, e.g. January 1, 2009
% grep -c 01/Jan/2009 access_log 2569

Compressed log file, e.g. January 1, 2009
% gzip -dc access_log.gz | grep -c 01/Jan/2009 2569

What period is covered covered in the log?
Uncompressed log file
% head -1 access_log; tail -1 access_log foo.example – –
[30/Dec/2009:23:55:25 +0200] "GET /~vnntech/ … bar.example – –
[06/Jan/2009:23:53:37 +0200] "GET /~vnntech/rates.html …

Uncompressed log file
% gzip -dc access_log.gz | head -1 ; gzip -dc access_log.gz | tail -1
foo.example – – [30/Dec/2009:23:55:25 +0200] "GET /~vnntech/ …
bar.example – – [06/Jan/2009:23:53:37 +0200] "GET /~vnntech/rates.html

Are there missing dates?
Uncompressed log file
% cut -d" " -f4 access_log | cut -d"/" -f1 | uniq [30 [31 [01 [03 [04 [05 [06

Compressed log file
% gzip -dc wlog0101.gz | cut -d" " -f4 | cut -d"/" -f1 | uniq [30 [31 [01 [03 [04 [05 [06

How many corrupted log entries?
This is just a very quick and dirty way to check the log.
Uncompressed log file
% perl -ane ‘print $_ if (scalar (split /"/)) != 7’ access_log | wc -l

7

Compressed log file
% gzip -dc access_log.gz | perl -ane ‘print $_ if (scalar (split /"/)) != 7’ | wc -l

7

How does the line number 15927 or lines 15920 – 15929 look like?
Uncompressed log file

% grep -n ‘.*’ access_log | grep ‘^15927:’ 15927:foo.vnntech.com – – [20/Jan/2009:11:23:45 +0200] "GET …



% grep -n ‘.*’ access_log | grep ‘^1592.:’ 15920:foo.vnntech.com – –
[20/Jan/2009:11:23:40 +0200] "GET … 15921:foo.vnntech.com – –
[20/Jan/2009:11:23:41 +0200] "GET … 15922:foo.vnntech.com – –
[20/Jan/2009:11:23:41 +0200] "GET … …

Compressed log file

% gzip -dc access_log.gz | grep -n ‘.*’ | grep ‘^15927:’ 15927:foo.vnntech.com – – [20/Jan/2009:11:23:45 +0200] "GET …

% gzip -dc access_log.gz | grep -n ‘.*’ | grep ‘^1592.:’
15920:foo.vnntech.com – – [20/Jan/2009:11:23:40 +0200] "GET …
15921:foo.vnntech.com – – [20/Jan/2009:11:23:41 +0200] "GET …
15922:foo.vnntech.com – – [20/Jan/2009:11:23:41 +0200] "GET … …

How to figure out the bandwith consumption (in bytes)?
Today:
% grep `date ‘+%d/%b/%Y’` access_log | awk ‘{ s += $10 } END {print s}’ 13113756

This month:
% grep `date ‘+../%b/%Y’` access_log | awk ‘{ s += $10 } END {print s}’ 569477018

Used by Googlebot:
% grep googlebot access_log | awk ‘{ s += $10 } END {print s}’ 29832233

Used by some rogue user from IP-address 169.254.22.12:


% grep ^169.254.22.12 access_log | awk ‘{ s += $10 } END {print s}’ 46760880

How to delete partial GET requests from the log?
Partial content requests are usually generated by download managers to
speed the downloading of big files and Adobe Acrobat Reader to fetch PDF
documents page by page. In this example 206 requests generated by
Acrobat reader are deleted so that they don’t infate the hit count.

% grep -v ‘.pdf .* 206 ‘ access_log > new_log

How to compress a selected portion from a log?
Use gzip to compress log entries in May 2009
% grep ‘ [../May/2009:’ access_log | gzip -9c > access_log-2009-05.gz

Use bzip2 to compress log entries in May 2009
% grep ‘ [../May/2009:’ access_log | bzip2 > access_log-2009-05.bz2

See in real-time how the log file grows?
Using tail
% tail -f access_log

With less you must hit "F" (and Ctrl-C q to quit)
% less access_log

Nguồn : http://vnntech.com/forums/dung-lenh-linux-de-phan-tich-log-file-cua-apache-linux-command-apache-log-file-a-d104.htm

Ý nghĩa của /dev/rmt/0cn trên Unix ???

/dev/rmt/0cn


Following the below info:
———————————————————————————-
# SYNTAX: /dev/rmt/<unit number><density>[<BSD behavior>][<no rewind>]

· Density: l, m, h, u/c (low, medium, high, ultra/compressed, respectively)
· BSD behavior: (optional) b
· no rewind: (optional) n
· For example, /dev/rmt/0hbn specifies unit 0, high density,
BSD behavior and no rewind.