Bastiaans Blog

Today I decided to install AWStats for analysing my webserver traffic.

Running Debian Etch with apache 2.2 on the webserver / webproxy, the following steps are taken:

• Install awstats with apt-get install awstats
• Put following in /etc/awstats/awstats.conf.local:
  
LoadPlugin="ipv6"
LoadPlugin="decodeutfkeys"

• Install the needed perl libs:
  apt-get install libnet-dns-perl libnet-ip-perl libnet-ipv6addr-perl liburi-perl
• Create config files for each website running:
  vim /etc/awstats/awstats.www.welmers.net.conf

  Include "/etc/awstats/awstats.conf"
  
  LogFile=”/var/log/apache2/www.welmers.net-access.log”
  LogType=W
  LogFormat=1
  SiteDomain=”www.welmers.net”
  DefaultFile=”index.php index.html”
  LevelForWormsDetection=2
  

  The log format is set to 1. This is the apache combined log format, with referer URLs and browser info. Use 4 for the simple apache log format (however I would recommend combined format anyway)

• After this, some weird bug showed up in awstats. It warns about nested includes that don’t work because perl version < 5.6 is installed. Great, I got 5.8.8. I found this bug on http://www.mail-archive.com/debian-bugs-dist%40lists.debian.org/msg385558.html and applied the patch.
• Next thing to do is to create initial awstat databases with the logfiles I already have.
  One problem: they are alreay rotated several times. This can be solved this way:
  rm /tmp/logfile

  for i in `ls www.welmers.net-access.log.* | sort -n -r -t . -k 5`; do
  zcat $i >> /tmp/logfile;
  done

  cat www.welmers.net-access.log.1 >> /tmp/logfile
  cat www.welmers.net-access.log >> /tmp/logfile

  Edit /etc/awstats/awstats.www.welmers.net.conf to temporarily use /tmp/logfile instead of /var/log/apache2/www.welmers.net-access.log
  Then:
  
perl /usr/lib/cgi-bin/awstats.pl -config=www.welmers.net -update

  And edit back /etc/awstats/awstats.www.welemrs.net.conf with the usual logfile.

• Great. all old stuf is in awstats database! Now make it visible:
  
cp /usr/share/doc/awstats/examples/apache.conf /etc/apache2/awstats.conf

  Include this file in your favorite virtual host container where awstats stuff can be viewed by authorized people only. Higly recommended, the reputation of awstats.pl CGI makes it neccessary to do so.
  After inclusion, the script alias ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ could be removed because it was already mentioned in my virtual host.
  Now, when calling https://securehost.domain.com/cgi-bin/awstats.pl?config=www.welmers.net , you should be able to see all gathered data in the log files.
• Now we want the awstats dbases update automagicly every 10 minutes, and just before rotating the apache logfiles:
  EDITOR=vim crontab -e
  
# m h dom mon dow command
*/10 * * * * for config in `ls /etc/awstats/awstats.*.conf | cut -f2,3,4 -d"."`; do perl /usr/lib/cgi-bin/awstats.pl -config=$config -update; done

  vim /etc/logrotate.d/apache2
  and add:
  
prerotate
for config in `ls /etc/awstats/awstats.*.conf | cut -f2,3,4 -d"."`; do
perl /usr/lib/cgi-bin/awstats.pl -config=$config -update
done
endscript

  in the /var/log/apache2/*.log block.

Done! My statistics get updated once per 10 minutes and I can view them easily via the awstats.pl CGI with the paramater ?config=www.mysite.com

Today I wrote a script for gathering health statistics on my servers. It gets data from hddtemp and sensors and writes it to a MySQL table.

The script and how-to-use can be found on the wiki: http://wiki.welmers.net/en/Health2MySQL

The MySQL table can easily be analized by (web) scripts to graphs e.d., functionality I still need to write.

Things that already can be done is finding interresting issues like:

SELECT AVG(temp_cpu) FROM health WHERE datetime BETWEEN '2008-07-25 00:00:00' AND '2008-07-30 23:59:59'

SELECT MIN(rpm_fan) FROM health


Yesterday and today I’ve set up a new server for colocation.
Properties and Features:

• celeron 2GHz
• 2,5GB mem
• 2×500GB ordinary sata disks in RAID-1
• debian etch installed on it
• serial console (both grub and linux itself)
• all it will serve and compute nicely sorted and devided in vservers

This was the first time I’ve installed a system in RAID and with serial console.

First thing to do was buying the hardware and wire and install everything properly.

Next thing to do was to check everything works in the BIOS. Nice to see 2,6GB RAM and 2×500GB disks in the BIOS startup screen at once without any trouble.

Then install the operating system. I attached an old DVD drive to the mainboard to start up from CD-ROM.

The choosen disk layout will be:

• both disks a 200MB boot partition and a near-500GB system partition as linux-raid
• combine partitions on each disk to md devices
• install ext3 for /boot on the first 200MB md and install LVM on the big second md for al the other partitions
• in LVM partitions:
  • root 512MB
  • /usr 5GB
  • /var 3GB
  • swap 3GB
  • /tmp 1GB
  • /home 20GB
  • /home/mail 10GB optimized for Maildir, 1 inode per 4kb (news options) and mount options noatime, nosuid, noexec
  • /home/backup 25GB
  • /var/local/nntpcache 2GB news cache spool, 1 inode per 4kb and mount options noatime

  This will be enough for a while in the future. I left almost 400GB of non-partitioned space on LVM, which can be used to extend volumes or add volumes for future expanding.

To fix this with de debian installer:

• First create a DOS label on both disks
• Create on each disk a 200MB for booting and a second partition of almost 500GB for LVM. Partition Type fd (linux-raid). Give the 200MB partition a boot flag.
• Set up RAID, couple both 200MB and the near-500GB partition to two RAID-1 devices, md0 and md1
• setup the 200MB md0 device to format as ext3 and use /boot as mount point
• setup the big md1 device to use as LVM
• setup LVM: make a volume group on the LVM device (I named it raid1)
• create logical volumes like root, swap, usr, var, etc.
• close the logical volume manager setup
• format all logical volumes properly (I choose ext3) and give it mount points
• finish partitioning!

After partitioning and installing base .deb packages I choose to install only the standard system set of packages. All other functionality will be in the vservers, which are already there and will be copied from the old servers.

Unfortunatly the debian installer left me no choice between boot loaders, I had to use lilo as boot loader. Strange, since I left a boot md0 specialy to use with grub. I’d like grub rather than lilo because if I once forget to run lilo properly after a kernel upgrade, I’ll get stuck with an unbootable server and need to travel to the colocation to fix up things. Grub will always work with serial console even if it can’t find a kernel, because it can load kernels dynamicly from the grub CLI.

After reboot I noted two things:

• the installer didn’t use my seperate /boot partition at all, although I’m sure I’ve setup a ext3 formated /boot file system for it in the installer. The md0 device was left ext3 formatted by the installer. Bug somewhere?
• The raid had to be build up according to /proc/mdstat and it took hours… I just left the server on without reboot to finish this, since I don’t know if rebooting before with slow down or restart the process…

Next thing to do is to install grub and get it work with serial console. More on this later…

Launched my first blog post after installing WordPress! Wow….

I installed wordpress in a codebase -> symlink jailed public_html structure. So the WordPress code is seperated from the blog specific content. On this way Wordpress can easliy be upgraded, and I can easily create another independend blogs.

• ~/wordpress/ contains all unchanged wordpress code and can be upgraded by extracting a new tar ball.
• ~/public_html/bastiaan/ contains a symlink jail to ~/wordpress/
  Non-shared items like wp-config.php are just individual files.

I had to hack a new ~/wordpress/wp-config.php so it uses the wp-config.php from the link jail directory (found by $_SERVER['SCRIPT_FILENAME']):

/* 2008 Bastiaan Welmers haasje@welmers.net - wp-config.php usefull for symlink jails */
$dir = dirname($_SERVER['SCRIPT_FILENAME']) . '/';
for ($i = strlen($dir); $i > 0; $i–) {
    if ($dir[$i] == '/') {
        if (file_exists(substr($dir, 0, $i) . '/wp-config.php')) {
              include(substr($dir, 0, $i) . '/wp-config.php');
              break;
        }
    }
}
?>

Update: fixed code block