Hi all,
I finally got a first version of a quite complete KDE 4.9.5 distribution compiled and working on CentOS 6.3.
After a while having troubles with kdesrc-build I decided to just download the sources linked on the page http://www.kde.org/info/4.9.5.php and compile them one-by-one. The most of the dependencies like QT 4.8, attica and so on, I already build by hand or with help of kdesrc-build.
(more…)
When I was measuring execution time in a new PHP application using Doctrine and Zend_Date, I discovered lots of time is consumed by the code. It will slow down the app really if you show 50 objects in a list on a page when every object uses 100ms to list the results. It’s hard to get that library code fast, it’s easier to write a cache class around the object.
(more…)
I recently installed OpenSUSE 11.2 on my netbook. There’s a lot of great new features like KDE 4.3, much better kernel 2.6.31 support for Intel Graphics and the Atheros wifi, but also good things disappear from the distro like some good working old kde3 applications. KDevelop3 is still a favorite for my daily work. However kdevelop4 is there, it is still having bugs and lacking functionality and not in a final release. I still want to work with KDevelop3. Well, it’s quite easy to fix that.
(more…)
After all the signs from the ICT business world about the lack of interrest in deploying IPv6 because it should be too difficult and wouldn’t worth the investment, yesterday I found an article (Dutch techworld.nl, read more on Googles blog post) about the deployment of IPv6 at Google. They stress about the ease it takes to implement it. Actually exactly the same I think about it. When I deployed IPv6 about 2 years ago on my own network with the SiXXS and XS4All IPv6 tunnel brokers, it took only a little effort, I even didn’t have to restart my old BSD machines that were used as network gateways to get the network connected via IPv6. (more…)
Today I rewrote the previous mysql - memcache classes, mentioned in my previous post.
I wrote a clas with static methods (why should you need an object at all?) with logic. supporting an easy way store items in a PHP application. A second DB class extends this one, for an easy way of caching complicated queries.
(more…)
Today I discovered a problem with my PHP/MySQL application. SQL data coming from the Zend_Db logic has the wrong encoding.
Like IMO all modern applications should, my application only uses UTF-8 for displaying and handling all non-ASCII characters. To make the MySQL server understand that I talk UTF-8 to it, I always use the following immediately after connecting:
(more…)
For a few years I have been working with a Toshiba Satellite Pro 4600. A nice one, but heavy and huge and the machine started to get ramshackle and having more and more serious problems, like a bad battery, unexpected fallouts and display backlight problems. Therefore I decided to buy a new laptop so I can still work in the train.
My attention felt on the new series of netbooks. Nice small laptops of only about one kilogram, I could easily take with me when traveling, instead of the almost 3kg Toshiba brick I was used to.
(more…)
Recently I had to improve performance of several PHP sites. One of the problems is complicated queries, that take a long time to compute by the MySQL server. Therefore, I decided to look for a solution.
I found a nice solution on http://pureform.wordpress.com/2008/05/21/using-memcache-with-mysql-and-php/ and I decided to work this out for my sites.
For php4 sites I rewrote this code to a simple class:
/**
* Bastiaan Welmers - 20081013
* instataniate this class and use mysql_query_cache instead of mysql_query for slow queries
*
* php4 style class
*
*/
class memCacheDb
{
var $memcache;
// constructor
function memCacheDb()
{
# Connect to memcache:
$this->memcache = new Memcache;
$this->memcache->connect('localhost', 11211) or die ("Could not connect to memcache");
}
# Gets key / value pair into memcache ... called by mysql_query_cache()
function getCache($key) {
return ($this->memcache) ? $this->memcache->get($key) : false;
}
# Puts key / value pair into memcache ... called by mysql_query_cache()
function setCache($key,$object,$timeout = 60) {
return ($this->memcache) ? $this->memcache->set($key,$object,MEMCACHE_COMPRESSED,$timeout) : false;
}
# Caching version of mysql_query()
function mysql_query_cache($sql, $linkIdentifier = false, $timeout = 600 /* = 10 minutes expiration */) {
if (!($cache = $this->getCache(md5("mysql_query" . $sql)))) {
$cache = false;
$r = ($linkIdentifier !== false) ? mysql_query($sql,$linkIdentifier) : mysql_query($sql);
if (is_resource($r) && (($rows = mysql_num_rows($r)) != 0)) {
for ($i=0;$i<$rows;$i++) {
$fields = mysql_num_fields($r);
$row = mysql_fetch_array($r);
for ($j=0;$j<$fields;$j++) {
if ($i == 0) {
$columns[$j] = mysql_field_name($r,$j);
}
$cache[$i][$columns[$j]] = $row[$j];
}
}
if (!$this->setCache(md5("mysql_query" . $sql),$cache,$timeout)) {
die('Error trying to connect to memcache');
# If we get here, there isn't a memcache daemon running or responding
}
}
}
return $cache;
}
}
For the PHP5 applications I wrote a PHP5 style style class and I added expiration logic. You can now use MemCacheDb::deleteMatching($arrayOfKeywords) to remove queries from cache that contain certain keywords. You can use this for example in your CMS, just call the deleteMatching method in the update/creation logic with the updated table name in the keywords and all cached queries will expire.
/**
* Bastiaan Welmers - 20081013
* instataniate this class and use this mysql_query methoid instead of mysql_query for slow queries
* it wil return a array with assosiative arrays with the output.
* So use foreach($mdb->mysql_query($query) as $rows) {}
*
* php5 style class
*
*/
class MemCacheDb
{
private $_memcache;
private $_appKey = 'Change this key for your application';
private $_queries = null;
// constructor
public function __construct()
{
// Connect to memcache:
$this->_memcache = new Memcache();
$this->_memcache->connect('localhost', 11211) or die ("Could not connect to memcache");
$this->_appKey = md5(__FILE__ . $this->_appKey);
}
public function __destruct()
{
if (is_array($this->_queries))
{
$this->_saveQueries();
}
}
public function getStats()
{
return $this->_memcache->getStats();
}
private function _loadQueries()
{
if (!is_array($this->_queries))
{
$queries = $this->_memcache->get($this->_appKey . '__queries');
if ($queries !== false && is_array($queries))
$this->_queries = $queries;
else
$this->_queries = array();
}
}
public function getQueries()
{
$this->_loadQueries();
return $this->_queries;
}
public function searchQueries($search)
{
$queries = array();
foreach($this->getQueries() as $sql => $timestamp)
{
if (strpos($sql, $search) !== false)
{
$queries[] = $sql;
}
}
return $queries;
}
public function deleteQuery($sql)
{
$this->_memcache->delete($this->_getQueryKey($sql));
$this->_loadQueries();
if (isset($this->_queries[$sql]))
unset($this->_queries[$sql]);
$this->_saveQueries();
}
public function deleteMatching($searchItems)
{
if (!is_array($searchItems))
{
$searchItems = array($searchItems);
}
foreach($searchItems as $searchItem)
{
foreach($this->searchQueries($searchItem) as $query)
$this->deleteQuery($query);
}
}
public function deleteAll()
{
foreach($this->getQueries() as $sql => $timestamp)
{
$this->deleteQuery($sql);
}
}
private function _saveQueries($timeout = 1800)
{
if (is_array($this->_queries))
{
// clean old stuff
foreach($this->_queries as $sql => $timestamp)
{
if (time() - $timestamp > $timeout)
unset($this->_queries[$sql]);
}
return $this->_memcache->set($this->_appKey . '__queries', $this->_queries, 0, $timeout);
}
else
return false;
}
// Gets key / value pair into memcache … called by mysql_query_cache()
private function _getCache($key) {
return ($this->_memcache) ? $this->_memcache->get($key) : false;
}
// Puts key / value pair into memcache … called by mysql_query_cache()
private function _setCache($key, $object, $timeout) {
$this->_loadQueries();
if ($this->_memcache instanceof Memcache)
return $this->_memcache->set($key, $object, 0, $timeout);
else
return false;
}
private function _cacheQuery($sql, $object, $timeout)
{
if ($this->_setCache($this->_getQueryKey($sql), $object, $timeout))
{
$this->_loadQueries();
$this->_queries[$sql] = time();
$this->_saveQueries($timeout);
return true;
}
else
{
return false;
}
}
public function expireQuery($query)
{
$this->_memcache->delete($this->_getQueryKey($query));
}
public function mysql_query($sql, $linkIdentifier = false, $timeout = 600)
{
return $this->mysql_query_cache($sql, $linkIdentifier, $timeout);
}
private function _getQueryKey($sql)
{
return md5($this->_appKey . $sql);
}
// Caching version of mysql_query()
public function mysql_query_cache($sql, $linkIdentifier = false, $timeout = 600 /* = 10 minutes expiration */) {
if (!($cache = $this->_getCache($this->_getQueryKey($sql)))) {
$cache = array();
$r = ($linkIdentifier !== false) ? mysql_query($sql,$linkIdentifier) : mysql_query($sql);
if (is_resource($r) && (($rows = mysql_num_rows($r)) != 0)) {
for ($i=0;$i<$rows;$i++) {
$fields = mysql_num_fields($r);
$row = mysql_fetch_array($r);
for ($j=0;$j<$fields;$j++) {
if ($i == 0) {
$columns[$j] = mysql_field_name($r,$j);
}
$cache[$i][$columns[$j]] = $row[$j];
}
}
if (!$this->_cacheQuery($sql, $cache, $timeout)) {
die('Error trying to connect to memcache');
// If we get here, there isn't a memcache daemon running or responding
}
}
}
return $cache;
}
}
TODO: this code need to get more documentation, and a static method to store the object instance so it can be called everywhere in the application.
Today I decided to install AWStats for analysing my webserver traffic.
Running Debian Etch with apache 2.2 on the webserver / webproxy, the following steps are taken:
apt-get install awstats
LoadPlugin="ipv6"
LoadPlugin="decodeutfkeys"
apt-get install libnet-dns-perl libnet-ip-perl libnet-ipv6addr-perl liburi-perlInclude "/etc/awstats/awstats.conf" LogFile=”/var/log/apache2/www.welmers.net-access.log” LogType=W LogFormat=1 SiteDomain=”www.welmers.net” DefaultFile=”index.php index.html” LevelForWormsDetection=2
The log format is set to 1. This is the apache combined log format, with referer URLs and browser info. Use 4 for the simple apache log format (however I would recommend combined format anyway)
for i in `ls www.welmers.net-access.log.* | sort -n -r -t . -k 5`; do
zcat $i >> /tmp/logfile;
done
cat www.welmers.net-access.log.1 >> /tmp/logfile
cat www.welmers.net-access.log >> /tmp/logfile
Edit /etc/awstats/awstats.www.welmers.net.conf to temporarily use /tmp/logfile instead of /var/log/apache2/www.welmers.net-access.log
Then:
perl /usr/lib/cgi-bin/awstats.pl -config=www.welmers.net -update
And edit back /etc/awstats/awstats.www.welemrs.net.conf with the usual logfile.
cp /usr/share/doc/awstats/examples/apache.conf /etc/apache2/awstats.conf
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/ could be removed because it was already mentioned in my virtual host.
# m h dom mon dow command
*/10 * * * * for config in `ls /etc/awstats/awstats.*.conf | cut -f2,3,4 -d"."`; do perl /usr/lib/cgi-bin/awstats.pl -config=$config -update; done
prerotate
for config in `ls /etc/awstats/awstats.*.conf | cut -f2,3,4 -d"."`; do
perl /usr/lib/cgi-bin/awstats.pl -config=$config -update
done
endscript
Done! My statistics get updated once per 10 minutes and I can view them easily via the awstats.pl CGI with the paramater ?config=www.mysite.com
Today I wrote a script for gathering health statistics on my servers. It gets data from hddtemp and sensors and writes it to a MySQL table.
The script and how-to-use can be found on the wiki: http://wiki.welmers.net/en/Health2MySQL
The MySQL table can easily be analized by (web) scripts to graphs e.d., functionality I still need to write.
Things that already can be done is finding interresting issues like:
SELECT AVG(temp_cpu) FROM health WHERE datetime BETWEEN '2008-07-25 00:00:00' AND '2008-07-30 23:59:59'
SELECT MAX(temp_sda) FROM health
SELECT MIN(rpm_fan) FROM health