Zein’s place on the net

Once upon a time, the options available to Linux users interested in managing their book collections were limited. The situation was so bleak that I attempted to create my own web-based application. That app was fine until I tried to host it on my 64MB VPS and failed (RoR had a footprint of 60MB all by itself)!

Lately, I have been missing the ability to browse through my books. I started thinking about having another attempt at a book collection manger. Maybe this time I would try and write a desktop application. I could use python or even try out mono. So today I did a bit of googling and stumbled upon Alexandria, a book collection manager for GNOME.

This application is brilliant! It is almost the perfect application. All I needed to do was get a list of books in my old app:

echo "select isbn from books;" | sqlite3 mybooklist_production.sqlite > import-list.txt

I was then able to import all my old books into Alexandria using the import ISBN list feature. I would like to recommend this app to anyone who is looking for a book collection manger for Linux. The site also has deb packages for those who use Debian or Ubuntu.

I have not found a netbook that I feel 100% comfortable with. All the products out there have either too much or too little. The EEEPC901 comes the closest but is a bit too expensive for my taste (I would guess it is all the extra SSD).

Zein’s perfect netbook:

o Linux based (No Windows tax for me!)
o 9″ or 10″ screen.
o Atom 1.6Ghz
o 1GB RAM
o Large track pad (the Acer for example is too small)
o 4GB (at least and 8GB at most) of storage.
o 4 hours (at least) battery life.
o 1.3 MP Webcam
o Wifi
o bluetooth (optional)

And so I got very excited when I saw that the Dell Mini 9 had been released. The Dell is a lot more customizable at purchase time them the EEEPC, Aspire One or the MSI Wind and therefore a lot cheaper since you only pay for what you want. I got a surprise when I discovered that the Linux and Windows XP versions are identical in price!!

Here are screen shots of the 2 systems after configuration. Both are $US 454 (after $40 instant savings on the XP version) and while that is a lot less then an EEEPC 901. I am baffled to as why I am not saving by not paying any windows tax! I would have expected the Linux version to be at least $US 50 less than the windows version.

The Linux version
The XP version

Is this a poor geeks dream or what? It is like I have been dreaming for a small low cost laptop for years and then all at once every one wants in on this market.

Acer is looking to enter the low cost laptop market with 8″ - 9″ designs. Gigabyte also announced that it will produce what seams to be a low cost laptop due in June. Linux is not mentioned in either case but given the success of the other low cost machines with Linux I would say it is reasonable to assume that there would be a Linux version of both laptops.

In other news, Everex plans to expand their cloudbook line with some nifty ideas.

If the cool (and cheap) Linux laptops were not enough, Shuttle is going to produce $200 Linux PCs. Sure the specs are not so good, but that looks like a good mythtv frontend to me. Or maybe a NAS that doesn’t have to live in the closet?

Different people like different things. Different people fantasize about buying different things. I fantasize about gadgets. My latest fantasies have been about the ASUS EeePC. The 4G model sells for $399, has a 7″ 800×640 screen, runs Linux and looks beautiful.

This is the sort of laptop that I have always dreamed of having yet could never afford. The only thing that is not perfect is the relatively low 800×640 resolution. At CES this year, ASUS have announced 8″ and 8.9″ versions of the EeePC. The 7″ and 8″ EeePCs are the same size and the screens have the same 800×640 resolution. I am hoping that the 8.9″ EeePC has a 1024×600 resolution. Interestingly ASUS EeePcs have been seen running splashtop (an instant on Linux based OS). Could things get crazier ?

Also of interest is the Everex cloudbook (also $399) based on the VIA nanobook design. It has been announced to start selling on the 25th of this month. The cloud book as a longer battery life and a 30GB HDD (vs the 4GB SSD on the EeePC). If after looking at the pictures, you are wondering where the touch pad is then take a look at this video

Anyway between them there should be a very cool laptop waiting to be bought when I visit the USA in July. If you are as interested as I am then suscribe to the RSS feeds of Cloudbooker and Eeeuser

Email has always been a problem for me. Using web interfaces is a pain. Even the new JavaScript, single-page apps nowadays are bloated and slow. You can’t compare them to evolution, thunderbird or even outlook Most Email Service Providers (ESPs) do provide POP3, but POP3 doesn’t support folders and is not a good option if you would like to access your mail from multiple locations.

So what is the solution? IMAP or the Internet Message Access Protocol gives me all the features I need:

• Multiple clients simultaneously connected to the same mailbox

  I check my email from my Desktop, Laptop and PC at Uni.

• supports folders

  can’t live without them. I don’t believe in labels, and virtual folders. I want to physically partition my mail.

• Read/Write Support 2-way Synchronization

  I am not sure what this is really called but you know how POP3 supports downloading only? Well with IMAP you can upload emails to you mailbox. This also allows you to copy emails between folders or even accounts. It also allows you to do things like have your sent mail folder in your mailbox and have all your clients upload the sent mail there.

So what is the problem then? Most ESPs do not support IMAP. My old ESP (gawab.com) did, but due to the stability issues of gawab, I had to look elseware.

Now I didn’t get your hopes up just to let you down! Remember that $5/month VPS I have? It is time to get some more use out of it!

The plan is to have an IMAP server on my VPS, I don’t want to have the responsibility of running a my own SMTP server, just the IMAP server. I would also like to consolidate my email addresses into one mailbox. This seams like a simple plan.. a good one too, if I don’t say so my self

Solving the SMTP Issue

So I don’t want the responsibility of running a SMTP server so I will pass the buck to google. Google Apps provides email for your domain so you don’t have to worry about 99.99% availability and all that.

Installing and Configuring the IMAP Server

I decided on Dovecot for the IMAP server because of it’s low memory footprint as well as it’s stability and good reputation.
in ubuntu all I needed was to:

apt-get install dovecot-imapd

I wanted to set up IMAP over SSL for better security so I needed to create a SSL certificate

cd /etc/ssl
openssl genrsa 1024 > private/domain.key
chmod 400 private/domain.key
openssl req -new -x509 -nodes -sha1 -days 365 -key private/domain.key > certs/domain.cert

I new that I planned to use a passwd like password file for authentication so I created one right away:

cat >>/etc/dovecot/passwd < <EOF
user@domain:{plain}password
EOF

Then you need to configure dovecot by editing /etc/dovecot/dovecot.conf

protocols = imaps
 
## Logging
log_timestamp = "%Y-%m-%d %H:%M:%S "
 
## SSL settings
ssl_cert_file = /etc/ssl/certs/domain.cert
ssl_key_file = /etc/ssl/private/domain.key
 
## Login processes
login_user = dovecot
 
## Mailbox locations and namespaces
mail_location = maildir:/var/lib/mail/%d/%n
mail_extra_groups = mail
 
## Mail processes
first_valid_uid = 107
last_valid_uid = 107
 
## IMAP specific settings
protocol imap {
  login_greeting_capability = yes
}
 
## Authentication processes
auth_debug = yes
auth_debug_passwords = yes
 
auth default {
  mechanisms = plain
  user = root
  passdb passwd-file {
    # Path for passwd-file
    args = /etc/dovecot/passwd
  }
  userdb static {
    args = uid=dovecot gid=mail domain=domain home=/var/lib/mail/%d/%u
  } 
}
 
## Dictionary server settings
plugin {
}

apt-get install getmail4
mkdir /etc/getmail
chown -R dovecot:mail /etc/getmail

create a getmailrc file /etc/getmail/domain-user that describes the retrieval and delivery phases
It will look something like:

[retriever]
type = SimplePOP3SSLRetriever
server = pop.gmail.com
username = user@domain
port = 995
password = your_password
 
[destination]
type = Maildir
path = /var/lib/mail/domain/user/
 
[options]
delete = false
verbose = 0
delete_after = 0
read_all = false

test it with:

sudo -u dovecot getmail --getmaildir=/etc/getmail  --rcfile=/etc/getmail/domain-user

if all is OK then create a cron job to run say every 15 minutes

# Crontab for getmail. Download mail every 15 minutes
0,15,30,45 * * * *    dovecot getmail --getmaildir=/etc/getmail  --rcfile=/etc/getmail/domain1-user1
5,20,35,50 * * * *    dovecot getmail --getmaildir=/etc/getmail  --rcfile=/etc/getmail/domain2-user2

And there you have it! Just one nifty IMAP mailbox that contains all you email from all your accounts.

About 2 months ago I got myself a Virtual Private Server(VPS) from http://www.vpsvillage.com. I am paying $5 a month for a Xen virtual machine with 64MB of RAM and a 4GB hard disk. I have lots of plans for my VPS. The most essential is hosting my website (basically this blog plus a small rails app). I immediately installed a LAMP environment and had my blog up and running in no time.

Now I have a very small number of hits per day… So small that I am embarrassed to post that number here:). Suffice to say that it would be better to calculate per week or month Anyhow my site site kept dieing on me. The Out of Memory thread (OOM) kept killing off processes on my server.

Rather that tune apache I decided to install lighttpd and try and tune the rest of my system. So I did a fresh install of 64-bit ubuntu (actually I now feel that the 64-bit part was a mistake but I am not sure enough to go through the whole install process again). I will attempt to describe how I went through the process of tuning my 64MB VPS.

Tuning Tools

I installed munin as it produces lovely graphs. The graphing process it quite heavy so to lower its impact I removed all unnecessary plugins (I kept: cpu, load, memory and swap only) and I also modified the cron script so that the graphs and html files are updated every 15 minutes (munin updates every 5 minutes)

I am now able to look at nice graphs of my memory usage

you can see munin in progress here

Tune Linux

Next thing I did was to try and tune the virtual memory subsystem (ie /proc/sys/vm/*). I fist tried to lower the swappiness value but that did not have the result I expected! You see I thought that a lower swappiness value would decrease the systems tendency to swap caches to disk. It in fact decreases the tendency to swap anything to disk. I was getting OOM with swap only a quarter full.
What I did at the end was:

• leave swappiness at the default value of 60
• set vfs_cache_pressure to 200
• set max_map_count to 1000

No doubt further understanding of the VM subsystem would allow more efficient tuning, but that will have to wait for another day.

Tune Lighttpd

With the small load I have and the small memory footprint of lighty (less than 1MB), I can’t see much to be done here! I have setup php o run as a fast-cgi process. I have set max-procs to 1 in lighty with PHP_FCGI_CHILDREN=1. This means we will have 2 php-cgi processes, one is the parent monitoring process (PP) and the other will actually serve requests (SP). I might increase PHP_FCGI_CHILDREN to 2 in the future.

Tune php

I wanted to see what I could do to decrease the memory footprint for my php-cgi processes so I hit my site with:
ab -n 50 http://azein.com/ and fired up top (using M to sort by memory usage) and got

2163 www-data 16 0 77396 13m 3100 S 0.0 21.6 0:08.36 php-cgi
2162 www-data 23 0 69212 6132 3668 S 0.0 9.4 0:00.00 php-cgi


I then edited my php.ini file and modified the following values:

memory_limit = 16M
register_long_arrays = Off
;engine = On

I also commented out the loading of the mysqli, pdo and pdo_mysql modules as I don’t use them.

I tried again with ab -n 50 http://azein.com/and got:

2278 www-data 16 0 71096 12m 3016 S 11.3 20.2 0:09.36 php-cgi
2277 www-data 24 0 62660 3596 3528 S 0.0 5.5 0:00.00 php-cgi

so we have a small decrease in memory foot print.

Tune MySQL

1st thing to do before you even start to tune your parameters is to skip-innodb this by itself is supposed to decrease the memory foot print by 100MB.

The 1st parameter I will tune is the key_buffer. This describes the amount of memory used to cache the tables keys and indexes. The sum of my *.MYI files are about 80K so I will set the key_buffer size to 512KB with lots of room for expansion.

The 2nd set of parameters are the threads and concurrency group, now I plan to have no more than 2 php-cgi threads running at any time so it make no sense to have more than 2 mysqld threads running for now. I also set the max connections to 10 as I do not expect any more than that.

max_connections = 20
thread_concurrency = 2


The 3rd set of parameters are the query cache configuration group. I do not have many queries that I think need to be cached. I suppose the front page of my blog (about 10 posts) is all that will benefit from this.

query_cache_limit = 64K
query_cache_size = 512K

mysqld now consumes just under 7MB but I will take a look at my hit/miss stats in a few days to how the query cache parameters are doing.

Last thing I did was turn off binary logging and I do not plan to use it.

Final Comments

No mater what I did I couldn’t get my rails app running without the system coming to a halt! The rails app has a memory footprint of 30MB and I don’t have the energy to see if it possible to bring that down and what the minimal memory footprint of a rails app is!

I could upgrade my VPS to 128MB RAM, but I think that 64M should be sufficient and I will continue trying to make the most out of it.

Last week one of hard disks in our cluster failed. all of a sudden it was not detectable by the BIOS and it emits loud clicking noises. It had 3 Xen guests running on it and as they were work in progress I stupidly did not back them up! One of the guests was the web server and it had the wiki with all the documentation on it. Needless to say I was in a very bad way. However I had read about this idea of putting the hard disk into a freezer for a while. I didn’t really have anything to lose so I decided to try.

I put it in the freezer for a while then tried to access it with a IDE to USB converter without any luck. It did however stop the clicking noises for a while. I repeated the freezing process then trying attaching it to the server vi the IDE cable. This time I got better results. The BIOS recognized the drive! it even reached the loading grub stage. Then it started making the clicking noises again.

It was my supervisor, Alistair who jokingly (at least I don’t think he was serious) talked about using it inside the freezer. So that is exactly what I did!

I took the drive home with me, attached the cables and put it all in the freezer. The next day I fixed it so that the power and USB cables are reaching outside the freezer with the freezer door shut and the hard disk freezing inside. I attached the USB cable to my laptop and attached the power cable and to my utter surprise the disk came up! I ‘dd’ed 2 of the Xen guests as their partitions were only 1GB each and I backed up some files from the 3rd at it was a 20GB partition.

All in all I am very happy that I was able to recover and I give thanks to Allah that he saw me through. So next time you have a hard disk failure, try the freezer trick. It just might work.

The first time I saw a sunray was when I was doing my undergrad (2001). It was love at first sight. And even though the AAST wasn’t using them (there were about 4 thrown in a box), there was no chance of me getting my hands on one

Fast forward to 2007! I am at the ANU and I end up with 2 sunray1 devices. One of the best things about the Department of Computer Science is that when they are about to throw things out, they leave it in front of the Technical Support Group’s Offices for about a week! usually a student will pick things up!

So you could say I get first picks either way I am now the proud owner of 2 sunray1 devices. During the last 2 months I also picked up a 17″ CRT and a few old machines that were pillaged to produce a dual PIII800Mhz system with 768MB of ram and 2 13GB HDDs!

Today I installed SRSS3.1 for Linux and I am presently writing this post on the sunray! words can’t describe how cool this is :). And when I finally move out of Fenner It will even better!! This is a silent device… no moving parts whatsoever. Since I have no room at Uni accommodation I have the sunray sitting next to the server (which is very noisy) but as soon as I move out I will be able to stick the server where it will not be heard.

Anyway here is how I went about setting up the server.

      apt-get install ubuntu-desktop

      apt-get install sun-java5-jre sun-java5-plugin

      ln -s /usr/lib/jvm/java-1.5.0-sun-1.5.0.08/jre /etc/opt/SUNWut/jre

The Installation was pretty easy. I do however seem to have a problem with audio. When I login in I get the login sounds. That is the last sound I do hear. I am unable to play any audio… so I am working on that.

The other thing that might be of interest is that I had always thought that the sunrays were net-bootable PCs. It turns out that they are not! they are rather a continuation of the server. so when you login on the sun you are working on the server.

I will of course post something if/when I get the audio working properly.

Yesterday I was lucky enough to attend a lecture by Andrew Tridgell at the ANU. He talked about some of the techniques used to analyze undocumented network protocols.

It was a very interesting talk and talk was well presented. I first found it a bit odd that he would just show up at the ANU like that, but it seems that Andrew was at the ANU from 1989 to 1999 and remains a Visiting Fellow of the University.