Monthly Archives: August 2008

SSL on the web server (update)

Know how sometimes you feel that you must be the stupidest person on the planet. Happened to me today. Was wondering why the redirect to the SSL protected server was not working outside the home network until I realize that I forgot to forward the port on the router. So it is working now. Ha ha.

SSL on the web server

Converted the web server to use SSL to encrypt the traffic. The following guides were extremely useful for this:

– http://www.tc.umn.edu/~brams006/selfsign.html
– http://www.tc.umn.edu/~brams006/selfsign_ubuntu.html

The first guide shows how to create a certificate and the second to configure apache for SSL.

Backups

Backups

This page describes my backup strategy. I have a linux machine (fillmore) which I use to store all my data. It has more than 2TB of storage so backing up everything is clearly not going to work and I really hate to use DVDs for backups. I do not really think they are very reliable. To better manage the backup, I categorize the data into the following types:

  • Important data – data which are generated by me or which are important for my work. These can be further split into two types, one for dynamic data and one for static data. Dynamic data are important data which are still being worked on while static data are ones which are not going to be changed. This type of data would include all files generated during my work and important personal data like photos.
  • Normal data – these are mostly stuff which are obtained from external sources and which can be obtained again or are not essential. These are not backed up.
  • System data – these are data which are used by and came with the operating system. They can always be obtained via a new installation so there is really no point in backing them up other than for the reason of a fast system restore.

Unison

All of the dynamic important data is found on my macbook pro (ernest) and that is usually the most recent copy. Unison is used to sync the non-code data to fillmore. This results in 2 copies of the data in two different places. There is a delay between ernest and fillmore but the difference is typically not much. I sync quite often so this is not too bad a problem.

Monotone

All my codes are kept in a versioning system called Monotone. Fillmore is typically the server while ernest checks in the codes. So again there are 2 copies of the codes and they are version controlled. Monotone allows checking in via network.

Rsync

Rsync is then used to sync important dynamic data on fillmore to another partition. A cron job is used to sync the data on a regular basis. Static important data is also housed on fillmore and rsync is used to maintain 2 copies of the data on 2 different harddrives.

Rsnapshot

Rsnapshot is used to maintain weekly snapshots of the dynamic important non-code data.

All these measures does not place the data in 2 different physical location other than data which I carry around on ernest. So this is probably the biggest drawback to this.

My setup 20080807 update

It has been a really busy week or two. The disks were filling up (1.7TB) and I wanted to have a RAID setup for the really important stuff. So off I went to get 2 1TB drives. Initially I wanted to get an additional SATA controller to drive the new drives since there was quite a fair bit of space in the casing. However this proved to be next to impossible. I wanted to get a Silicon Image 3124 SATA controller since it was cheap and know to work well in Linux. I searched around and there appeared to be one shop in Sim Lim that carries a card that uses that chip. So off I went down to Sim Lim to buy the card and the hard drives. When I got there, it was out of stock and they do not know when they will be getting new stock. Bummer, so off I went running from shop to shop to locate for a SATA controller. And I could not find a single Silicon Image 3124 in the whole of Sim Lim. Maybe I was not trying hard enough, but there were a lot of other controllers from Adaptec, Highpoint and 3ware. Unfortunately, they are kind of expensive since they had hardware RAID on board which I did not need. So for the first time in a long time, I went back empty handed.

Back home, I started to look online for a cheap SATA controller. Searched ebay and found a cheap Silicon Image 3114 which again appears to work with Linux. So I ordered it from Hong Kong and it was far cheaper than the 3124 that I wanted. The 3124 was going for $150 while the 3114 was about $35 with postage. But the only thing is that I had to wait for it. Well as least it was cheap.

In the meantime, I decided to just max out the current SATA ports that I have. I acquired a second hand WD 36GB Raptor and I wanted to use it for the root mount point. So that is one port down. There were 3 500GB WD which I wanted configured as a software RAID5. That leaves 2 ports (I have a ICH9R with 6 ports). So 2 1TB drives it is to house the rest of the data while I construct the raid and start putting the important stuff on it.

So off I went to Sim Lim again and again the Samsung F1 1TB drives which were deemed to be very good were out of stock. So I had to settle for Seagate 1TB drives. I also got 2 2GB RAM modules since I wanted to do a lot of virtualization to separate the production and development environments.

Back home, I got into trouble also right away. The RAM modules were defective. memtest was complaining like crazy. The installation with a fresh install of 64bit Ubuntu 8.04 went fine but on reboot, GRUB went nuts. Well, I still could start up the system using the LiveCD, so I thought that I fixed the GRUB issue after I have the hard drives setup in the manner that I wanted.

Then I started on the copy process which takes a while since there were like 1.7TB to move around. A couple of months back, I noticed some data corruption issues so I decided to generate MD5 checksums and check all the files. I got the shock of my life when I found significant amounts of data corruption. Initially, I thought it was some silent data corruption issues but after a couple of days of crazy troubleshooting including the purchase of a new mainboard, I discovered that the mainboard was defective. And we are talking about a system that could withstand a full 24 hours of mprime. Ok so the cpu and the memory were alright but certainly not the storage part. I think there is some problems with the ICH9R southbridge. I should have expected it since the mainboard had a weird startup sequence where it would power up and then down and then finally up again.

Oh, well, at least the problem was resolved. The new mainboard was a ASUS P5Q-Pro with 8 SATA ports, 6 by ICH10R and 2 by a Silicon Image 5723. Ubuntu 8.04 did not recognize the Silicon Image 5723 and I could not see anything in ”lspci” for it. Strange, perhaps there were some options not been set in the BIOS. I will work on it later. But finally everything is fine now. The new configuration is as follows:

– 1 WD 36GB Raptor WD360ADFD-00NLR1 – root partition
– 3 WD 500GB SATA WD5000AAKS-65YGA0 – RAID5
– 2 Seagate 1TB SATA ST31000340AS – Storage
– 1 LG DVD writer HL-DT-ST DVD-RAM GSA-H55N – DVD drive

There is 3TB of storage with 1TB in RAID5. I moved all the PATA drives to the E8400 machine and copied the data back to the server. So the windows machine now has 4 PATA hard drives and 2 SATA hard drives. This is the same number of drives as the server but of much lower capacity. I plan to use this as a secondary backup storage. So I have less than 50% space left on the server.

The Sil3114 was delivered in a couple of days time, which I thought was pretty quick. Maybe I should buy online next time. And it worked perfectly well in Ubuntu 8.04 so that was good news. So I now have 4 extra SATA or 6 if I can get the Sil5723 to work.