What a long strange trip it's been
Sometime on Wednesday March 11, the server running this blog died a horrible death.
I'd been happily working on a consulting sub-contract when I returned home late at night to the realization that things weren't right. I couldn't get on the internet, but thought it was an ISP problem. In the morning, I discovered that it was the server, which besides running the blog and a few other web apps, was also running a dns server for local lan addresss, a dhcp server and other vital services here at denhaven2.
When I found time to investigate, I discovered that one of the disks was acting badly. I rebooted, and it took quite a while, but eventually I could snoop around, but when I tried rebooting again, it wouldn't.
My good friend, Mark Imbriacco, who is one of the two sys admins at 37signals, and who lives in the neighborhood, came by to help me diagnose things. The more we snooped around, the more it looked like it was actually the IDE controller on the mother board which had gone flakey.
Mark, bless his heart, offered up a machine he was no longer using, and the process of rebuilding began.
Luckily, I had put a graphics card in the old system, did I tell you that it wasn't just a server? I used to use it as a development workstation as well. This was lucky because Mark had used the new system as a gaming machine, and it had a nice gamer's graphics card installed, whose on board fan was hanging limply from it's attachment. I invested in a 500GM SATA drive for the install, I wanted to bring the system up before touching the drives from the old server.
I'd been holding off updating the system, which was running on Ubuntu 6.06 "Dapper Drake" for nearly 3 years. I'd fallen for the "Long Term Support" pitch, which had the effect of leaving me behind on things like Apache. So this seemed like the right time to take the step up. I decided to just use the latest Ubuntu 8.10 release.
Once the system was installed, it was time to examine the wreckage. It turned out that one of the two 160GB drives had gone bad, or had perhaps been corrupted by the controller. One of the two drives held the home directories, and various other directories, the other was used to hold a nice rsync backup set, kind of a home-built Time Machine using techniques I'd learned at the local Linux user's group. Right after the crash, I thought that it was actually the backup drive which had gone bad, since the first time I booted, I did seem to have access to the data on the 'live drive.' But now linux couldn't find the superblocks, and all I had was the backup.
Two minor problems. The first was that I discovered that the last good backup I had was from sometime in January, so I was going to lose some data. The second was that my nice backups was full of files whose permissions were based on the /etc/passwd and /etc/groups files on the OLD system. I'd faced this problem back when I'd moved the system from Redhat/Fedora to Ubuntu. Redhat distros tend to assign fixed uid's and gid's for the users associated with server processes, while Debian based systems assign them when packages are installed. Reconciling the differences is a major pain, and most of the data is still on the backup drive waiting for me to slog through it.
What I did have were good copies of my database backups (at least until sometime in January, and my local code, like the instance of Typo running this blog. I decided to reconstruct the blog on my MacBook, upgrade it from Typo 4.1 to the latest version of typo, and make it deployable on the server using capistrano, with git for the scm, and passenger, now that I had a version of apache that was modern enough to handle it. No more of the old rube-goldberg setup of Apache talking to pen for load balancing, talking to a mongrel cluster.
I haven't done so much concentrated system admin in quite a while. I've got to thank Mark, and my other friend Ben Burdick, who used to be the sysadmin and QA guru at Near-Time and now works for RackSpace/Slicehost, who have been subject to repeated iChat sessions, when my googling skills flagged.
Mark also had my blog posts for quite a while cached in his RSS reader, and provided me with the contents of the articles I'd written since the last backup. The few image files I needed were still on my MacBook. So I think I've been able to recover everything from the blog, except for any comments made on those missing posts.
Some of the things I've learned from this episode are:
- How to get the apache user to ssh back to the same server to access git.
- How to configure capistrano using git and passenger instead of subversion and mongrel/mongrel-cluster
- That running apache/passenger using the passenger pane on os x, is cool. It let me easily check out the blog. But, I had a final head-scratching moment. I'd changed /etc/hosts on the Mac so that talklikeaduck.denhaven2.com would resolve to local host, which worked great. But when I changed it to point to the lan address of the server, the Mac stubbornly refused to resolve it to anything but 127.0.0.1. I was chatting with Ben during this episode, and he told me to use
to clear out the Mac's dns cache, but this didn't work, even after a reboot . Ben just kept saying "That's wierd." I was thinking that some other configuration file must be causing this, so I tried:
grep -R talklikeaduck /etc
Which revealed that the only other place the word occurred was in the VHost configuration file which the Passenger pane had generated. I tried removing the blog entry from the Passenger pane and restarting apache, and lo and behold, after clearing the cache, the host was now resolving to the server!
You'll notice that the blog has a new style. The old theme wasn't compatible with Typo 5.2, and getting the blog back up was a higher priority than porting it. Actually I like the new theme, and although I kind of miss old Ruby the Duck Dog. I suspect that he'll she will be coming back in a new form when I get the time to do it.
So having written this first article after the great crash of 2009, I think I'm ready to once again expose the server to the inter-tubes.
Previous Topic
