Last Wednesday started like most days, tackling the project list. While multi-tasking I began putting final touches on my documentation for 3rd Party integration with Zenoss. At the same time I was testing the new web server I recently built which would serve as a replacement for our current server. It was just about noon when I needed to test reboot functionality and I issued a reboot command to our current web server (accidently). I figured it would take 5 or so minutes to reboot, so I waited around and set a “ping -t www.plixer.com”…

The server wasn’t coming up…

This was about the time my heart started racing. The corporate site was down and I didn’t know why. I grabbed my cell phone and numbers for the NOC at the Time Warner Data Center, where our server is hosted. Suddenly remembering that I dropped my Jeep off that morning to have new speakers installed. I borrow a friend’s car and drive up to Time Warner. By the time I get there 45 minutes have passed and I have a bad feeling things aren’t going to go smoothly.The stress builds.

I hook the server up to a KVM and sure enough, my server has crashed with a fatal kernel error. The server wasn’t going back online any time soon. Luckily I have a backup server, for just this purpose, with most of the websites up and running. I call in a quick IP change and www.plixer.com is back up. This blog, however, was running on a slightly outdated backup and needs updating immediately. I race back to the office with my old dead server in hand. There is a kernel error and the server just won’t boot. Since our websites are up, I just need to pull the latest copy off of the dead server. This is where the live linux distro comes into play.

I have always walked around with a live CD in my bag whether it’s Knoppix, Ubuntu, or openSuSE. In the past I’ve used a live CD to save files from a corrupt windows servers and workstations, but never a Linux server. This was the first time, and I stumbled my way through it. With the live distro I was able to mount the disk, tar up needed directories, and sftp them up to the new server. Phew! All is well.

Since I struggled through this I might as well lay out exactly what I did to get the job done in high hopes that it will help someone else down the road.

First of all you need a live CD. Download a live distro and burn to CD; in this example I used openSuSE 11.0 Live CD.

Boot the server to the live CD. In my experience it will automatically pick up an IP address from our DHCP server, this is great!

I start looking around to see if the hard disk mounted automatically, it did not. I try a few commands to mount sda, sda1, hda, etc… they all fail. I don’t know the name of the physical disk to mount. In an attempt to find out what the physical disk is called, I launch Yast and click on Hardware.

Now click on Hardware Information to find disk information, it will scan your server looking for all installed hardware. Once the inventory is returned, expand out Disk and you should see any physical disks you have. Found it, /dev/sda2

Now I need to mount the drive. To do this, launch xterm and run

su root

This will give you privilege to mount the drive. Now run:

mount /dev/sda2 /mnt

This command mounts the physical disk to /mnt.

At this point you can cd to /mnt and run ls to see your file system.

Now that the physical disk has been mounted and access is granted you will need to tar up any directories you want to save. It is very useful to tar a directory because all files and subdirectories  will tar recursively unless you specify otherwise. The nice part about tarring a directory is that it is untarred as a directory rather than as individual files.

I wanted to grab the latest copy of our blog. To tar up the blog directory I ran:

tar cvzf blog.tgz www.plixer.com/blog/

use sftp to upload or move your tar file and to extract the contents of your tarball run:

tar xvzf blog.tgz

With the site extracted, everything is back up and running. My live linux distro saves the day again.

Thomas

Thomas Pore is the Director of IT and Field Engineering at Plixer. He developed and leads, the Malware Incident Response and Advanced NetFlow Training programs which are being offered in cities across the USA. He is also an adjunct professor at the local community college and teaches ethical hacking. Thomas travels the globe meeting with customers and trying improve the Scrutinizer network incident response system. He helps clients optimize threat detection strategies and aids in the configuration of custom incident response solutions. He has a Bachelor of Science in Computer Science from Dickinson College.

Related

Leave a Reply