On data loss

This week I’ve had a couple of data loss stories.

I’m applying for promotion which requires creating a CDR  (believe it or not) containing evidence of all the service, scholarly work, and librarianship work I’ve accomplished over the 8 years. I submitted the CDR, which also contains my teaching portfolio, CV, and self-evaluation for review. I thought it would be a good idea to backup my work.

At my institution, we have a shared network drive for all faculty for backup purposes. As I was copying over my work to this shared drive, my destination partition filled. This stopped the data transfer, and at the same time erased most of my source files. I didn’t discover this for a few days because I usually let file transfers happen in the background as I’m doing other work. I found a backup – although a few months old – that I could reconstruct from. Maybe at the end of the semester I could get a copy from my file, but my department chair doesn’t know if that’s possible.

My other story is a bit more interest. I migrated our library’s website onto an Amazon instance. One of the things I really wanted to change from our server to this new one is updating WordPress from the browser without using FTP. This includes the core, plugins, and themes. This can be done by adding( ‘FS_METHOD’, ‘define’ ); to the wp-config.php which forces a direct IO exchange using PHP.

This also requires messing with permissions using the chmod command on the commandline. As I was changing the subdirectories of a few files, I accidently typed sudo chmod 644 /*

This basically changed the permissions for the whole file structure of the instance. I didn’t catch it until I noticed that I had denied myself permission from much of the OS on the new server. In a frantic effort, I rebooted the instance thinking that this would fix the permissions problems. Instead, this caused a kernal panic and the system couldn’t even boot. It was dead in the water.

After talking to a friend (Thanks Adam!) who has experience working with AWS instances, he suggested I take a snapshot backup of the inaccessible OS. That way, my data could still be accessible. He also suggested to start a similar built instance and try to access the snapshot that way.

Basically we have 2 AWS instances: broken and fixed. Take a snapshot of the broken instance and turn that into a EBS volume. Then, use the fixed instance to access the EBS volume. Detailed instructions on how to access a EBS volume can be found here.

Surprisingly, this worked. I was able to access a file structure whose whole permission scheme had been changed accidentally. I think this is the real power of cloud computing, to generate machines quickly and easy, for data recovery.

Moral of the story: make backups. Sometimes your backups can crash, but if you make consistent backups that won’t be a problem. Also, make multiple backups!

My next project is to install Apache Solr on a micro instance to drive our site’s search engine. Fun!

Leave a Reply

Your email address will not be published. Required fields are marked *

Connect with Facebook