Posts Tagged‘site’

How Everything Went To Shit (or No Admin is Immune to Being Stupid).

This blog has had a pretty good run as far as data retention goes. I’ve been through probably a dozen different servers over its life and every time I’ve managed to maintain continuity of pretty much everything. It’s not because I kept rigorous backups or anything like that, no I was just good at making sure I had all my data moved over and working before I deleted the old one. Sure there’s various bits of data scattered among my hard drives but none of it is readily usable so should the unthinkable happen I was up the proverbial creek without a paddle.

And, of course, late on Saturday night, the unthinkable happened.

Picard FacepalmSo I logged into my blog to check out how everything was going (as I usually do) and noticed that something strange was appearing in my header. It appeared to be some kind of mass mailer although it wasn’t being pulled in from a JavaScript file or anything and, to my surprise, it was embedding itself everywhere, even on the admin panel. Now I’ve never been compromised before, although people have tried, so this sent me into something of a panic and I started Googling my heart out to find out where this damn code was coming from. Try as I might however I couldn’t find the source of it (nothing in the Apache configuration, all WordPress files were uncompromised, other sites I’m hosting weren’t affected) and I resigned myself to rebuild the server and to start anew. Annoying, but nothing I haven’t done before.

Like a good little admin I thought it would be good to do a cleanup of the directory before I embarked on this as I was going to have to move the backup file to my desktop, no small feat considering it was some 1.9GB big and I’m on Australian Internet (thanks Abbott!). I had a previous backup file there which I moved to my /var/www directory to make sure I could download it (I could) and so I looked to cleaning everything else up. I’ve had a couple legacy directories in there for a while and so I decided to remove them. This would have been fine except I fat fingered the command and typed rm -r which happily went about its business deleting the entire folder contents. The next ls I ran sent me into a fit of rage as I struggled to figure out what to do next.

If this was a Windows box it would’ve been a minor inconvenience as I’d just fire up Recuva (if CTRL + Z didn’t work) and get all the files restore however in Linux restoring deleted files seems to be a right pain in the ass. Try as I might extundelete couldn’t restore squat and every other application looked like it required a PhD to operate. The other option was to contact my VPS provider’s support to see if they could help out however since I’m not paying a terrible amount for the service I doubt it would been very expedient, nor would I have expected them to be able to recover anything.

In desperation I reached out to my old VPS provider to see if they still had a copy of my virtual machine. The service had only been cancelled a week ago and I know a lot of them keep copies for a little while just in case something like this happens, mostly because it’s a good source of revenue (I would’ve gladly paid $200 for it). However this morning the email came from them stating unequivocally that the files are gone and there’s no way to get them back, so I was left with very few options to get everything working again.

Thankfully I still had the database which contains much of the configuration information required to get this site back up and running so all that was required was to get the base WordPress install working and then reinstall all the necessary plugins. It was during this exercise that I stumbled across the potential attack vector that let whoever it was ruin my site in the first place: my permissions were all kinds of fucked, essentially allowing open slather to anyone who wanted it. Whilst I’ve since struggled to get everything working like it was before I now know that my permissions are far better than they were and hopefully should keep it from happening again.

As for the rest of the content I have about half of the images I’ve uploaded over the past 5 years in a source folder and, if I was so inclined, could reupload them. However I’ve decided to leave that for the moment as the free CDN that WordPress gives you as part of Jetpack has most of those images in it anyway which is why everything on the front page is working as it should. I may end up doing it anyway just as an exercise to flex my PowerShell skills but it’s no longer a critical issue.

So what has this whole experience taught me? Well mostly that I should practice what I preach as if a customer came running to me in this situation I’d have little sympathy for them and would likely spend maybe 20% of the total effort I’ve spent on this site to try and restore theirs. The unintentional purge has been somewhat good as I’ve dropped many of the plugins I no longer used which has made the site substantially leaner and I’ve moved from having my pants around my ankles, begging for attackers to take advantage of me, to at least holding them around my waist. I’ll also be implementing some kind of rudimentary backup solution so that if this happens again I at least have a point in time to restore to as this whole experience has been far too stressful for my liking and I’d rather not repeat it again.

 

Why I Dropped CloudFlare.

I’m always looking out for ways to improve my blog behind the scenes mostly because I’ve noticed that a lot more people visit when the page doesn’t take more than 10 seconds to load. Over the course of its life I’ve tried a myriad of things with the blog from changing operating systems to trying nearly every plugin under the sun that said it could boost my site’s performance. In the end the best move I ever made was to put it on a Windows virtual private server in the USA that was backed up by a massive pipe and everything I’ve tried hasn’t come close since.

However I was intrigued by the services offered by CloudFlare, a new web start up that offered to speed up basically any web site. I’d read about them a while back when they were participating in TechCrunch Disrupt and the idea of being able to back my blog with a CDN for free was something few would pass up. At the time however my blog was on a Linux server with all the caching plugins functioning fine, so my site was performing pretty much as fast as it could at the time. After the migration to my new Windows server however I had to disable my caching plugins as they assumed a Linux host for them to function properly. I didn’t really think about CloudFlare again until they came up in my feed reader just recently, so I decided to give them a go.

They’re not wrong when they say their set up is painless (at least for an IT geek like myself). After signing up with them and entering in my site details all that I needed to do was update my name servers to point to theirs and I was fully integrated with their service. At first I was a bit confused since it didn’t seem to be doing anything but proxying the connections to my site but it would seem that it does cache static content. How it goes about this doesn’t seem to be public knowledge however, so I got the feeling it only does it per request. Still after getting it all set up I decided I’d leave it over the weekend to see how it performed and come this morning I wasn’t terribly impressed with the results.

Whilst the main site suffered absolutely 0 downtime my 2 dozen sub domains seemed to have dropped off the face of the earth. Initially I had thought that this was because of the wildcard DNS entry that I had used to redirect all subdomain requests (CloudFlare says they won’t proxy them if you do this, which was fine for me in this instance). However after manually entering in the subdomains and waiting 24 hours to see the results they were still not accessible. Additionally the site load times didn’t improve noticeably, leaving me wondering if this was worth all the time I had put into it. After changing my name servers back to their previous locations all my sites came back up immediately and soured me on the whole CloudFlare idea.

It could be that it was all a massive configuration goof on my part but since I was able to restore my sites I’m leaning it towards being a problem with CloudFlare. For single site websites it’s probably a good tool and I’d be lying if I said I wasn’t interested in their DDOS protection (I was on edge after doing that LulzSec piece) but it seems my unique configuration doesn’t gel with their services. Don’t let me talk you out of trying them however since so many people seem to be benefiting from their services, it’s just that there might be potential problems if you’re running dozens of subdomains like me.

Notice Anything?

Have a look at the top of your web browser, notice anything different? If I’ve done everything correctly you should now be looking at this page from www.therefinedgeek.com.au and not my old address. Yes I decided to listen to my peers and buy the domain name. For now the DNS routing to this address is a bit hacky but that will all change come the 26th when I get my static IP address, so if the site is down temporarily it’s probably because my Internet disconnected and I had to manually update the host record, but I don’t see it staying down for long.

When I first built this site I was doing it mostly to get some exposure to web technologies, predominately Windows 2008 server and the goodies that it comes with. I was happy with a DynDNS account that would automatically route everyone to my website no matter what happened to my connection, but that all changed after one of my old friends contacted me.

Whilst getting a domain name was always on the table I had never really considered the potential benefit of getting one. Sure there’s the whole brand recognition stuff and the small amount of prestige from having a unique name on the web but what really got to me was how someone else could be making money off my work, without even having to do anything apart from hosting a DNS service. I guess he knew one of my weak points and wanted to help out; I get pretty motivated when I find out someone is making more from me then I think they should 😉

It’s also a natural progression from a site that started out as just a test bed for various web technologies but evolved into the creative outlet I use it for today. I’ve also never really worked with a proper domain name and if you were unfortunate enough to come across the site whilst I was getting everything right you would’ve been greeted by various levels of errors, funny looking pages and redirection loops. All part of the process, and it was a good hour of fooling around to get everything right.

So, update your bookmarks, RSS feeds and whatever else you may have this site flagged as. I’ll probably keep the old link up for a little while before turning it off, as I don’t want people relying on that one 🙂

Technology Integration Testing.

Just to see how this goes, I’ve created a horrible mess of web 2.0 applications so that several different websites will update themselves when I post on this blog.

I believe this is what those crazy web kids call “mash ups” these days, when really its just programs talking to each other. Or maybe I’m getting cynical in my old (HA!) age 🙂

Expect a few more of these kinds of posts if I find I’ve broken something.

EDIT:

Appears that I’ve made it work. All it took was an hour and few non G rated words yelled at my server to get it working 😉