The Refined Geek

Long time readers will know how much this blog has struggled with its various incarnations over the past 4 years. Initially I ran it from home on a server that I was using for development purposes so it ran inside a virtual machine that contained not one, but two database engines (MS-SQL for development and MySQL for the blog) all behind the tenuous 1.5Mbit upstream connection. This held up ok until I wanted to do anything fancy like put pictures on there (which would kill the connection for anything over 50kb) and it was relatively unstable, going down for days at a time since I couldn’t get a reliable remote connection to it. Since then I’ve churned my way through different virtual private servers (and all the issues they have) before landing on my current Burst.NET Ubuntu box which has been the best of the bunch so far.

Well, on the surface at least.

Since my blog as attained a steady amount of traffic it usually doesn’t take long for someone to pipe up when it goes down, especially if it happens during the day time in Australia. Since I now have remote access to the server I’m one command away from rebooting it should anything happen to it and have done so multiple times when it has come to my attention. However there’s a good 12 or so hours during the day when I’m not really paying attention to the blog due to being at home and/or sleep and downtime during this period usually goes unnoticed until I try to login during the morning. Since a good chunk of my audience is in the USA this can mean an awful amount of missed traffic which isn’t the greatest way to start the day.

Now when I first set up the blog on this host there were a couple teething issues (mostly due to my rusty Linux skills) but for probably 2 months afterwards everything ran without the slightest indication of an issue. Then every so often the blog would simply stop responding, the server would be up and everything else on it was running fine but try as I might I couldn’t get it to serve out a PHP page. Wanting to get it back up as quickly as I could I recycled the Apache service and it came back up instantly and I figured it was just some transient error and went back to my everyday blogging routine. However it kept happening, usually at the most inopportune times, and so last weekend I sat down to find the root cause of the issue.

Turns out its WordPress itself.

The above screenshot shows the error pretty quickly, essentially Apache has reached the maximum number of clients it can serve and will start to reject users after that point. Whilst the causes of this are wide and varied it can usually the culprit can usually be traced down to some WordPress plugin or script that’s opening up connections and then not closing them properly. The best way to take care of this is to fix the script in question but since I have little interest in diving into the mess that is PHP I’ve simply upped the MaxClients setting, reduced the time out period and scheduled an Apache reboot to clear out anything that gets stuck open. All of these combined seems to be an effective solution to this issue in the mean time and once I feel up to the task of delving through all the code to find the offending script I’ll nip it in the bud for good.

Apart from that little quirk though this iteration of the blog’s underlying infrastructure has been pretty fantastic with all the plugins functioning the way I expect them to without me having to fiddle with web.config settings for hours on end. It’s also significantly faster as well, reducing page load times by half for dynamic pages and becoming near instant when its served from cache. You could attribute this to the fact that it’s a lot beefier than its predecessor but neither of them showed significant load for an extended period of time. I guess where I’m going with this is that if you’re going to host your own WordPress blog it’s just plain better on Linux, especially if you’ve better things to be doing (like, you know, blogging).

The defacto platform of choice for any gamer used to be the Microsoft Windows based PC however the last decade has seen that change to be some form of console. Today, whilst we’re seeing something of a resurgence in the PC market thanks in part to some good releases this year and ageing console hardware, PCs are somewhere on the order take about 5% of the video game market. If we then extrapolate from there using the fact that only about 1~2% of the PC market is Linux (although this number could be higher if restricted to gamers) then you can see why many companies have ignored it for so long, it just doesn’t make financial sense to get into it. However there’s been a few recent announcements that shows there’s an increasing amount of attention being paid to this ultra-niche and that makes for some interesting speculation.

Gaming on Linux has always been an exercise in frustration, usually due to the Windows-centric nature of the gaming industry. Back in the day Linux suffered from a lack of good driver support for modern graphics cards and this made it nearly impossible to get games running on there at an acceptable level. Once that was sorted out (whether you count binary blobs as “sorted” is up to you) there was still the issue that most games were simply not coded for Linux leaving their users with very few options. Many chose to run their games through WINE or Cedega which actually works quite well, especially for popular titles, but many where still left wanting  for titles that would run natively. The Humble Indie Bundle has gone a long way to getting developers working on Linux but it’s still something of a poor cousin to the Windows Platform.

Late last year saw Valve open up beta access to Steam on Linux bringing with it some 50 odd titles to the platform. It came as little surprise that they did this considering that they did the same thing with OSX just over 2 years ago which was undoubtedly a success for them. I haven’t really heard much on it since then, mostly because none of my gamer friends run Linux, but there’s evidence to suggest that it’s going pretty well as Valve is making further bets on Linux. As it turns out their upcoming Steam Box will be running some form of Linux under the hood:

Valve’s engineer talked about their labs and that they want to change the “frustrating lack of innovation in the area of computer hardware”. He also mentioned a console launch in 2013 and that it will specifically use Linux and not Windows. Furthermore he said that Valve’s labs will reveal yet another new hardware in 2013, most likely rumored controllers and VR equipment but we can expect some new exciting stuff.

I’ll be honest and say that I really didn’t expect this even with all the bellyaching people have been doing about Windows 8. You see whilst being able to brag about 55 titles being on the platform already that’s only 2% of their current catalogue. You could argue that emulation is good enough now that all the titles could be made available through the use of WINE which is a possibility but Valve doesn’t offer that option with OSX currently so it’s unlikely to happen. Realistically unless the current developers have intentions to do a Linux release now the release of the Steam Box/Steam on Linux isn’t going to be enough to tempt them to do it, especially if they’ve already recovered their costs from PC sales.

That being said all it might take is one industry heavyweight to put their weight behind Linux to start a cascade of others doing the same. As it turns out Blizzard is doing just that with one of their titles slated for a Linux release some time this year. Blizzard has a long history with cross platform releases as they were one of the few companies to do releases for Mac OS decades ago and they’ve stated many times that they have a Linux World of Warcraft client that they’ve shied away from releasing due to support concerns. Releasing an official client for one of their games on Linux will be their way of verifying whether its worth it for them to continue doing so and should it prove successful it could be the shot in the arm that Linux needs to become a viable platform for games developers to target.

Does this mean that I’ll be switching over? Probably not as I’m a Microsoft guy at heart and I know my current platform too well to just drop it for something else (even though I do have a lot of experience with Linux). I’m very interested to see how the Steam Box is going to be positioned as it being Linux changes the idea I had in my head for it and makes Valve’s previous comments about them all the more intriguing. Whilst 2013 might not be a blockbuster year for Linux gaming it is shaping up to be the turning point where it starts to become viable.

Just going to make a quick post housekeeping post today as there’s a couple things I want to update you guys on. If you’re a dedicated LifeHacker reader you may have noticed that my ugly mug graced the front page for a while yesterday   and yes it’s true I’ll be covering TechEd 2012 Australia for them. It’s an incredible opportunity and I’m very excited to be doing it so for most of next week I’ll probably be recapping my day on here with all the real writing appearing on LifeHacker’s site. The posts on here probably won’t be at their usual time however so if you’re looking for your regular lunch time-ish article I’m going to have to disappoint you for a while.

I’m in the middle of migrating this blog over from my old Windows VPS that’s served me well over the past couple years to a Linux VPS with a ton more capacity. I tried to make the move last night but after getting everything up and running everything seemed to go pear shaped and nothing but index.php was being served by Apache so I trashed it all and started again this morning. I’m hopeful that this migration will go along smoothly but if things disappear it’s mostly because the two databases weren’t completely in sync at the time. This post was written on the old server and will likely disappear when the real migration occurs. Once that happens though I’ll know everything has worked and I’ll be working to get everything back up again.

Also, if you’ll allow me to get a little sappy for a second, I want to give you my heartfelt thanks for reading my tripe for the past 4 years as that was what motivated me to enter the LifeHacker competition in the first place. I didn’t start off as a great writer (as I’ve been told several times in no uncertain terms) but the feedback, comments and pageviews you guys gave me were enough incentive to keep on writing and improving my craft to a point where I felt confident enough to attempt something like this. That being said the true test is going to be how well the wider public receives my writing which is making me both excited and extremely nervous at the same time. Still I have no doubt it’s going to be great and I really do feel that all of you helped me get there in some way.

Now back to configuring Apache…

My first interaction with Steam wasn’t a pleasant one. I remember the day clearly, I was still living out in Wamboin when Valve released Half Life 2 and had made sure to grab myself a copy before heading home. After going through the lengthy install process requiring multiple CD swaps I was greeted by a login box asking me to create an account. Frustratingly all my usual gamer tags: PYROMANT|C, SuperDave, Nalafang, etc. were already taken leaving me to choose   a random name. That wasn’t the real annoyance though, no what got me was the required update that needed to be applied before I could play it which, on the end of a 56k connection, was going to take me the better part of an hour to apply.

This soured me on the idea of Steam for quite a few years, at least until I got myself a stable form of broadband that let me update without having to wait hours at a time. Still it wasn’t until probably 3 years or so ago that I started buying most of my games through Steam as buying the physical media and then integrating with Steam later was still a much better experience. Today though it’s my platform of choice when purchasing games and it seems that I’m not alone in this regard with up to 70% of all digital sales passing through the platform. We’ve also seen Steam add many more features like SteamCloud and SteamWorks which have provided a platform for developers to add features that would have otherwise been too costly to develop themselves.

With all the success that Steam has enjoyed (in the process making Valve one of the most profitable companies per employee) it makes you wonder what the end game for Steam will end up being. Whilst they’d undoubtedly be able to coast along quite easily on the recurring sales and the giant community they’ve built around the platform history has shown that Valve isn’t that kind of company. Indeed the recent press release from Valve saying that traditional applications will soon be available through the Steam platform seems to indicate that they have ambitions that extend past their roots of gaming and digital distribution.

And its at this point that I start speculating wildly.

Valve has shown that it is dedicated to gamers regardless of the platform with Steam already on OSX and will soon be finding its way onto Linux alongside a native port of Left 4 Dead 2. With such a deep knowledge of games and an engine that runs on nearly any platform it would make sense that Valve might take a stab at cutting out the middle man entirely, choosing to create their own custom operating system that’s solely dedicated to the purpose of gaming. If such an idea was to come to fruition it would most likely be some kind of Linux derivative with a whole bunch of optimizations in it to make Source titles run better. I’ll be honest with you when this idea was suggested to me I thought it was pretty far out but there are some threads within this idea that have some merit.

Whilst the idea of SteamOS as a standalone operating system might be a bit far fetched I could see something akin to media centre software that transforms a traditional Windows/Linux/OSX PC into a dedicated gaming machine. Steam’s strength arguably comes from the giant catalogue of third party titles that they have on there and keeping the underlying OS (with its APIs in tact) means that all these games would still be available. This also seems to line up with the rumoured SteamBox idea that was floating around at the start of the year and would mean that the console was in fact just a re-badged Windows PC with some custom hardware underneath. The console itself might not catch on (although the success of the OUYA seems to indicate otherwise) but I could very well see people installing SteamOS beside their XBMC installation turning their Media PC into a dual use machine.

With all this in mind you have to then ask yourself what Valve would get out of something like this. They are already making headway into getting Steam in one form or another onto already existing consoles (see Steam for the PS3) and they’ve arguably already captured the lion’s share of PC gamers, the ones who’d be most likely to use something like SteamOS. The SteamBox would arguably be targeted at people who are not traditionally PC gamers and SteamOS then would simply be an also ran, something that would provide extra value to its already dedicated PC community. Essentially it would be further cementing Steam as the preferred digital distribution network for games whilst also attempting to capture a market that they’ve had little to do with up until this point.

All of this though is based on the current direction Valve seems to be going but realistically I could just be reading way too far into it. Their recent moves with the Steam platform are arguably just Valve trying to grow their platform organically and could very easily not be part of some grander scheme for greater platform dominance. The idea though is intriguing and whilst I have nothing more than speculation to go on I don’t think it would be a bad move by Valve at all.

With the release of ESXi 5.0 VMware permanently did away with one of the core components of its flagship product: the service console. Owing to its roots in Linux the original versions of ESX were in fact modified versions of Red Hat Enterprise Linux and had all the trimmings of a full Linux host. Of course modifying anything in there was done at your own peril but for the most part you could treat it just like a Linux box when there was a challenge you needed to overcome. However the Service Console was also the most patched aspect of ESX and thus moving away from it meant a more secure hypervisor, at the cost of removing some flexibility.

Of course some of the functionality remains but its far from what it used to be. Instead of being a fully fledged Linux box under the hood it’s now a custom BusyBox implementation giving you a somewhat familiar environment but with all the cruft reduced to a minimum. When ESXi was first introduced I was quite sceptical as I spent a good deal of my time in the Service Console when troubleshooting virtual environments but since I’ve spent the last 6 months developing an ESXi 5.0 standard operating environment I’ve grown to appreciate the simplicity, mostly because the tools to manage ESXi outside the service console have matured considerably.

One thing it doesn’t do well however is remote access. Back in the ESX days it was a pretty simple matter of creating a new user and giving them shell access and there was nothing more that needed to be done. Whilst it’s trivial to re-enable SSH you’ll soon find out that whilst you’re able to access it with the built in root account you can’t access it with any other account, even if you’ve granted them shell access (the complete opposite to how it has been in the past). For my SOE this isn’t acceptable and so I’ve been developing a solution to the problem.

Engaging my Goolge-Fu I quickly stumbled across two posts that detailed a process of how to do this on ESXi 4.1. Now many of the problems in 4.1 are shared with 5.0 so I figured that these would probably work for me. Upon testing them however neither of them work meaning that there’s been some changes to the way that ESXi does its SSH authentication in 5.0. Thankfully those posts, coupled with some insight from work mates, pointed me to the right solution.

One of the solutions we found that worked for us was modifying /etc/security/access.conf to have the user in it that you wanted. That ends up looking something like this:

+:dcui:ALL
+:root:ALL
+:vpxuser:ALL
+:vslauser:ALL
+:UserToGiveSSHTo:ALL
-:ALL:ALL

That works, right up until you reboot. That particular file isn’t blessed with the sticky bit that ESXi puts on certain files so any changes to it are lost upon reboot. Additionally that appears to only work for a single user which would make adding additional users something of a bother. As it turns out both these problems are rather easily solved.

The access.conf file also works with groups, although this isn’t documented anywhere. So for the example file above if you just use a group name instead of a single user name all users contained in that group will get shell access, so long as the user was granted that when it was created. To get around the rebooting problem you can edit the rc.local file (which does have the sticky bit on it) which is a script file that runs after ESXi is done loading, allowing you to run scripts at start up. All that you need to do then is sed in the permission line for the group/user you want to enable access for. An example rc.local file is below:

#!/bin/sh

export PATH=/sbin:/bin

log() {
echo "${1}"
/bin/busybox logger init "${1}"
}

# execute all service retgistered in ${rcdir} ($1 or /etc/rc.local.d)
if [ -d "${1:-/etc/rc.local.d}" ] ; then
for filename in $(find "${1:-/etc/rc.local.d}" | /bin/busybox sort) ; do
if [ -f "${filename}" ] && [ -x "${filename}" ]; then
log "running ${filename}"
"${filename}"
fi
done
fi

sleep 4
sed -i '$ i\+:GroupOrUserThatNeedsAccess:ALL' /etc/security/access.conf

I’ve tested this and it works on one of my freshly minted ESXi 5.0 boxes.

One of our concerns with this was the potential security implications of enabling SSH in this way. ESXi uses PAM for authentication and what the above modifications to access.conf does is tell PAM to authorize that account for any check that invokes it, including SSH authentication. Now that sounds bad since it seems like that would be akin to granting those users administrator rights (indeed that is the supported way of giving users SSH access to ESXi hosts) but ESXi only uses PAM for authentication, not authorization. This means that whilst they’ve got wide open access when it comes to PAM ESXi uses its own in built permissions to control user access. So this means you can have restricted accounts that are allowed to login via SSH and then su to root if they need more permissions, which is still supported in BusyBox.

So this is probably as close as you’ll ever get to having the same level of control that you have in your current ESX environments when you make the switch to ESXi. Personally I think it’s pretty elegant and its very maintainable unlike many of the other solutions I tried before reaching this one. If you have any questions or comments feel free to hit me up in the comments, on twitter or send me an email.

This blog has had a variety of homes over the past few years although you wouldn’t know it by looking at it. Initially it was hosted on a Windows 2008 server I built myself, sitting behind the tenuous link of my ADSL connection. Don’t get me wrong this is a great way to get started if you’ve got admin roots like me but inevitably my ADSL connection would go down or people would just plain give up waiting for it to load, what with my upstream only able to handle 100KB/s. Still for most of its life the blog remained in that configuration as I couldn’t find a hosting provider I was happy with.

Of course the day came when WordPress decided to stop playing nice with IIS and started returning internal server 500 errors. Thankfully it would usually right itself after a reboot but it was always a count down to the time when it would start erroring out again and being the busy man that I am I never had the time to troubleshoot it. Eventually I caved and set up an Ubuntu box to host it, figuring that all my woes would be solved by switching to the platform that everyone expects WordPress to run on. I’ll be honest it was a good change as I could finally use all the caching plugins, and traffic took an upward trend thanks to the faster loading times.

Unfortunately that didn’t last particularly long either as whilst the blog was particularly zippy the Linux VM would sometimes stop responding to requests and would only start behaving itself after a reboot. The cause of this I’m still not sure of as the VM was still up but it just refused to keep on serving web pages, including all the funky admin tools my PHPMyAdmin and Webmin. It was around this time I found myself in possession of a shiny new VPS that was only hosting my fledgling app Lobaco so I figured a small time WordPress blog wouldn’t be too much for it to handle. Indeed it wasn’t and the blog has been steaming along on it ever since.

However the unfortunate internal server errors returned eventually and whilst I was able to get around them with the trusty old reboot a couple times they became more persistent until I eventually couldn’t get rid of them. After digging around in the event logs for a while I eventually stumbled across references to php_wincache.dll which upon googling lead me to posts like these, showing I wasn’t alone in this internal server error hell. Disabling the plugin fixed the problem and all was well with the world. Of course many months later I found myself  trying to optimize my blog again and I started looking at the things I had removed in order to keep this thing up and running.

The first was the caching plug-ins which are unequivocally the best thing for performance on a dynamic PHP site. The vast majority of WordPress caching plug-ins don’t play nice with Windows as they make the assumption they’re on Linux and attempt to write files in all sorts of whacky locations that simply don’t exist. WP-SuperCache, although still suffering from some Linux based assumptions, can be wrangled into working properly with IIS and has been doing so for the past couple months. I also found that WinCache had been updated since I had unceremoniously removed it from my php.ini file so I decided to give it another try. Again everything was rosy for a time, that was until last weekend.

I fired up my blog on Saturday to find the home page coming up fine but I was logged out for some reason. This happens from time to time so I wasn’t worried but trying to login left me with the dreaded internal server 500 error. Poking around it looked like any non-cached page was failing meaning the majority of my site was unavailable. The event logs showed the dreaded WinCache dll failing again and disabling it brought my website back around again. It seems, at least for now, that I’ll have to give WinCache a miss as the last update to it was almost 3 months ago and its past performance has led me to believe that it’s not entirely stable.

So if you’re crazy like me, trying to run WordPress on IIS and all, and you’re WordPress blog seems to take a dive more often than not make sure to get rid of WinCache at least until they get their act together. I haven’t delved into my previous VMs to see if it was the culprit back then but my most recent set of problems can be traced directly back to WinCache wrecking havoc by attempting to cache PHP objects and if this post can save 1 person the headache of trying to track it down I’ll consider it a huge success. 

I’ve spent the last 4 days lost in a technical limbo. On Saturday I was making good progress with Geon and left it to spend the night out with a bunch of my great friends and to see Inception (well worth seeing by the way). However Sunday morning saw me stuck on what seemed to be the most simple of problems, wasting at least 5 hours trying to get past this point. To make matters worse my web server decided to give up the ghost and just stop serving out PHP pages and neither upgrades, reboots or the hours of troubleshooting I threw at it would bring it back to life. Since you’re reading this now you know that I’ve managed to fix it, but it wasn’t an easy journey.

You see I was running this blog from IIS 7.0 which is Microsoft’s web server. For the most part it’s pretty good, the administration tools are top notch and it only took me a couple weeks to get my head around the whole idea of hosting a web site. Before starting up this site I’d never really tried any web stuff, preferring to stick to my cosy little offline world. Still I knew when starting this blog that I was using the least preferred web server on the market and the tutorials I found for getting it all started seemed more complicated than they needed to be. But I had been given a free copy of Windows Server 2008 and I didn’t really feel like re-educating myself with Linux, which had burned me in the past.

For a long time everything was good, the site was always up (pending my Internet connection of course) and seemed to be quite responsive. Over the course of almost 2 years though I’ve added quite a lot of things to the site like photo albums, sexy comments and all manner of behind the scenes stuff. About 6 weeks ago though I started getting PHP timeouts on this site, but all the others were running fine. The weekend just past saw all of the sites die a similar death and nothing I could do seemed to revive them. I was then faced with a choice, either move this site to one of my external providers or rebuild the server completely. After considering my options I went for the rebuild and 2 days later I’m back online serving this page to you from an Apache web server running on a freshly minted Ubuntu server.

Now I’m no stranger to the world of Linux. I spent quite a good chunk of my University years programming in Linux and my time at the National Archives of Australia had me administering a couple Linux servers that were powering the Digital Preservation Project there. Still I shyed away from it mostly because it was always a pain to administer when compared to its Window’s cousins which I always able to get functioning after 1 or 2 Google searches. To its credit Ubuntu made this process far less painful that it had been in the past and the tools have matured to a point when even a Windows administrator shouldn’t feel out of place. Last night saw this website resurrected in all its glory and hopefully I’ll figure out how to get my other hosted sites up and running sometime soon.

So you might be wondering: what happened to your old server? Well it’s still there, sitting alongside it’s new Ubuntu brother on my ESXi box and it won’t be going anywhere for quite a long time. You see, as great as Apache is, it’s still not IIS and with all my code relying on many things that IIS provides I’ll still need it in some form for quite a while to come. Additionally I had it set up as my Microsoft SQL box, again for development, and whilst I could use MySQL or PostgreSQL to achieve the same end the integration with the Microsoft tools just isn’t there yet. Plus I get it for free because of my Technet subscription so the cost factor that usually accompanies it is a non-issue.

Hopefully the move to the new server will mean the site will load faster for you, won’t cause me anymore administrative headaches and will reacquaint me with the Linux world I once knew so well. It’s a load off my mind to finally see this site back up and running again, especially after 4 days of feeling like I couldn’t make progress in anything technical. We’ll be back to our regularly scheduled programming tomorrow and I hope to see you guys back here then