Posts Tagged‘black magic’

Unreasonable Expectations and Arbitrary Boundaries.

There’s an old saying that goes something along the lines of “The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man“. As someone who had lived much of his life trying to learn the rules of the world so I could work within them the notion that being unreasonable about something would be the catalyst for progress was initially met with harsh scepticism. However I began to notice that the ones who actually managed to enact change were in fact those who were making unreasonable demands, not just of others but also of themselves. They also seemed to flourish within boundaries, seeming to be far more capable working with some kind artificial constraint than they were with completely open ended problems.

I really started to believe in this whole progress comes from being unreasonable idea when I started working on my own projects and I started running up against things that people had never come across before. Now for us .NET developers it’s pretty much guaranteed that you’re not the first one to run up against a certain problem since there are so many people out there developing with it. However following on from that idea you’ll tend to find then that if people can’t find a solution to particular issue they’ll instead find some other way of achieving the same goal that’s been done before. This is the double edged sword of Microsoft’s black magic and it definitely traps the wise programmers in the loop of adapting themselves instead of trying to make changes to the world they’re operating in.

I had this recently when I was working on my latest project. I was working with an ASP.NET MVC 3 site and I had set up the web site to make multiple calls back to the server in order to retrieve the data it needed. Now this worked well to get it off the ground but anyone looking to optimize a website will tell you reducing the number of calls to your server will lead to much better performance, for both the client and your server. Eliminating all these calls and wrapping them up into the @Model of the view for this part of the website would do that, but I had no idea of how to get the same results as I had done with the multiple requests. After searching, hacking and testing several different things I eventually found myself with a very workable solution and I was left for many more ideas for improvements to the site.

Now had I been more reasonable with my expectations I would’ve instead just kept on doing what I had been doing (since it was functional) and wouldn’t have dared to consider changing. Indeed I sat on the whole idea for a day before pulling the trigger on it, precisely because of the amount of rework that was involved in doing so. However the changes I made will make it far more easier going forward since it allows me to work in the areas I’m much more comfortable with rather than fooling around with technologies I’m still in the process of understanding. Sure going the other way might have been a better learning experience but I’ve learnt quite a lot in the process of overcoming the goal I set myself, perhaps more than I would have should I have continued down the same path.

Having unreasonable expectations frees you from being constrained by perceived limitations, allowing you to push to the very limits of what is possible. Arbitrary boundaries help to limit the problem space considerably enabling you to focus more clearly on the ultimate goal rather than getting stuck in the multitude of minutia. Combining these two ideas has helped me push past my own limitations in many aspects of my life, from coding to fitness and even to unlocking my hidden creative self. So I put it to you to start being more unreasonable in your expectations and using arbitrary limitations as enablers rather than blockers to progress.

Microsoft’s Blackmagic: A Double Edged Sword.

I’m a really big fan of Microsoft’s development tools. No other IDE that I’ve used to date can hold a candle to the mighty Visual Studio, especially when you couple it with things like ReSharper and the massive online communities dedicated to overcoming any of the shortcomings that you might encounter along the way. The same communities are also responsible for developing many additional frameworks in order to extend the Microsoft platforms even further, with many of them making their way into official SDKs. There have only been a few times when I’ve found myself treading new ground with Microsoft tools which no one has before, but every time I have I’ve discovered so much more than I initially set out to.

I’ve come to call these encounters “black magic moments”.

You see with the ease of developing with a large range of solutions already laid out for you it becomes quite tempting to slip into the habit of seeking out a completed solution, rather than building one of your own. Indeed there were a few design decisions in my previous applications that were driven by this, mostly because I didn’t want to dive under the hood of those solutions to develop the fix for my particular problem. It’s quite surprising how far you can get into developing something by doing this but eventually the decisions you make will corner you into a place where you have to make a choice between doing some real development or scraping a ton of work. Microsoft’s development ideals seem to encourage the latter (in favor of using one of their tried and true solutions) but stubborn engineers like me hate having to do rework.

This of course means diving beneath the surface of Microsoft’s black boxes and poking around to get an idea of what the hell is going on. My first real attempt at this was back in the early days of the Lobaco code base when I had decided that everything should be done via JSON. Everything was working out quite well until I started trying to POST a JSON object to my webservice, where upon it would throw out all sorts of errors about not being able to de-serialize the object. I spent the better part of 2 days trying to figure that problem out and got precisely no where, eventually posting my frustrations to the Silverlight forums. Whilst I didn’t get the actual answer from there they did eventually lead me down a path that got me there, but the solution is not documented anywhere nor does it seem that anyone else has attempted such a feat before (or after for that matter).

I hit another Microsoft black magic moment when I was working on my latest project that I had decided would be entirely cloud based. After learning my way around the ins and outs of the Windows Azure platform I took it upon myself to migrate the default authentication system built into ASP.NET MVC 3 onto Microsoft’s cloud. Thanks to a couple handy tutorials the process of doing so seemed fairly easy so I set about my task, converting everything into the cloud. However upon attempting to use the thing I just created I was greeted with all sorts of random errors and no amount of massaging the code would set it straight. After the longest time I found that it came down to a nuance of the Azure Tables storage part of Windows Azure, namely the way it structures data.

In essence Azure Tables is one of them new fangled NOSQL type databases and as such it relies on a couple properties in your object class  to uniquely identify a row and provide scalability. These two properties are called PartitionKey and RowKey and whilst you can leave them alone and your app will still work it won’t be able to leverage any of the cloud goodness. So in my implementation I had overridden these variables in order to get the scalability that I wanted but had neglected to include any setters for them. This didn’t seem to be a problem when storing objects in Azure Tables but when querying them it seems that Azure requires the setters to be there, even if they do nothing at all. Adding one in fixed nearly every problem I was encountering and brought me back to another problem I had faced in the past (more on that when I finally fix it!).

Like any mature framework that does a lot of the heavy lifting for you Microsoft’s solutions suffer when you start to tread unknown territory. Realistically though this is should be expected and I’ve found I spend the vast majority of my time on less than 20% of the code that ends up making the final solution. The upshot is of course that once these barriers are down progress accelerates at an extremely rapid pace, as I saw with both the Silverlight and iPhone clients for Lobaco. My cloud authentication services are nearly ready for prime time and since I struggled so much with this I’ll be open sourcing my solution so that others can benefit from the numerous hours I spent on this problem. It will be my first ever attempt at open sourcing something that I created and the prospect both thrills and scares me, but I’m looking forward to giving back a little to the communities that have given me so much.