Posts Tagged‘solution’

Microsoft’s Blackmagic: A Double Edged Sword.

I’m a really big fan of Microsoft’s development tools. No other IDE that I’ve used to date can hold a candle to the mighty Visual Studio, especially when you couple it with things like ReSharper and the massive online communities dedicated to overcoming any of the shortcomings that you might encounter along the way. The same communities are also responsible for developing many additional frameworks in order to extend the Microsoft platforms even further, with many of them making their way into official SDKs. There have only been a few times when I’ve found myself treading new ground with Microsoft tools which no one has before, but every time I have I’ve discovered so much more than I initially set out to.

I’ve come to call these encounters “black magic moments”.

You see with the ease of developing with a large range of solutions already laid out for you it becomes quite tempting to slip into the habit of seeking out a completed solution, rather than building one of your own. Indeed there were a few design decisions in my previous applications that were driven by this, mostly because I didn’t want to dive under the hood of those solutions to develop the fix for my particular problem. It’s quite surprising how far you can get into developing something by doing this but eventually the decisions you make will corner you into a place where you have to make a choice between doing some real development or scraping a ton of work. Microsoft’s development ideals seem to encourage the latter (in favor of using one of their tried and true solutions) but stubborn engineers like me hate having to do rework.

This of course means diving beneath the surface of Microsoft’s black boxes and poking around to get an idea of what the hell is going on. My first real attempt at this was back in the early days of the Lobaco code base when I had decided that everything should be done via JSON. Everything was working out quite well until I started trying to POST a JSON object to my webservice, where upon it would throw out all sorts of errors about not being able to de-serialize the object. I spent the better part of 2 days trying to figure that problem out and got precisely no where, eventually posting my frustrations to the Silverlight forums. Whilst I didn’t get the actual answer from there they did eventually lead me down a path that got me there, but the solution is not documented anywhere nor does it seem that anyone else has attempted such a feat before (or after for that matter).

I hit another Microsoft black magic moment when I was working on my latest project that I had decided would be entirely cloud based. After learning my way around the ins and outs of the Windows Azure platform I took it upon myself to migrate the default authentication system built into ASP.NET MVC 3 onto Microsoft’s cloud. Thanks to a couple handy tutorials the process of doing so seemed fairly easy so I set about my task, converting everything into the cloud. However upon attempting to use the thing I just created I was greeted with all sorts of random errors and no amount of massaging the code would set it straight. After the longest time I found that it came down to a nuance of the Azure Tables storage part of Windows Azure, namely the way it structures data.

In essence Azure Tables is one of them new fangled NOSQL type databases and as such it relies on a couple properties in your object class  to uniquely identify a row and provide scalability. These two properties are called PartitionKey and RowKey and whilst you can leave them alone and your app will still work it won’t be able to leverage any of the cloud goodness. So in my implementation I had overridden these variables in order to get the scalability that I wanted but had neglected to include any setters for them. This didn’t seem to be a problem when storing objects in Azure Tables but when querying them it seems that Azure requires the setters to be there, even if they do nothing at all. Adding one in fixed nearly every problem I was encountering and brought me back to another problem I had faced in the past (more on that when I finally fix it!).

Like any mature framework that does a lot of the heavy lifting for you Microsoft’s solutions suffer when you start to tread unknown territory. Realistically though this is should be expected and I’ve found I spend the vast majority of my time on less than 20% of the code that ends up making the final solution. The upshot is of course that once these barriers are down progress accelerates at an extremely rapid pace, as I saw with both the Silverlight and iPhone clients for Lobaco. My cloud authentication services are nearly ready for prime time and since I struggled so much with this I’ll be open sourcing my solution so that others can benefit from the numerous hours I spent on this problem. It will be my first ever attempt at open sourcing something that I created and the prospect both thrills and scares me, but I’m looking forward to giving back a little to the communities that have given me so much.

Necessity is the Mother of Invention.

I’ve been developing computer programs on and off for a good 7 years and in that time I’ve come across my share of challenges. The last year or so has been probably the most challenging of my entire development career as I’ve struggled to come to grips with the Internet’s way of doing things and how to enable disparate systems to talk to each other. Along the way I’ve often hit various problems that on the surface appear to be next to impossible to do or I come to a point where a new requirement makes an old solution no longer viable. Time has shown however that whilst I might not be able to find an applicable solution through hours of intense Googling or RTFM there are always clues that lead to an eventual solution. I’ve found though that such solutions have to be necessary parts of the larger solution otherwise I’ll just simply ignore them.

Take for instance my past weekend’s work gone by with Lobaco. Things had been going well, the last week’s work had seen me enable user sign ups in the iPhone application and had the beginnings of an enhanced post screen that allowed users to post pictures along with their posts. Initial testing of the features seemed to work well and I started testing the build on my iPhone. Quickly however I discovered that both the new features I had put in struggled to upload images to my web server, crashing whenever a picture was over 800 by 600 in size. Since my web client seemed to be able to handle this without an issue I wondered what the problem would be, so I started digging deeper.

You see way back when I had resigned myself to doing everything in JavaScript Object Notation, or JSON for short. The reason behind this was that thanks to it being an open standard nearly every platform out there has a native or third party library for serialising and deserialising objects, making my life a whole lot easier when it comes to cross platform communication (I.E. my server talking to an iPhone). Trouble with this format is that whilst it’s quite portable everything done in it must be text. This causes a problem for large files like images as they have to be changed into text before they can be sent over the Internet. The process I used for this is called Base64 and it has the unfortunate side effect of increasing the size of the file to be transferred by roughly 37%. It also generates an absolutely massive string that brings any debugger to its knees if you try to display it, making troubleshooting issues hard.

The image uploading I had designed and successfully used up until this point was now untenable as the iPhone client simply refused to play nice with ~300KB strings. I set about trying to find a solution to my problem hoping to find a quick solution to my problem. Whilst I didn’t find a good drag and drop solution I did come across this post which detailed a way in which to program a Windows web service that could receive arbitrary data. Implementing their solution as it is detailed there still didn’t actually work as advertised but after wrangling the code and over coming the inbuilt message size limits in WCF I was successfully able to upload images without having to muck around with enormous strings. This of course did mean changing a great deal of how the API and clients worked but in the end it was worth it for something that solved so many problems.

The thing is before I went on this whole adventure had you asked me if such a thing was possible I would’ve probably told you no, at least not within the whole WCF world. In fact much of my early research into this problem was centred around possibly implementing a small PHP script to accomplish the same thing (as there are numerous examples of that already), however the lack of integration with my all Microsoft solution means I’d be left with a standalone piece of code that I wouldn’t have much interest in improving or maintaining. By the simple virtue that I had to come up with a solution to this problem meant I tried my darnedest to find it, and lo I ended up creating something I couldn’t find anywhere else.

It’s like that old saying that necessity is the mother of all invention and that’s true for both this problem and Lobaco as an idea in itself. Indeed many of the current great Internet giants and start ups were founded on the idea of solving a problem that the founders themselves were experiencing and felt that things could be better. I guess I just find it fascinating how granular a saying like that can be, with necessity driving me to invent solutions at all levels. It goes to show that embarking into the unknown is a great learning experience and there’s really no substitute for diving in head first and trying your hardest to solve an as of yet unsolvable problem.

Technological Solutions to Human Problems.

It would seem as more time passes the more we are in control of the restrictions that have been placed on us by our ancestors. Natural Selection did a pretty good job of giving us a foundation of a large brain relative to our body mass, giving us a leg up on cognitive functions. Yet the more we progress we also find ourselves stuck with problems that are intrinsically human and as such will probably not be solved technologically.

The Turing test is a simple example of one such problem. In essence the test seeks to develop a computer program “intelligent” enough to fool a human observer into thinking it’s human. At its heart this is a human problem, communication with another being at a meaningful level. If you’ve ever tried to talk with a chat bot as if it were human you’ll notice some characteristics after a short period of time:

  • Lack of memory: Most chat bots won’t remember the conversation you had with them past the last sentence. This can lead to some silly conversational loops since they’ll ask you to clarify points they appeared to understand not two sentences earlier.
  • Can’t be taught: A lot of chat bots are based on Natural Language Processing where they’re using algorithms based on grammatical constructs and the sentence structure. Whilst you can use this to form some sort of idea about what the other party is saying you can’t teach this dog any tricks once you’ve programmed it. I had a lovely conversation with the “winner” of last years Turing test about regolith. After teaching it what it was I then asked it to describe it back to me. Needless to say it couldn’t, and asked me what I was talking about.
  • Spout nonsensical gibberish: If you hit one of the mid points in some of the NLP algorithms you’ll notice that most chat bots come back with a question which takes part of your sentence and regurgitates it back at you. If you said something like “Well I do enjoy opera whilst knee deep in peanut butter” they will come back with something like “What makes it that you enjoy opera whilst knee deep in peanut butter?”. Forgive the awkward grammar and you might be able to pass that off as normal conversation, but no one talks like that normally.

No program has yet completed this test successfully, although there are many good candidates.

The old saying “when all you have is a hammer all your problems look like nails” is something that applies here. Using technology to solve our day to day problems and better our lives is the hammer, and our problems arising from the human condition can all look like nails. Whilst I believe that no problem is above being solved, given enough time and resources, there are some things about life that are just so intrinsically human that technology will struggle to overcome them. Sometimes it is easier to adapt ourselves to overcome such problems; akin to a kind of artificial evolution.

So, what’s the point of trying then? Well I can put it down to two reasons. The first being our insatiable desire to overcome any obstacle that is placed in front of us. Routinely in our past we humans have been faced with problems that appeared monumental. Just on 100 years ago international travel took many months across the sea and was frought with danger. Now you can walk up to almost any airport and choose a country and be there in less than a day.

Secondly, on a time scale that we experience evolution makes negligible changes to us. We are at the stage where we have evolved to a point that if we want to go any further mother nature’s course will take thousands of generations for us to get there. If the human species is to survive and thrive in this barren universe we have to learn to master the world we exist in and then continue the process throughout the solar system and beyond. This is the only way to ensure that our race can survive through catastrophic events such as the loss of the entire earth.

Overall, we seek to overcome our shortcomings due to our innate desire to thrive.