The promise of Big Data has been courting many a CIO for years now, the allure being that all the data they have on everything can be fed into some giant engine that will then spit out insights for them. However like all things the promise and the reality are vastly different beasts and whilst there are examples of Big Data providing never before seen insights it hasn’t really revolutionized industries in the way other technologies have. A big part of that is that Big Data tools aren’t push button solutions, requiring a deep understanding of data science in order to garner the insights you seek. IBM’s Watson however is a much more general purpose engine, one that I believe could potentially deliver on the promises that its other Big Data compatriots have made.
The problem I see with most Big Data solutions is that they’re not generalizable, I.E. a solution that’s developed for a specific data set (say a logistics company wanting to know how long it takes a package to get from one place to another) will likely not be applicable anywhere else. This means whilst you have the infrastructure and capability to generate insights the investment required to attain them needs to be reapplied every time you want to look at the data in a different way or if you have other data that requires similar insights to be derived from it. Watson on the other hand falls more into the category of a general purpose data engine that can ingest all sorts of data and provide meaningful insights, even to things you wouldn’t expect like helping to author a cookbook.
The story behind how that came about is particularly interesting as it showed what I feel is the power of Big Data without the required need to have a data science degree to exploit it. Essentially Watson was fed with over 9000 (ha!) recipes from Bon Appétit‘s database which was then supplemented with the knowledge it has around flavour profiles. It then used all this information to derive new combinations that you wouldn’t typically think of and then provided them back to the chefs to prepare. Compared to traditional recipes the ingredient lists that Watson provided were much longer and involved however the results (which should be mostly attributed to the chefs preparing them) were well received showing that Watson did provide insight that would otherwise have been missed.
That’d just be an impressive demonstration of data science if it wasn’t for the fact that Watson is now being used to provide similar levels of insight across a vast number of industries from medical to online shopping to even matching remote workers with employers seeking their skills. Whilst it’s far short of what most people would class as a general AI (it’s more akin to a highly flexible expert system on the data it’s provided) Watson has shown that it can be fed a wide variety of data sets and can then be queried in a relatively straightforward way. It’s that last part that I believe is the secret sauce to making Big Data usable and it could be the next big thing for IBM.
Whether or not they can capitalize on that though is what will determine if Watson becomes the one Big Data platform to rule them all or simply an interesting footnote in the history of expert systems. Watson has already proven its capabilities numerous times over so fundamentally it’s ready to go and the responsibility now resides with IBM to make sure it gets in the right hands to further develop it. Watson’s presence is growing slowly but I’m sure a killer app isn’t too far off for it.
In a world where Siri can book you a restaurant and Google Now can tell you when you should head for the gate at the airport it can feel like the AI future that many sci-fi fantasies envisioned is already here. Indeed to some extent it is, many aspects of our lives are now farmed out to clouds of servers that make decisions for us, but those machines still lack a fundamental understanding of, well, anything. They’re what are called expert systems, algorithms trained on data to make decisions in a narrow problem space. The AI future that we’re heading towards is going to be far more than that, one where those systems actually understand data and can make far better decisions based on that. One of the first steps to this is IBM’s Watson and it’s creators have done something amazing with it.
Whilst currently only open to partner developers IBM has created an API for Watson, allowing you to pose it a question and receive an answer. There’s not a lot of information around what data sets it currently understands (the example is in the form of a Jeopardy! question) but their solution documents reference a Watson Content Store which, presumably, has several pre-canned training sets to get companies started with developing solutions. Indeed some of the applications that IBM’s partner agencies have already developed suggest that Watson is quite capable of digesting large swaths of information and providing valuable insights in a relatively short timeframe.
I’m sure many of my IT savvy readers are seeing the parallels between Watson and a lot of the marketing material that surrounds anything with the buzzword “Big Data”. Indeed much of the concepts of operation are similar: take big chunks of data, throw them into a system and then hope that something comes out the other end. However Watson’s API suggests something that’s far more accessible, dealing in native human language and providing evidence to back up the answers it gives you. Compare this to Big Data tools, which often require you to either learn a certain type of language or create convoluted reports, and I think Watson has the ability to find widespread use while Big Data keeps its buzzword status.
For me the big applications for something like this come for places where curating domain specific knowledge is a long, time consuming task. Medicine and law both spring to mind as there’s reams of information available to power a Watson based system and those fields could most certainly benefit from having easier access to those vast treasure troves. It’s pretty easy to imagine a lawyer looking for all precedents set against a certain law or a doctor asking for all diseases with a list of symptoms, both queries answered with all the evidence to boot.
Of course it remains to be seen if Watson is up to the task as whilst it’s prowess on Jeopardy! was nothing short of amazing I’ve still yet to see any of its other applications in use. The partner applications do look very interesting, and should hopefully be the proving grounds that Watson needs, but until it starts seeing widespread use all we really have to go on is the result of a single API call. Still I think it has great potential and hopefully it won’t be too long before the wider public can get access to some of Watson’s computing genius.
I’m not sure why but I get a little thrill every time I see something that’s been completely automated that used to require manual intervention from start to finish. It’s probably because the more automated something is the more time I have to do other things and there’s always that little thrill in watching something you built trundle along its way, even if it falls over part way through. My most recent experiment in this area was crafting the rudimentary trainer for Super Meat Boy to get me past a nigh on impossible part of the puzzle, co-ordinating the required key strokes with millisecond precision and ultimately wresting me free of the death grip that game held on me.
The world of AI is an extension of the automation idea, using machines to perform tasks that we would otherwise have to do ourselves. The concept has always fascinated me as more and more we’re seeing various forms of AI creeping their way into our everyday lives. However most people won’t recognize them as AI simply because they’re routine, but in reality many of the functions these weak AIs perform used to be in the realms of science fiction. We’re still a long way from having a strong AI like we’re used to seeing in the movies but that doesn’t mean many facets of it aren’t already in widespread use today. Most people wouldn’t think twice when a computer asks them to speak their address but going back only a few decades would see that be classed as the realms of strong AI, not the expert system it has evolved into today.
What’s even more interesting is when we create machines that are more capable than ourselves at performing certain tasks. The most notable example (thus far) of a computer be able to beat a human at a certain non-trivial task is Deep Blue, the chess playing computer that managed to beat the world chess champion Kasparov albeit under dubious circumstances. Still the chess board is a limited problem set and whilst Deep Blue was a super computer in its time today you’d find as much power hidden under the hood of your Playstation 3. IBM’s research labs have been no slouch in developing Deep Blue’s successor, and it’s quite an impressive beast.
Watson, as it has come to be known, is the next step in the evolution of AIs performing tasks that have only been in the realms of humans. The game of choice this time around is Jeopardy a gameshow who’s answers are in the form of a question and makes extensive use of puns and colloquialisms. Jeopardy represents a unique challenge to AI developers as it involves complex natural language processing, searching immense data sets and creating relationships between disparate sources of information to finally culminate in an answer. Watson can currently determine whether or not it can answer a question within a couple seconds but that’s thanks to the giant supercomputer that’s backing it up. The demonstration round showed Watson was quite capable of playing with the Jeopardy champions, winning the round quite with a considerable lead.
What really interested me in this though was the reaction from other people when I mentioned Watson to them. It seemed that a computer playing Jeopardy (and beating the human players) wasn’t really a big surprise at all, in fact it was expected. This was telling about how us humans view computers as most people expect them to be able to accomplish anything, despite the limitations that are obvious to us geeks. I’d say this has to do with the ubiquity of computers in our everyday lives and how much we use them to perform rudimentary tasks. The idea that a computer is capable of beating a human at anything isn’t a large stretch of the imagination if you treat them as mysterious black boxes but it still honestly surprised me to learn this is how many people think.
Last night saw Watson play its first real game against the Jeopardy champions and whilst it didn’t repeat its performance of the demonstration round it did tie for first place. The second round is scheduled to air sometime tomorrow (Australia time) and whilst I’ve not yet had a chance to watch the entire round I can’t tell you how excited I am to see the outcome. Either way the realm of AI has taken another step forward towards the ultimate goal of creating intelligence born not out of flesh, but silicone and whilst some might dread the prospect I for one can’t wait and will follow all developments with baited breath.