The promise of Big Data has been courting many a CIO for years now, the allure being that all the data they have on everything can be fed into some giant engine that will then spit out insights for them. However like all things the promise and the reality are vastly different beasts and whilst there are examples of Big Data providing never before seen insights it hasn’t really revolutionized industries in the way other technologies have. A big part of that is that Big Data tools aren’t push button solutions, requiring a deep understanding of data science in order to garner the insights you seek. IBM’s Watson however is a much more general purpose engine, one that I believe could potentially deliver on the promises that its other Big Data compatriots have made.
The problem I see with most Big Data solutions is that they’re not generalizable, I.E. a solution that’s developed for a specific data set (say a logistics company wanting to know how long it takes a package to get from one place to another) will likely not be applicable anywhere else. This means whilst you have the infrastructure and capability to generate insights the investment required to attain them needs to be reapplied every time you want to look at the data in a different way or if you have other data that requires similar insights to be derived from it. Watson on the other hand falls more into the category of a general purpose data engine that can ingest all sorts of data and provide meaningful insights, even to things you wouldn’t expect like helping to author a cookbook.
The story behind how that came about is particularly interesting as it showed what I feel is the power of Big Data without the required need to have a data science degree to exploit it. Essentially Watson was fed with over 9000 (ha!) recipes from Bon Appétit‘s database which was then supplemented with the knowledge it has around flavour profiles. It then used all this information to derive new combinations that you wouldn’t typically think of and then provided them back to the chefs to prepare. Compared to traditional recipes the ingredient lists that Watson provided were much longer and involved however the results (which should be mostly attributed to the chefs preparing them) were well received showing that Watson did provide insight that would otherwise have been missed.
That’d just be an impressive demonstration of data science if it wasn’t for the fact that Watson is now being used to provide similar levels of insight across a vast number of industries from medical to online shopping to even matching remote workers with employers seeking their skills. Whilst it’s far short of what most people would class as a general AI (it’s more akin to a highly flexible expert system on the data it’s provided) Watson has shown that it can be fed a wide variety of data sets and can then be queried in a relatively straightforward way. It’s that last part that I believe is the secret sauce to making Big Data usable and it could be the next big thing for IBM.
Whether or not they can capitalize on that though is what will determine if Watson becomes the one Big Data platform to rule them all or simply an interesting footnote in the history of expert systems. Watson has already proven its capabilities numerous times over so fundamentally it’s ready to go and the responsibility now resides with IBM to make sure it gets in the right hands to further develop it. Watson’s presence is growing slowly but I’m sure a killer app isn’t too far off for it.
Many moons ago I was checking out GoPros for the upcoming Tough Mudder event because I wanted to record some first person perspective footage, much like many of the other participants did. Of course this entailed me actually going to the GoPro website and checking out their wares which, after careful consideration, lead me to lust after the most recent model. Since it was still a fair way out from the event I hadn’t planned to grab one then and there so I bookmarked the model I wanted and then proceeded to go about my usual browsing activities. Only something had changed in the time between my first visiting the GoPro site and leaving it and it wasn’t the first time I’d noticed such behaviour.
Indeed I wrote about this at the start of the year when my thinking was along the lines of these being the highest CPC ads that the network could deliver at the time but I’ve started to notice similar behaviour on other sites. Amazon for instance routinely sends me a list of items that I might be interested in which is actually a service that I’ve opted in for (my traditional means of product discovery are quite laborious). However I couldn’t help but notice that every single product that Amazon recommends to me are things that I’ve either searched for on the site previously or even products I attempted to buy from them only to be told that they wouldn’t ship them outside the USA. It seems really strange as they seem to be able to recommend other products on their site without too much trouble but with anything else it seems they’re left dumbfounded.
So this got me thinking, all these analytical engines out there, which are apparently the magic sauce behind all of these targeted advertising systems, must be spectacularly crap. I’m not the most private person I’m constantly spamming this blog, Twitter and Facebook with all manner of inane stuff I’m interested in so its not like there isn’t a whole lot of data these guys could be pillaging in order to figure out what they should be peddling to me. Indeed Google has the poorest excuse of the lot of them as I browse through a logged in Google Chrome and search whilst logged into my Google account. Still their algorithms seem to be heavily weighted to advertise things to you that you’ve already seen which, at least in my case, seems counter to what you’d want to do.
The flip side of this is that I’m somehow not giving out information for these things to be able to make accurate recommendations which I just don’t believe is the case. Amazon and Google have a treasure trove of information related to my searching, viewing and buying habits and yet I rarely see advertisements or recommendation for things like cameras, supplements and tech gadgets all of which can be high value/high margin sales. I could just have the blinkers on for the text ads (I rarely read them any more, but the graphical ones do catch my eye) but I highly doubt that’s the case.
Facebook is probably the one who gets it the closest as whilst there was a long period where they were simply allowing targeting based on someone’s likes it does seem to do a rather good job of inferring what I would be interested in without referring to it often. You could argue that’s because it has a deeper insight into me thanks to the tendency for people to share details they wouldn’t otherwise on that particular network but there’s not really much more on there than anywhere else, certainly not for Google.
This could all be an artefact of my better-than-average memory which remembers things like this. It’s quite possible that the vast majority of people do in fact do the majority of their product discovery themselves and simply forget about it which means that kind of targeting would be effective. Indeed when I’ve talked about this phenomena with other people I’m usually met with blank stares as they don’t seem to notice any trends like this. Whatever it is every time I notice it I get pushed just a little closer to installing AdBlock, even though I want to keep supporting sites who pony up their content free. That irritates me as I shouldn’t have to make that kind of decision if these algorithms were doing their job properly.