Posts Tagged‘national archives of australia’

RED Epic Side Shot

You Can’t Archive Digital Video? Surely You Jest.

On recommendation of a friend I recently watched a documentary called Side by Side which details the history of the primary technology behind the cinema: the cameras. It starts off by giving you an introduction into the traditional photographic methods that were used to create films in the past and then goes on to detail the rise of digital in the same space. Being something of a photographic buff myself as well as a technological geek who can’t get enough of technology the topic wasn’t something I was unfamiliar with but it was highly interesting to see what people in the industry were thinking about the biggest change to happen in their industry in almost a century.

RED Epic Side Shot

Like much of my generation I grew up digitally with the vast majority of my life spent alongside computers and other non-analog style equipment. I was familiar with film as my father was something of a photographer (I believe his camera of choice was a Pentax K1000 which he still has, along with his Canon 60D) and my parents gave me my own little camera to experiment with. It wasn’t until a good decade and a half later that I’d find myself in possession of my first DSLR and still not another few years until after then that I’d find some actual passion for it. What I’m getting at here is that I’m inherently biased towards digital since it’s where I found my feet and it’s my preferred tool for capturing images.

One of the arguments that I’ve often heard levelled at digital formats, both in the form of images and your general everyday data, is that there’s no good way to archive it in order for future generations to be able to view it. Film and paper, the traditional means with which we’ve stored information for centuries, would appear to archive quite well due to the amount of knowledge contained in those formats that has stood the test of time. Ignoring for the moment that digital representations of data are still something of a nascent technology by comparison the question of how we archive it has come up time and time again and everyone seems to be under the impression that there’s no way to archive it.

This just isn’t the case.

Just before I was set to graduate from university I had been snooping around for a better job after my jump to a developer hadn’t worked out as I planned. As luck would have it I managed to land a job at the National Archives of Australia, a relatively small organisation tasked with the monumental effort of cataloguing all records of note that were produced in Australia. This encompassed all things from regular documents used in the course of government to things of cultural value like the air line tickets from when the Beatles visited Australia. Whilst they were primarily concerned with physical records (as shown by their tremendous halls filled with boxes) there was a small project within this organisation that was dedicated to the preservation of records that were born digital and were never to see the physical world.

I can’t take much credit for the work that they did there, I was merely a care taker of the infrastructure that was installed long before I arrived but I can tell you about the work they were doing there. The project team, consisting mostly of developers with just 2 IT admins (including myself), was dedicated to preserving digital files in the same way you would do with a paper record. At the time a lot of people were still printing them off and then archiving them in that way however it became clear that this process wasn’t going to be sustainable, especially considering that the NAA had only catalogued about 10% of their entire collection when I was there (that’s right, they didn’t know what 90% of the stuff they had contained). Thankfully many of the ideas used in the physical realm translated well to the digital one and thus XENA was born.

XENA is an open source project headed by the team at NAA that can take everyday files and convert them into an archival format. This format contains not only the content but also the “essence” of the document, I.E. it’s presentation, layout and any quirks that make that document, that document. The viewer included is then able to reconstruct the original document using the data contained within the file and since the project is open source should the NAA cease development on the project the data will still be available for all of those who used the XENA program. The released version does not currently support video but I can tell you that they were working on it while I was there but the needs of archiving digital documents was the more pressing requirement at the time.

Ah ha, I’ll hear some film advocates say, but what about the medium you store them on? Surely there’s no platform that can guarantee that the data will still be readable in 20 years, heck even 10 I’ll bet! You might think this, and should you have bought any of the first generation of CD-Rs I wouldn’t fault you for it, but we have many ways of storing data for long term archival purposes. Tapes are by far the most popular (and stand the test of time quite well) but for truly archival quality data storage that exists today nothing beats magneto-optical discs which can have lives measured in centuries. Of course we could always dive into the world of cutting edge science for likes like a sapphire etched platinum disc that might be capable of storing data for up to 10 million years but I think I’ve already hammered home the point enough.

There’s no denying that there are challenges to be overcome with the archival of digital data as the methods we developed for traditional means only serve as a pointer in the right direction. Indeed attempting to apply them to digital the world has often had disastrous results like the first reel of magnetic tape brought to the NAA which was inadvertenly baked in an oven (done with paper to kill microbes before archival), destroying the data forever. This isn’t to say we don’t have anything nor are we not working on it however and as technology improves so will the methods available for archiving digital data. It’s simply a matter of time until digital becomes as durable as its analogue counterpart and, dare I say it, not long before it surpasses it.

Why Macs and Enterprise Computing Don’t Mix.

I’m a big fan of technology that makes users happy. As an administrator anything that keeps users satisfied and working productively means more time for me to make the environment even better for them. It’s a great positive feedback loop that builds on itself continually, leading to an environment that’s stable, cutting edge and just plain fun to use and administer. Of course the picture I’ve just painted is something of an IT administrator nirvana, a great dream that is rarely achieved even by those who have unlimited freedom with the budgets to match. That doesn’t mean we shouldn’t try to achieve it however and I’ll be damned if I haven’t tried at every place I’ve ever worked at.

The one thing that always come up is “Why don’t we use Macs in the office? They’re so easy to use!”. Indeed my two month long soiree into the world of OSX and all things Mac showed that it was indeed an easy operating system to pick up and I could easily see why so many people use it as their home operating system. Hell at my current work place I can count several long time IT geeks who’ve switched their entire household over to solely Apple gear because it just works and as anyone who works in IT will tell you the last thing you want to be doing at  home is fixing up PCs.

You’d then think that Macs would be quite prevalent in the modern workspace, what with their ease of use and popularity amongst the unwashed masses of users. Whilst their usage in the enterprise is growing considerably they’re still hovering just under 3% market share, or about the same amount of market share that Windows Phone 7 has in the smart phone space. That seems pretty low but it’s in line with world PC figures with Apple being somewhere in the realms of 5% or so. Still there’s a discrepancy there so the question still remains as to why Macs aren’t seen more often in the work place.

The answer is simple, Apple simply doesn’t care about the enterprise space.

I had my first experience with Apple’s enterprise offerings very early on in my career, way back when I used to work for the National Archives of Australia. As part of the Digital Preservation Project we had a small data centre that housed 2 similar yet completely different systems. They were designed in such a way that should a catastrophic virus wipe out the entire data store on one the replica on the other should be unaffected since it was built from completely different software and hardware. One of these systems utilized a few shelves of Apple’s Xserve RAID Array storage. In essence they were just a big lump of direct attached storage and for that purpose they worked quite well. That was until we tried to do anything with it.

Initially I just wanted to provision some of the storage that wasn’t being used. Whilst I was able to do some of the required actions through the web UI the unfortunate problem was that the advanced features required installing the Xserve tools on a Mac computer. Said computer also had to have a fibre channel card installed, something of a rarity to find in a desktop PC. It didn’t stop there either, we also tried to get Xsan installed (so it would be, you know, an actual SAN) only to find out that we’d need to buy yet more Apple hardware in order to be able to use it. I left long before I got too far down that rabbit hole and haven’t really touched Apple enterprise gear since.

You could write that off as a bad experience but Apple has continued to show that the enterprise market is simply not their concern. No less than 2 years after I last touched a Xserve RAID Array did Apple up and cancel production of them, instead offering up a rebadged solution from Promise. 2 years after that Apple then discontinued production of its Xserve servers and lined up their Mac Pros as a replacement. As any administrator will tell you the replacements are anything but and since most of their enterprise software hasn’t recieved a proper update in years (Xsan’s last major release was over 3 years ago) no one can say that Apple has the enterprise in mind.

It’s not just their enterprise level gear that’s failing in corporate environments. Whilst OSX is easy to use it’s an absolute nightmare to administer on anything larger than a dozen or so PCs as all of the management tools available don’t support it. Whilst they do integrate with Active Directory there’s a couple limitations that don’t exist for Windows PCs on the same infrastructure. There’s also the fact that OSX can’t be virtualized unless it runs on Apple hardware which kills it off as a virtualization candidate. You might think that’s a small nuisance but it means that you can’t do a virtual desktop solution using OSX (since you can’t buy the hardware at scale to make it worthwhile) and you can’t utilize any of your current investment in virtual infrastructure to run additional OSX servers.

If you still have any doubts that Apple is primarily a hardware company then I’m not sure what planet you’re on.

For what its worth Apple hasn’t been harmed by ignoring the enterprise as it’s consumer electronics business has more than made up for the losses that they’ve incurred. Still I often find users complaining about how their work computers can’t be more like their Macs at home, ignorant of the fact that Apple’s in the enterprise would be an absolutely atrocious experience. Indeed it’s looking to get worse as Apple looks to iPhoneizing their entire product range including, unfortunately, OSX. I doubt Apple will ever change direction on this which is a real shame as OSX is the only serious competitor to Micrsoft’s Windows.