Not all of the “Stuff” sloshing around the lab is the same, and distinguishing between them helps tease out the best place to store things. We use a simple Triangle Diagram (originally proposed by John Trigg of PhaseFour which really just tries to point out that stuff is related, but it’s at different levels of abstraction:

It is quite hard to draw definite lines around things, but I think most people can appreciate that a raw data dump from an instrument is somewhat different from a report to management, or that an experimental write up in word is different from some tabular data in a spreadsheet. The differences between the levels come out in:

• The software that’s used to read the file and interpret the content. Some will require very specific software (e.g. from an instrument vendor), but a PDF or text file can be read by many different things.
• Who might be interested in the data. Again, some files are useful to anyone (for example, a report) but some only useful to certain people with specific training.
• How long your company might want to keep the data, and indeed how long you are realistically able to keep the data. Typically the lower you go, the harder it is to keep something, so if you feel it’s business critical you really need to pay attention to the formats used.

This differentiation can really help in ELN System design. Partly it draws your attention to what needs to be stored in the ELN (typically the “Experiment” write up level), and what can be left in systems e.g. a database or a file server, which can be pointed to from the ELN.

Not everything needs to be stored in the ELN, and indeed it would be unrealistic to expect to be able to do so. The important thing is common keys so you can offer the user a link to more information, and the advent of web-based systems has made this level of “integration” so trivial one sometimes feels a bit of fraud describing it as such.

By building on the storage tools you have in place, and focusing an ELN on Experiments, the resulting system is cheap to run, costs little to acquire, and results in little disruption to existing practices.

You can read the final part of this series here.

Standards can not force a vendor to be interoperable. If a vendor wishes deliberately to withhold interoperability from the market, then they will always be able to do so, and, in most cases, devise an excuse using the text of the standard as a scapegoat.

You can’t rely on any representations about a “standard” unless there’s clear evidence of different implementations working together.

This really matters for ELNs, for two reasons:

• Your data is locked up in the ELN unless you know you have proved you can get it out.
• The need for long term (typically 30+) access to proof of experimental activity for Patent purposes means you can’t rely on any of the existing vendors being around, or the software you are currently using being functional.

We recommend that customers reassure themselves before they purchase any ELN product that they can get the data they need out into some suitable format – you have leverage with your supplier before you purchase, after that you’re locked in. I am amazed how many customers are aggressive about price discounts etc. but don’t look carefully at the “open data” situation – which is surely much more important than a 3% cost saving on the license price.

There are a number of ELNs around at the moment which can’t reliably create a PDF of the experiment. To some extent it’s a hard problem, but my concern is the “How do we get our data out” is answered with a wave of the hand and “You can just export to PDF” and everyone takes that at face value. In fact in some cases the PDF export facility is unreliable, doesn’t contain all the data, and can’t be automated! Buyer beware…

In terms of picking good “standards” we prefer published, straightforward, and well-used file formats. In our case that currently means XML and PDF/A, both of which are very open and used extensively by a wide range of software.

The project is on GitHub – here’s the project’s GitHub page.

The checker script is a completely separate implementation of the signature and repository code, and is a useful way for anyone – Amphora customer or not – to check that things are OK with their data.

An important part of PatentSafe’s value is that it creates an open repository which you can read and take in to court without needing any additional software from Amphora. Everything is completely open as standard, no need for a complicated export step, or any software except a PDF reader, a text reader, and OpenSSL. The open release of the checker script is just part of this.

Some of our clients have scans of their Paper Notebooks, and you can put them into PatentSafe quite easily. However, most people are put off by the expense of scanning (& tagging), which is a real pity.

Given the amount of knowledge that’s locked in legacy Paper Lab Notebooks, let’s hope Google’s technology is made available outside of Google.

However, my inner geek keeps looking at the existing Chemistry ELN products from a technical perspective and thinking:

• They’re mostly based on quite old platforms. Web 2.0 isn’t really making an appearance – this is very traditional enterprise software with all the associated problems.
• I’m not sure what value these really thick clients are adding. Seems like a really expensive and painful way to make Excel harder to use, and are a general support nightmare.
• What’s going to happen when someone takes something like WordPress and adds structures to it?

The more I play with things like WordPress and Rails, the more impressed I am with what can be done with web UIs. And then I bump into things like Metamolecular’s ChemWriter and I think the future will be rather exciting…

The next question is if these solutions will come out of the Open Source community or commercial providers. Having met some of the guys in Academia I suspect it’ll be taking what they’ve done and bending it slightly to the needs of commercial organisations (plus adding support services etc.)

Anyway, hopefully all of this innovation on the front end means more people with the problem our PatentSafe product solves. Which of course is delightful…

James Governor’s MonkChips: digital lard for the enterprise: DRM meets document formats: “What I am saying is that DRM creates new escrow challenges, and organisations should know exactly what they are using it for, and why, and what risks they are mitigating, before embarking on an enterprise DRM strategy.”

Firstly only did Massachusetts make their decision in a very public way (so we can all “see their working”), clearly documenting why they felt Open Formats were important from a business perspective, and how they defined what one way. Now the resulting backlash from Microsoft is producing an awful lot of good analysis about the problems of file formats, which is very relevant for companies considering ELN purchases.

To me, this is increasingly looking like the end of proprietary file formats (and hence the use of file formats as a commercial weapon) and the more the vendors kick & scream, the faster their customers will understand they need demand the protection that Open Formats provide.

Groklaw has a really good wrap up of all the issues and considerations with pointers to other material. If you haven’t been watching this then it is a good first place to start your research into why Open File Formats are a crucial consideration in any ELN purchase.

Here’s a particularly relevant quote:

Few leading vendors ever really want to have an agreed-on open standard, because that opens the market to competitors. But once a technology stabilizes, wise customers demand standards that let them pick between suppliers, and customers are the ones paying the bills. This happens all the time, in all technology areas.

As a result of this, the community has had the opportunity of seeing Microsoft’s “Talking Points” and soundly rejecting them. We’ve also seen how desperate Microsoft is to protect their Office cash cow.

Why is this important for Electronic Lab Notebooks?

• It demonstrates (yet again) that Open File formats are an issues that needs to be taken seriously at the highest levels. Hopefully as governments around the world begin to insist on open formats, corporates will be being to take the problem seriously too.
• The arguments vendors use to justify their proprietary file formats are increasingly being shown to be false.
• The increasing experience with open formats shows that it isn’t that hard to build systems that are open, and give customers first-hand experience of the benefits of Open Systems.

Electronic Lab Notebooks systems benefit more than normal from open file formats because:

• Almost ELN systems are made up of software from more than one source (commercial vendors, as well as a large number of small, focused applications developed in-house).
• The information in ELN systems often needs to be accessible in 30 years or longer (“Lifetime of the Company” is very typical). The chances of the original application being available is minimal.
• ELN systems often need to do an awful lot of unusual stuff, which wasn’t anticipated by the original designers. Yet the market for ELNs is comparatively small compared to the overall market for products like Office. So ELN functionality is often an after-the-fact addon implemented internally or by niche vendors (like my company, Amphora).

My feeling is that the market is going to punish ELN vendors who are still holding to their closed proprietary file formats, and preventing their users getting access to their data (by not releasing APIs). I spoke to one vendor at a conference a few months ago and he was genuinely confused why anyone would need their data outside their ELN – and felt strongly that moving to an open file format would be unduly hard for them. Duh. All they’d have to do would be to publish their internal documentation and they’d be OK – no code changes required. But that would require giving up control….

In contrast, there are ELN suppliers who do strongly support open file formats, and encourage our customers to access their data which is stored in our systems, etc. Our PatentSafe product is one, but there are many others. Did I mention we’re growing like crazy….. I guess the market likes vendors who look after their customers’ interests

For more information you might want to readWhy OpenDocument Won (and Microsoft Office Open XML Didn’t) by David A. Wheeler. There’s also a very interesting letting from Adam Barr (of Microsoft) to Jeff Raikes (Group VP at Microsoft) explaining the reasons for Massachusetts’s choice, and why it shouldn’t be a problem for Microsoft to work with open formats rather than resisiting. Replace “Microsoft” with your favourite ELN vendor and see how it feels…

I hope it won’t be long before companies start to require “that all electronic documents “created and saved” by state employees would have to be based on open formats”. That will be a major advance for the interests of customers.

How long will it be before industry also wakes up to the problem? There’s no need for proprietary file formats, all it does is increase user lock-in to that specific vendor.

Hashes are an important part of any Digital Signature scheme, and SHA-1 is one of the more popular (and until now well-respected) Hash algorithms. Any flaw found in the hashing algorithm is a serious problem for any ELN system that uses Digital Signatures to prove a record has not been changed. As Jon Callas, PGP’s CTO, puts it: ‘It’s time to walk, but not run, to the fire exits. You don’t see smoke, but the fire alarms have gone off.’

So this isn’t a crisis – if you were using digital signatures yesterday, they haven’t suddenly become worthless today. But it is a reminder that nothing lasts forever, especially in Cryptography. Flaws are found in algorithms, computers get faster, and soon it costs a mere $1m to forge the Digital Signature on the document that proves you have the rights to a Blockbuster drug. Picking a “better” algorithm or longer key isn’t going to help – erosion of Cryptographic tools is a fact of life.

Any Evidence system needs to have a series of controls and capabilities to show that a document has been unaltered. Digital Signatures are one of the tools you would use in such a system (we use them in our PatentSafe product) but they shouldn’t be the only thing you rely on. Indeed PatentSafe has a whole series of checks & balances in it and if you run the system properly, even if it turned out the signature algorithm was worthless, you’d still be able to use your records in court.

Unfortunately there are some vendors who are relying on Cryptography alone to prove authenticity. Listening to their marketing, they’re treating the technology as a silver bullet – “Buy this magic, and all your concerns are over”. This has always made me intensely uncomfortable, especially given the timescales that our customers expect to be able to use the records they put into our systems. Fortunately given the level of concern over Electronic Records for patents, few customers have actually implemented such systems – the laggards have been proven correct!

It is possible to do perfectly safe, effective, Electronic Records systems for Patent Evidence Creation and Preservation. Unfortunately, it is a lot harder than just implementing some Cryptographic magic, as the demise of SHA-1 shows. My problem is that it is much easier (and more seductive) to talk about apparently sexy technology rather than the real (but less exciting) issues around good electronic records systems