Electronic Lab Notebook Requirements – possible pitfalls

I wrote this article by mistake (yeah, I know) but thought it was worth putting up here anyway…

Requirements Gathering – a Broken Process

Project teams have been drawing up lists of requirements since the dawn of time, and since that first list the fate of a project has to a great extent been sealed the moment the requirements have been finalized. As a Civil Engineer my Father was always bemoaning the unrealistic requirements forced upon him by “dreamer” architects, a feeling that I suspect has dominated construction since the pyramids. I recall many a tale of tense ad-hoc negotiation on the construction site, and even the removal of troublesome architects from site entirely!

IT projects share a great many similarities with Construction, although a Civil Engineer has the advantage that she can point out the very obvious real-world difficulties (“You want me to build a roof of glass that large without any supports to spoil the view?”) whereas IT implementors often suffer from the perception that everything is easy. The progress in our field has been so rapid that our customers are used to apparent miracles, and of course there’s always the potential for distrust to arise due to the often large culture gap between the “Geeks” and “Normal people” – and that’s nothing compared to what people often think of Vendors and their dastardly sales people!

It is well known that IT projects can fail to meet expectations, and indeed some of the statistics on IT project success make sobering reading. Plenty of studies have shown the criticality of requirements setting in project success and indeed some practitioners (for example in the “Agile” and “Customer Development” movement) have gone so far as to remove what they perceive to be a very error-prone requirements setting step from their project entirely.

Unfortunately most large companies have purchasing processes which require the business to decide what they need, and then go out to the market to get the “best” solution from what they hope is a selection of competing solutions via a Request for Proposal (RFP) or other formal process. The need to maintain fairness to all concerned means the RFP process if often very rigid and leaves little scope for modification of the requirements in the light of the reality of implementation.

Therefore we are faced with a situation where a potential user’s desire to get the best possible solution forces them into a situation where they have to place almost total reliance on their up-front requirements setting process, an “Aim, and hope” approach which would be abhorred in another sphere of corporate activity. Who would design a process which explicitly prevented feedback from influencing the initial conditions?

And yet, we are here – to paraphrase Winston Churchill, the RFP process and it’s reliance on Requirements Specification maybe the worst way of purchasing systems but in today’s corporate environment is better than anything else available. In the spirit of making the best of the circumstances, this article will take the opportunity to make some suggestions and point out common pitfalls which so many implementation teams fall into.

The problems of requirements and the resulting issues in the RFP process are all the more critical for ELN projects because:

  • The ELN replaces an existing paper process, which through the mists of time is often badly understood in itself.
  • The term “ELN” can cover such a wide range of functionality and domains that in itself it is a foundation for confusion.
  • Most of the current ELN offerings are still only “first generation” solutions which come with their own set of problems.

Thus an issue which bedevils all IT projects is often the founding cause of ELN project failure and requires particular attention.

Apparent “Solutions” which often aren’t

The difficultly in requirements capture/generation are apparent to anyone who has participated in such a project. A common approach is to hire a consultant, and when you get the right one they can single-handedly turn the situation around, although it does require the customer to listen! With apologies to the great consultants out there, the presence of a “consultant” leads to groans in the vendor community because (fairly or not) consultants:

  • Are sometimes viewed as having an interest in creating long running complex projects, rather than quick productive wins.
  • Despite having lots of industry experience, seldom get a long term view of a project. Often their involvement is restricted to the purchasing process, after which the vendor takes over. This gives them a rather one-sided view of the process.
  • Need to get paid and that means making sure the customer is happy. Unfortunately sometimes the customer needs to be told some uncomfortable truths which might lead to them being “fired” as a customer – something a vendor with a large customer base can do, but very hard for a solo consultant to do.

Another approach is to “Stick with who/what you have and know” – for example, if you have an established implementation of SAP or a Document Management package, those solutions might be bent or tailored to meet the new requirements. Unfortunately this doesn’t remove the requirements generation pitfall, and leaves you with an expertise gap as domain-specific solutions ideally come with a vendor who spends their days working on a particular area, who can bring that expertise to bear both in the solution itself and the implementation process.

Some companies will already have an “ELN” deployed in one area, and there is a temptation to view this as being suitable for all kinds of science. This is a sad outcome of the rather generic “Electronic Lab Notebook” term, and is one of the primary reasons why we prefer to avoid the term in day-to-day use; “science” is by definition a very varied activity and you can’t assume that just because two different departments use the same Paper Lab Notebook, that a single ELN will work well in both places. Often all these groups will have in common is they both call themselves “Scientists” and work for the same company – hardly a basis for a common toolset.

It is important to note that these solutions – consultants, re-use of a horizontal tool, and a common ELN across multiple disciplines – aren’t in themselves inherently flawed and can indeed lead to a successful project. There is however a risk of viewing them as the solution to what is at heart a very tricky problem, and project teams who think they’ve somehow reassured themselves of success are often painfully brought back to reality. As the Financial Crisis has taught us, risk doesn’t go away by magic and sometimes the very approaches we take to remove it in fact just increases it, more dangerously so because we’ve stopped being sensitive to it.

Perhaps as Andy Grove says, “only the paranoid survive” and the ultimate key to project success is the recognition that any solution to risk reduction has the potential for problems in itself, often in ways you least expect it.

A Modest, Sadly Unrealistic, Proposal

This article has presented a bleak assessment of how most teams are forced by circumstances to approach their ELN projects, as well as pointing out some common pitfalls that requirements gathering processes fall into. Whilst the problems implicit in these approaches can’t be removed, I hope I have provided at least the opportunity for some reflection. In closing perhaps I might offer some suggestions which I know are unrealistic, but might one day mitigate the issues I’ve described.

One of the problems with an RFP process is the lack of feedback from the implementors; I am sure I am not alone in looking at some requirements and thinking “This project is doomed”. I for one would welcome the opportunity to answer some additional questions, such as:

  • “If you could remove 5 of our requirements what would they be and why?”
  • “What are the most expensive/troublesome requirements listed above?”
  • “Which of these requirements do you think we don’t really need, based on your experience of similar projects?”
  • “What are we missing?”
  • “If you were us, what are the three things you would be most worried about going forward?”
  • “Please rate our chance of success if we go with you, and if we go with another vendor, with reasons”

These questions would afford thoughtful vendors the opportunity to reflect and contribute their experience – after all, for all the conflict of interest that you might perceive in a vendor/customer relationship, a vendor only ultimately succeeds when their customer succeeds. Any vendor team is easily going to see ten times the number of ELN projects that any customer or indeed consultant will see in a year.

Sadly whilst all these questions are interesting I don’t know what project teams would do with this information! In so many cultures project managers are rewarded for following a process and thus any failure is blameless, any reconsideration a failure.

Other Approaches

One very interesting approach we’ve just experienced was where the prospective customer held an RFI (non-binding Request For Information) process which was rather like an RFP but held outside a commercial purchasing process. Crucially the RFI submission and scoring was then followed up with a 1 hour feedback meeting between the customer and the vendor team which allowed for a lot of constructive discussion which no doubt benefited both sides.

We have had good results from projects which have used a Six Sigma methodology with plenty of contributions from all parties – end users, management, IT departments, and outside vendors. This approach tends to be too “heavy” for smaller companies but has delivered great results in larger companies where Six Sigma is part of the culture.

This illustrates the final and most important point: a successful solution is the product of a partnership between everyone involved, and even the largest most process-driven companies reinforce this in their process. Perhaps the greatest danger lies in taking a very formal approach in the purchasing process without counterbalancing that with an up-front listening process – a trap that growing companies often fall into as they formalize their purchasing process without having developed the experience and resources to learn from themselves and the rest of the industry.

What gets kept in Informatics Systems, and where?

Not all of the “Stuff” sloshing around the lab is the same, and distinguishing between them helps tease out the best place to store things. We use a simple Triangle Diagram (originally proposed by John Trigg of PhaseFour which really just tries to point out that stuff is related, but it’s at different levels of abstraction:

51DC4C72-5A54-4E90-B9D7-BA9AA67F3E31.jpg

It is quite hard to draw definite lines around things, but I think most people can appreciate that a raw data dump from an instrument is somewhat different from a report to management, or that an experimental write up in word is different from some tabular data in a spreadsheet. The differences between the levels come out in:

  • The software that’s used to read the file and interpret the content. Some will require very specific software (e.g. from an instrument vendor), but a PDF or text file can be read by many different things.
  • Who might be interested in the data. Again, some files are useful to anyone (for example, a report) but some only useful to certain people with specific training.
  • How long your company might want to keep the data, and indeed how long you are realistically able to keep the data. Typically the lower you go, the harder it is to keep something, so if you feel it’s business critical you really need to pay attention to the formats used.

This differentiation can really help in ELN System design. Partly it draws your attention to what needs to be stored in the ELN (typically the “Experiment” write up level), and what can be left in systems e.g. a database or a file server, which can be pointed to from the ELN.

Not everything needs to be stored in the ELN, and indeed it would be unrealistic to expect to be able to do so. The important thing is common keys so you can offer the user a link to more information, and the advent of web-based systems has made this level of “integration” so trivial one sometimes feels a bit of fraud describing it as such.

By building on the storage tools you have in place, and focusing an ELN on Experiments, the resulting system is cheap to run, costs little to acquire, and results in little disruption to existing practices.

You can read the final part of this series here.

Breaking down ELN functionality

Most R&D organisations have more than one scientific discipline under their aegis. These groups will have developed their own suites of IT tools to help them do their work, everything from common desktop infrastructure to instruments and specialist tools. Some of these might be common to other groups, some will be very specific (and often unknown to anyone else). Meanwhile, the corporate-centric record keeping functions have remained in the commonly used Paper Lab Notebook.

B05B10C3-C321-40E5-BAB8-1412EFDE9143.jpg
“Broad” and “Deep” functions

The “ELN” question often arises when companies buy more and more IT to support the science, yet the only record keeping option is the paper notebook – a situation that generally arises because of the patent/legal issues around lab work in Discovery. Broadly, the the more computers you use, the more the paper notebook sucks – and to add insult to injury whilst you can search the entire planet using Google, the paper notebook is very much a “Write only” device!

The challenge for project teams is how to replace the paper notebook, and this is where splitting out what you mean really helps. Unfortunately most vendors have tried to expand the definition of “ELN” as far as possible which turns any project into a high-risk “do it all” venture, with an associated price tag.

By focusing on either the improvement of support for a particular niche or on replacing the paper notebook’s general record keeping process, projects can easily build on what’s already in place and achieve faster, more predictable ROI. Later projects run according to the same framework will by definition build on what’s already in place so there should be little scope for missed opportunities.

Vendor-driven approaches to use a single product for more than one area bring increased risk and often the promised cost savings are overwhelmed by the costs of replacing perfectly serviceable existing tools.

The next post examines what is stored in ELN systems, which is another useful way of looking at the ELN problem.

How can we do ELNs safely?

If the traditional large-ELN model no longer fits, what options are there for organisations who still need to find a better replacement for the Paper Lab Notebook? Fortunately, there are a number of concepts which have grown out of ELN deployments outside of the Life Sciences industry (which never had the money to afford “ERP for R&D”-style approaches) which can allow organisations to tease apart their requirements and clearly identify what they need.

Using these approaches, organisations can often achieve swift return on investment (often in a matter of months) for minimal outlay and vastly reduced risk. The more a project can focus, build on what’s already in place and solve the problems that need solving then it will by definition involve less of everything – less requirements, less code, less consulting, less change, and less money. Smaller, quicker projects are vastly less risky than larger ones, are easier to get approved, and allow the organisation to react easily to changing business circumstances. By building a culture of “quick wins” they lay the foundation for further improvements in tools.

The two models which help tease part the “ELN” functionality are relatively straightforward and indeed most thinkers in this space have their own versions. Their real power comes when they allow project teams to understand what they have and what they need, and develop a road map such that the future is protected even though the initial project may not have the widest possible scope.

We’ll examine each model in a separate post, first looking at ELN functionality and then what gets stored.

On to the next post in this series

ELN 2.0 Vs ELN 1.0, in the new world

The problem with the term “ELN”

One of the problems that has confronted the “Electronic Lab Notebook” industry for a long time is the ambiguity of the term. Many a consultant and vendor has attempted to hijack the term with their own favored definition, complete with impressive diagrams in outdated and expensive reports.

ELN 1.0 started from the perspective of the suppliers – be they the vendors, or the consultants, or indeed the internal project teams. It was very much a “This is what we think you need” kind of approach, which led to a lot of worthy complexity which looked good but in practice got in the way of delivering the promise.

(For full disclosure, I was very much part of that group, and indeed the very first commercial ELN was written by me and others for Eastman Kodak – it was very functional but very complex. Then I led a Management Buy Out which created Amphora, and we had our own mini credit crunch which forced us to build something people could buy and use, not what we thought would be a good ELN!)

Today’s ELN 2.0 needs to focus on delivering value, and that means laser-like focus on the end user’s problem – which starts out something along the lines of “Since we bought all these computers, this paper notebook doesn’t fit with the way we work”.

ELN 2.0’s “End user first” approach doesn’t mean you end up with something that “isn’t a proper ELN”. What it means is you start focusing on the end user’s problems, prioritising them, and starting with the biggest wins first. Which is exactly what you need to do when there isn’t so much money around and everyone has become very risk averse. If we can do 20% of the work and get 80% of the benefit, doesn’t it make sense to do the 20% first and then look at where we want to spend our money next?

Interestingly this is directly against the interests of those who feel that a large, “fully functional” ELN is the only proper ELN. If we can deliver most of the benefit without the complexity, there’s no motivation to look further. So they have to bundle it all together and make the complex stuff appear necessary to achieve any kind of return. (for those who have attended my workshops, this is the “Toaster Problem”.

Differing needs

Some disciplines clearly benefit from a science-centric working environment which supports their niche requirements for example Medicinal Chemistry, and the considerable number of ELNs targeted at this sector is proof of the value that these solutions bring. The ability to draw structures and reactions, calculate properties, and structure/reaction-based search demonstrably increases the productivity of those scientists. Ironically, these solutions often don’t replace the Paper Notebook which is still required due to the concern that niche science-centric tool cannot provide adequate long term legal protection. However their value is only slightly diminished by the requirement to cut & stick into a Paper Notebook once the experiment has been completed.

For some scientific disciplines a gradual investment in IT tools, starting with a fairly typical desktop computer and then expanding into niche applications has provided them with all the tools they need to do their work. Once you deal with the legal issues, a lot of Discovery research looks a lot like any other kind of knowledge work and there’s a massive number of tools, sometimes already available to people for no additional cost, which can support that work. Indeed, one of the reasons the Paper Notebook is no longer suitable is that they are actually using those IT tools, and as a result a paper-based record keeping process is an unproductive overhead. Microsoft Office might not be blessed by the consultants as an ELN but it surely is the repository of more scientific thought and data than any “fully functional” ELN; that some products claim their similarity to, or integration with Office just reinforces the point! In these cases, project teams need to have a good answer to the CFO’s question “Why are you spending $1,000 a head to make Office harder to use?”

It is interesting to compare the health of the Medicinal Chemistry ELN market with the Biology ELN market, and indeed the discipline-neutral ELNs. Some have postulated that “Biology is next” and the approaches that work for a relatively homogeneous Medicinal Chemistry market will guarantee success in Biology. The implicit assumption is that Biologists have just been waiting for the ELN Gods to come and rescue them which rather implies that there are no biologists able to innovate in the same way as the chemists who developed the first Chemistry ELNs! A more realistic assessment might be that Biology is different – much more heterogeneous – which means the rise of a single Biology ELN is very unlikely. The adoption of the Biology-centric ELNs seems to be proceeding at a departmental level rather than the mass rollout to 100’s of scientists.

One size doesn’t fit all

Historically the ELN industry has been pushing a “One size fits all” approach, perhaps more due to the agendas of IT departments and suppliers. These projects are necessarily large, complex, and of course come with an associated price tag. With increasing size, complexity, and diversity of users also comes increased risk, and the success rate of such heroic endeavors has never been good. Projects of this type, which were always hard to justify anyway, are increasingly out of step with the new commercial realities. We just can’t afford to waste so much money stroking our “big project” egos – in today’s world, spending unnecessary money ultimately means there’s less money to spend on our own salaries.

If we want an ELN industry that’s healthy and can hold it’s head up high, we have to focus on delivering value, in a way that is acceptable in today’s environment. The subject of the next article in this series.