Pharmaceutical research powers ahead

While the hype around the Human Genome Project is receding, drug-discovery and development teams are under mounting pressure to increase output and success rates, while reducing costs and research cycle times. Manuel Peitsch, Jürgen Basse-Welker and Pascal Afflard consider the importance of high-performance computing to pharmaceutical research and, in particular, to research work at Novartis.

While the sequence of the human genome is known, our knowledge of the function and physiological role of each gene and translation product has grown only modestly. Linking genomic information to the true biological role of each molecular entity is a vital means to achieving this understanding. This, combined with increased knowledge of disease pathways, will allow us to link diseases to genetic variations and defects. Based on this knowledge, medicinal chemistry experts, in their broadest sense, will look for chemical compounds or biologicals that can influence the course of a disease and design novel therapeutic approaches.

The constant evolution in discovery technology has caused individual research scientists and teams of scientists to produce an ever-increasing scope and complexity of data. Data production has become so prolific that the numbers of data points have increased exponentially. Understanding and using this data has been further complicated by the addition of new data types emerging on a regular basis from various disciplines.

Although computers have been incorporated into the area of biology, making significant breakthroughs in drug discovery by applying computer technology has been slowed by the complexity of natural biological systems and drug-discovery technology. Emerging technologies in large-scale biology and chemistry laboratories now call for a concerted effort, not only in data and information management, but also in data mining and the modelling and simulation of biomolecular and biomedical processes. The demands for such approaches will be increasing dramatically in years to come, providing drug-discovery teams with a new way to plan and design experiments.

Most of these approaches call for a powerful computing environment and, consequently, high-performance computing (HPC) is becoming an essential component of pharmaceutical research.

Explaining our HPC strategy

At Novartis, we have designed an HPC strategy that is able to offer a portfolioof computing cycles aimed at a broad spectrum of computationally intensive problems. Indeed, bioinformatics, chemoinformatics, computer-aided molecular modelling and design, computational pharmacology and toxicology, as well as clinical trial simulation, require a combination of a distributed computing system, Linux clusters, large shared memory compute servers, and special purpose hardware accelerators.

Our strategy consists of the right mix of computer hardware, bound together in a computational grid, which can be addressed transparently from all sites through a job submission layer that manages load and resource allocation across our intranet.

When we first analysed our existing hardware assets, we realised we had a very large computing capacity available in the form of our associates’ PCs. The Novartis Institutes of Biomedical Research had about 2,700 desktop machines lying unused for up to 90 per cent of the time (Figure 1). By scavenging these unused compute cycles, we would have access to immense amounts of raw compute power. We worked out what harnessing these machines might mean in processing-power terms. Such calculations are, of course, largely theoretical. Nevertheless, our peak performance estimation is that our existing 2,700 desktop PCs have the capability to provide five teraflops of processing. With ten times as many PCs harnessed, we believe we would have well over 50 teraflops, which would put us near the top end of the world super-computer rankings.

Consequently, the first step towards implementing our strategy was to harness this capacity so that the problems we wished to solve could be run in a highly distributed and secure fashion. If we could do this, we knew we would have an immensely powerful computing device at little incremental cost.

The PC grid: from pilot to implementation

In order to access so many potential cycles, we needed to implement a consistent environment on every participating desktop PC. Here, our investment in consistency paid off. We were able to exploit two activities that blended together nicely, namely, combining the potential of the grid with a former corporate decision to standardise our PC environment. We have invested in approximately 63,000 new PCs (from laptops to desktops) in a coherent company-wide approach, providing a very limited choice of hardware and a single software image across all geographies and departments. One advantage of this is that wherever you are working, you will have a recognisable system. This is of enormous assistance as without these standardised PCs, realising our grid virtual super computer would be far more challenging.

To evaluate the feasibility of this approach, we started a series of pilot projects and selected the United Devices Metaprocessor to implement our PC grid. In an initial productive system, a grid of only 560 PCs was linked up to achieve a 1,200-fold acceleration of the simulation of the interaction between a protein structure and around 320,000 small molecules1. What was once a prohibitively time-consuming project is now becoming a routine tool in drug discovery.

Today, we have rolled out the grid software to all research PCs and are running both a productive and small test environment (50 PCs) in which new applications are tested before productive rollout.

Our experiences so far

Our associates who use the grid software cannot help but be amazed by its possibilities and future prospects. These scientists can now think about addressing computational problems that they never thought they would have the raw power to address, even a year ago.

As for the practicalities, even issues like job scheduling have been made simple. Essentially, if you are authorised to run jobs on our grid, you submit these to a central queue and they are executed. So a user who wants to do a large computation simply places the job in a queue and waits until the results are returned. It is that transparent.

One of the early issues that concerned us was whether it would be difficult to describe problems in ways that were computationally appropriate for a distributed grid. If we could not break down the computations into units that could be sent out and shared across over 2,000 processors, then the whole initiative would not be of use.

Of course, it is not always straightforward. While many scientific modelling, simulation and mining algorithms can be easily split into small work units – which need a small amount of data, but run long calculations – others need a more profound reorganisation of the work process to take advantage of the grid architecture. In these cases, we needed to revisit algorithms, particularly those that had been written before the advent of parallel computing. In these areas, especially in statistics, we found that a degree of rethinking and reworking was needed to parallelise those algorithms. Once this was done, we could parcel out the computations to our grid.

One important aspect of algorithm development or rework for the grid architecture is that one might agree to a sub-optimal process given the availability of this large raw computing power. While one would optimise the process and algorithm to squeeze every cycle out of a smaller powerful system, the availability of so many grid nodes can easily substitute for sub-optimal processes. For example, running a specific algorithm on a grid might be only 50 per cent as efficient as on a large compute server. But the problem with large compute servers is that you may not be able to afford much more than 256 central processing units (CPUs) because of the sheer cost of the hardware. If you have got 2,700 PC CPUs (Pentium 4), and even if you are 50 per cent less efficient, it still means you are around five to ten times faster than with 256 CPUS on a dedicated server. Our conclusion is that while you may well accept some computational inefficiency, this will not matter with so much brute force behind you.

The largest issue we are facing is the idea that some problems cannot be ported to the grid. We are challenging our teams about this and have asked them to come back with possible solutions, bearing the above comments in mind. We have yet to discover the limitations of where grid computing can be applied. Indeed, we are increasingly convinced that nearly every computational problem can be parcelled out in a suitable way for computation to be undertaken on the grid. However, the future will tell how right or wrong we are.

In order to make maximum use of the grid, we had to change the power-off policy for the PCs. Indeed, our associates were asked not to turn off their PCs overnight and at weekends to allow maximum use of our computers.

A final issue is related to commercial software providers, who have yet to integrate the grid concept into their licensing models. Indeed, many software packages are based on per CPU or node licences, which represent clear limitations when many grid nodes are to be equipped with such software. The consequence is that we are now running only proprietary and free software on the grid, and are in conversations with various software providers to address this issue. All things considered, licence costs are prohibitive; it is cheaper to write your own applications and implement either well documented algorithms or take the opportunity to rethink your current approach and invest in next-generation algorithms and applications. The advent of the grid might have an important impact on software providers and change their business model more than any previous event in the IT community.

Look and learn

We needed to ensure that our PC users (whether desktops or laptops) understood that these systems belong to the company. They are corporate not personal possessions that can be allocated as Novartis sees fit. The relevance of this applies to placing the metaprocessor on each machine and exploiting unused computing cycles.

Another lesson we learnt is the importance of a champion willing to start with a small-scale, controlled pilot or experiment that is contained in its impact should anything go wrong. Demonstrating what can be done is much better than talking about it or presenting it. Showing people that a task that usually takes eight days can now be done in half a day has an amazing impact. It breeds interest and involvement, especially when there is neither a paradigm shift nor major change needed to be able to exploit the grid.

The grid has enabled scientists and IT professionals to think up new ways to answer problems they never thought could be solved computationally. We are now doing computation in chemistry and biochemistry, which we could not have afforded a couple of years ago. Too many difficult computational problems in science are not being tackled because the computational power is not available. We have proved that it is – to our competitive advantage.

What the future holds

Today, our grid capabilities are not part of the standard image distributed to all our PCs. Instead, the United Devices Metaprocessor is an additional software element – not unlike a specialised application – which is added once the standard image has been installed. We have recently started a project to incorporate grid functionality into the standard image.

Those 2,700 desktop PCs not include the large numbers of laptops that also exist. It is our intention to harness in the future. But because they are mobile, we have some additional issues to resolve before this is practical, including the management of battery power and job hibernation.

Within the overall Novartis organisation, we have a further 25,000 or so PCs that could be brought into the grid environment. We have started a project to expand our grid and grow it to include these machines.

Reference

1 Vangrevelinghe, E., Zimmermann, K., Schoepfer, J., Portmann, R., Fabbro, D. & Furet, P., ‘Discovery of a potent and selective protein kinase CK2 inhibitor by high-throughput docking’, Journal of Medical Chemistry (Volume 46, 2003, pp. 2656-62)

Manuel Peitsch is global head of informatics and knowledge management at Novartis. He can be contacted at manuel.peitsch@pharma.novartis.com.

Jürgen Basse-Welker is global head of operations at Novartis. He can be contacted at juergen.basse-welker@pharma.novartis.com.

Pascal Afflard is head of advanced infrastructure at Novartis. He can be contacted at pascal.afflard@pharma.novartis.com.