denotes premium content | Feb 7 2012 
Feature
posted 1 Mar 2006 in Volume 2 Issue 8
When disaster strikes...
There is no real ‘best practice’ approach to disaster recovery. Some organisations place more importance on IT, while others focus on their paper documents. Many have no plans in place at all. Kate Clifton examines the varied approaches of three organisations and looks at ways to reduce risk and cope when disaster strikes.
In recent years, it seems, natural disasters – such as Hurricane Katrina and the South-East Asian tsunami – have become bigger and more devastating, wiping out not just homes, crops and countryside, but also, in many cases companies and other organisations operating in the most severely hit areas.
While countries such as the UK, France and Germany are unlikely to be hit by earthquakes or tidal waves, disk failures, system crashes and burst water mains are far more frequent events and equally capable of wreaking havoc. And who would have predicted the Buncefield oil fire in otherwise peaceful
On top of that, a tougher regulatory climate means that information managers must know where all their data is – and protect it accordingly. “Managers are now, more than ever, forced to consider how protected – and retrievable – their documents are and how much of an element of risk they are prepared to take,” says Carl Greiner, senior vice president of infrastructure at analyst group Ovum.
People, he says, are beginning to ask more questions regarding disaster recovery, or what the industry prefers to label, ‘business continuity’. Yet at the same time, many do not act until it is too late.
Case study – Keeping Ark Group afloat
Ark Group organises conferences and publishes several business-to-business magazines in three core industry sectors – legal, information management and finance.
The organisation, which is headquartered in
The intranet is the backbone of
Complementary to this is the central document store, which is housed on a network-attached storage unit. Again, this contains important documents used by staff in all
Cause and effect
Like many organisations, Ark Group only formulated a disaster-recovery plan following a ‘near miss’, which almost caused a major headache in its day-to-day business operations.
This ‘minor disaster’ was caused by nothing more serious than a disk failure on one of the company’s servers. “But it made us realise how unprepared we were in the eventuality of something more serious,” says Shane O’Connor, head of IT at Ark Group. The resulting strategy, which took two years to develop, had to be as far-reaching as possible to cover all eventualities.
And the IT department needed to move quickly. “After a massive panic to get the intranet up and running again – and about four hours of down time – we looked at how we could improve things so that we could reduce future down time and plan for a more complete recovery,” says O’Connor. “We were very lucky that time and realised how important it was to begin planning immediately.”
In the initial stages of the planning process, a significant amount of time was spent weighing up the most efficient – and cost-effective – options for managing such a strategy.
O’Connor enlisted the help of a third-party services provider, Virtual IT. It recommended that
In addition, a copy of the intranet database is kept on a separate remote server based offshore. This is an exact replica of the live database and is updated using merge-replication technology (where the server captures any data changes to the source and target databases, reconciling any conflicts according to the user’s configurations). Changes to the main database are reflected on the copy within minutes, which provides another option to restore from. “Our strategy hinges on this,” says O’Connor.
In the event of an ‘all out’ disaster in the
“Our skeleton crew, plus any home and remote workers, would be able to use the offshore intranet while either the
Once the intranet is up and running, the IT department’s priority shifts to making all documents ‘share available’ again. The quickest method is using a series of large, portable disks to physically move the files from the data centre back to the office. This would take anything up to six hours, with the original intranet restore taking a further hour. Then, the
The individual elements of the strategy are tested regularly to ensure reliability and it is documented, both on- and off-site, so that almost any member of staff can take the reins in the event of an incident.
Case study – Business continuity at Norfolk County Council
The business-continuity strategy at Norfolk County Council is planned on three levels, using an ‘emergency services’ structure of:
-
Gold (corporate/strategic);
-
Silver (tactical/departmental);
-
Bronze (team).
The silver and gold plans are higher-level documents, providing a bird’s eye view of the team plans. The bronze-level plans contain more specific details for the disaster-recovery team on the ground and information that each team will need to help them deliver normal services during an incident.
“There is some overlap in the plans and this is quite deliberate,” says Diane Mortimer, corporate business-continuity manager at Norfolk County Council. “If we have a major incident, it is hard enough to work under the pressures it imposes without having to search through three plans for the information required.”
Continuity and disaster recovery for the council’s IT infrastructure is handled separately, and Mortimer is currently co-ordinating the various team’s business-continuity projects to improve their effectiveness and ability to work together if and when disaster strikes. This, says Mortimer, resulted from the introduction of a director-led group to look into this area of the business.
Mortimer was employed on a temporary two-year contract in 2002, by the end of which the organisation had a full incident-management strategy, an emergency-communications plan and a strategy for the main council building. Since then, Mortimer has been employed on a permanent basis, reflecting an increased emphasis on disaster and risk management at the council.
“This is not a project and we have plans for many other premises,” she says. “We have already started an incident-management strategy for schools and a template specifically for care homes.”
Already, the incident-management strategy has been tested – more than once. “[It is] usually due to an actual incident, rather than an exercise, although we have a full exercise and training strategy in place,” says Mortimer. “Luckily, our incidents so far have not moved away from the response phase, so time spent in planning the initial response has been time spent well.”
Technology’s poorer cousin?
Mention disaster recovery and many people in the industry will assume that you are talking about computers. But what about those organisations that have vast paper or microfilm-based files?
A 2003 survey by Fujitsu found a vast difference in the disaster-recovery plans of the cross-section of international organisations that it canvassed. More than half – 55 per cent – admitted that they had no disaster-recovery plan in place that covered their paper-based business documents.
That, perhaps, is why business is booming at companies such as Document SOS, which provides disaster-restoration services for business paperwork, books and art damaged by floods, fires or explosions. It claims that up to half of all critical business information is still held in hard-copy format – a factor often overlooked when organisations draw up their disaster recovery plans.
Yet paper is more durable than people think. According to Document SOS, although between 85 and 90 per cent of paper damaged by fire – and almost all water-damaged files – can be fully restored, four-fifths of existing disaster-recovery plans still do not include measures for recovering information held on paper or microfilm.
Furthermore, when disaster strikes, many people do the wrong thing because they simply do not know best practice. For example, by removing a file that has been submerged in water. It is actually better to leave it there to prevent any further deterioration that could be bought on by exposure to air, such as the growth of mould, until it can be properly dealt with.
Case study – The academic approach…
The
Under this plan, localised incidents – for example, a single office flooded by a burst water main – are dealt with by the relevant unit manager. More widespread damage from severe weather is covered by the ‘Emergency Protocol’ and ‘Safety Plan’, which take precedence over the disaster-preparedness plan during a major disaster.
Most of the university’s paper academic records are stored in a fireproof vault when not in use. In addition, the majority of its academic transcripts, which were created before the emergence of electronic storage, have been transferred onto microfilm and are stored in a fire and waterproof container. Most administrative records are also stored in similar vaults and cabinets.
The plan itself also provides extensive information on the use of air conditioning, fans and de-humidifiers to remove moisture from the air of paper storage areas. It also advocates that dry, damp and saturated records are separated and labelled.
The freezers used in the institute’s dining facility remain on standby, in case saturated papers need to be frozen in preparation for professional freeze-drying. In the case of a huge influx of sodden files, the plan also identifies off-site frozen storage facilities.
The plan recognises that fire damage is harder to deal with than water damage. It also advocates the duplication and offsite storage of all records – a vital component of the plan. Any documents that are recovered may need to be dry cleaned to remove smoke odours.
Making it count
Disaster-recovery planning requires a great deal of thought, work and preparation. And, like any other management activity, demands clever budgeting, an ability to maximise existing capabilities and the leadership to inspire people to take disaster recovery seriously.
Mortimer at Norfolk County Council cites a lack of resources as a chief obstacle in her business-continuity-planning initiatives. “I have found during my time in this area that there is no ‘slack’ in the system; we are already working to full capacity,” she says.
This was mirrored in a recent survey by Applied Research (commissioned by Symantec). Nearly half the respondents did not have a disaster-recovery plan in place and one-third of these blamed this on a lack of resources.
And while Mortimer has never had any problems securing management buy in: “It was there to start with and has increased as time has progressed,” – she acknowledges that despite recent disasters, many managers are “already drowning in bureaucracy”. For this reason, she says, you will sometimes need to use the stick rather than the carrot: “As far as management is concerned, unless it is happening right now, business continuity can tend to sink to the bottom of the pile.”
Greiner agrees. “Until people really understand how exposed they are – and do a thorough risk evaluation of the organisation – they won’t take this too seriously,” he says.
Cost, as always, is another issue. At
Organisational culture is key, as is keeping your plans as simple and user-friendly as possible. As with any change process, disaster-recovery planning requires clear communication, realistic goals and sensitivity to every department within the business.
By covering all bases in your risk analysis and speaking to people – at all levels – who will be directly influenced by an incident, a comprehensive and up-to-date plan that covers any eventuality should not be too difficult to achieve.
Business continuity according to MI5
The
According to the security advice posted on its website, nearly one-in-five businesses suffer a major disruption every year. It provides a summary of recommended planning and action points, while reminding organisations to rehearse any system or strategy that they set up, to ensure complete dependability. Its recommendations are summarised below:
What you can do:
-
Establish top-level ownership of business-continuity planning within your organisation;
-
When analysing risks, consider natural disasters, IT or infrastructure failure and terrorism (while your organisation may not be attacked directly, resulting industrial unrest or civil disorder could have a major impact);
-
Consider how each of these events could affect your business – for example, loss of communication, damage to a building or staff unavailability;
-
Estimate the resources you will need to maintain critical business functions;
-
Ensure your plan is simple, ‘crisis friendly’ (easy to understand under stressful conditions), robust, clear about responsibilities and authority, understood by everyone that needs to know, up-to-date, proven to work, regularly tested, and consistent with the plans of partner organisations and stakeholders;
-
In addition, think about a communication strategy to raise awareness and think about those that could be affected outside of your organisation (for example, customers, clients and contractors);
-
If you have no plan, consider what arrangements you could implement now to make your organisation more resilient (for example, a designated crisis-management team or off-site storage.
The full list of recommendations can be viewed at: http://www.mi5.gov.uk/output/ Page267.html
Floods
There are two types of water damage. ‘Direct’ damage refers to paper files that have been completely saturated, while ‘secondary’ damage refers to those that are affected by high humidity from flooding, resulting in damp documents.
When you plan…
-
Avoid basement storage where possible;
-
Raise the bottom shelf of storage at least 150mm off the floor;
-
Leave small gaps between shelving and walls, so that documents don’t come into direct contact with them;
-
Where possible, use acid-free storage boxes;
-
Always have a secondary (non-mains) source of electricity;Install flood alarms that can be monitored out of hours.
When disaster strikes…
Following a flood, you need to move quickly to prevent the growth of mould, which could occur within two or three days. First, of course, ensure the area is not ‘live’ from the electricity supply;
-
In addition to air drying and freeze drying, polythene sheeting can be used, as a temporary measure, to protect lightly-damaged documents. Any affected items should be moved to a dry, preferably cool room, starting with material on upper shelves;
-
Clean blotting paper can be used to inter-leave sheets of paper (as temporary first aid), although items on coated paper will require swifter attention;
-
Saturated documents should never be piled on top of each other and no attempt should be made to separate those that have become stuck together;
-
After the area has been drained, ventilate and, if necessary, use a de-humidifier until the humidity has been stabilised to a level of between 45 and 60 per cent. Monitor any ‘unaffected’ items, which may suffer longer-term damage if the humidity has been seriously altered.
SOURCE: The National Archives (www.nationalarchives.gov.uk/archives/advice/pdf/memo6.pdf)
Terrorist attacks/bomb damage
This threat is the most difficult to counter merely through planning, due to the unpredictable nature of targets. General guidelines recommend that organisations in ‘high-risk’ inner-city areas should consider depositing their documents in a secure repository outside of this zone.
The Metropolitan Police recommend that documents should not be left on tables or beside windows, out of hours, but in secure and fire-proof storage – preferably with a minimum of one window.
SOURCE: The National Archives (www.nationalarchives.gov.uk/archives/advice/pdf/memo6.pdf)
