The ticking time-bomb at the heart of our big banks' computer systems

After two hours waiting to get through to the HMRC helpline she learned that the fault was with her Scottish bank.

The part-time call centre worker from Ashby in Leicestershire turned out to be just one victim of the more than half million transactions that 'vanished' from RBS's IT systems last month. While Smith was forced to put off buying essentials for herself and her three-year-old son, on the same day thousands of bank customers around the country found themselves unable to pay for the petrol they had just poured into their tanks while others were unable to pay household bills.

Last month's outage - which also involved NatWest, Ulster Bank and Coutts customers - was of course not the first time that RBS was in the firing line for shortcomings with its obsolete systems. So how did we get here, what do these incidents say about the state of our high street banks and is enough being done to avoid similar outages in the future?

Experts have for years been warning that the legacy systems of high street banks - some dating back to the 1960s - are an IT disaster waiting to happen. Systems built on costly and unwieldy platforms regularly buckle under the strain amid warnings over vulnerability to hacking attacks.

Last year the 79 per cent taxpayer-owned RBS paid a record £56 million fine following a much larger systems outage in 2012 which left millions alienated from their cash. In 2013 a further glitch left customers unable to pay by card or withdraw cash on one of the busiest shopping days of the years. Shopping trolleys were abandoned at check outs.

Those failures - which led to regulators reprimanding the bank for nearly crashing the country's entire financial system - forced RBS chief executive Ross McEwan to apologise for "unacceptable" technical faults and to admit that the bank would have to put right decades of under-investment.

Last month's glitch at RBS - a week after chancellor George Osborne unveiled plans to return the bank to the private sector at a loss to the taxpayer - led to MP Andrew Tyrie, the chair of the Treasury Select Committee, writing to McEwan for reassurance that such incidents would not re-occur.

But that hope seemed increasingly forlorn, after the bank's chief administrative officer Simon McNamara admitted he could not guarantee that technical failings would not re-occur. Clearly chickens are coming home to roost, but what are they? And how were they hatched?

Rewind 15 years to RBS's acquisition of NatWest in 2000. Although the integration of the two banks was lauded by City analysts for huge cost savings, Ian Fraser's acclaimed account of the demise of RBS Shredded recounts how a decision was taken at an early stage to ditch NatWest's more advanced computer systems and migrate all of the enlarged group's IT onto RBS's smaller IBM-based platform.

The reason? RBS's mainframe system was considered more adaptable to new types of transactions and seemed the most cost-efficient solution. But that decision to go for the quickest and cheapest fix, as well as the bank's previous and subsequent lack of investment in IT, has come back to bite it.

Although RBS has the worst record IT failures, there have also been well-publicised outages at Lloyds, the Cooperative Bank, TSB and the Bank of Scotland.

And the UK's banks are not alone. The technological meltdown narrative is mirrored worldwide in the sector. Last year banks in Europe spent an estimated £40 billion on IT but only £7bn of that was investment in new systems: the remaining £33bn was spent patching and maintaining increasingly creaky and fragmented legacy systems.

The big players throughout the developed world (with the exception of countries such as Turkey and Poland whose banks were slower to install computers in the sixties and seventies and, as a result, have fewer legacy issues to cope with now) use computer systems that have been built up and adapted over several decades.

Acquisitions have led to compromises being found and new product launches have led to bolt-on systems being patched onto layers of other complex systems surrounding the "legacy core" - typically a 1960s mainframe which carries out the bank's central ledger functions.

A report by Deloitte from as far back as 2008 said that "many banks have now reached a tipping point where the cost and risk of doing nothing outweighs the cost and risk of taking action". And yet, seven years on, little has since changed.

According to Chris Skinner, author of a leading book on digital banking and chairman of the Financial Services Club, banks don't want to take the risk of replacing their "groaning" legacy systems because any serious outage would lead to more fines from regulators and could even put them out of business. Because the stakes are so high, there has been a lack of leadership in the industry to tackle the issue.

But Skinner also has some sympathy with the banks for their reluctance to replace their core systems: a process which he describes as being akin to replacing the cement foundations for steel ones, with people still in the house.

"The banks have to change but no CEO has the stomach to take the risk of installing a new system because it could bring down the bank if things went wrong," he said. "Such a migration would need a five to ten year plan, so the benefits would be far from immediate."

Rather than taking the plunge and replacing entire systems there is now a move among some multinational operators to set up new smaller banks unencumbered by legacy issues. A notable example was the successful launch in 2013 by BNP Paribas of its new Hello Bank! in four European countries.

While most banks keep their hardware up to date the opposite is true of their software according to veteran capital markets analyst and blogger Richard Smith.

"Software in the industry can be very ancient indeed, and if the software is very old it doesn't really matter how new the hardware is," he says.

Many of the designers of the original bank software are now retired or dead and this loss of knowledge about their own core systems has been exacerbated by outsourcing data operations overseas, principally India, to reduce costs.

"Cheap inexperienced graduates were employed overseas and knowledge was lost in the UK," observes Smith. "The banks decided to go for cost reduction rather than reliability and maintainability and they now have an accumulating and accelerating problem brought on by the loss of that knowledge."

"The problem was evident to the far-sighted 15 to 20 years ago but it has now become a runaway problem with an awful lot of momentum and it's getting worse."

Along with other European and American banks, the UK's high street banks were early adopters of technology, but that embracing of computing power in the 1960s is now hampering their development, believes Smith. Faced with a choice between implementing the potentially dangerous "all or nothing" solution of a complete system replacement or a less risky incremental approach, most banks go for the latter.

Nevertheless, Smith is not entirely pessimistic. The reluctance of the old world retail banks to grasp the nettle of investment in new core systems is now giving the new challenger banks - who can launch with brand new, more reliable software - the opportunity to grow market share (see panel).

A particularly wasteful aspect to the situation, Smith laments, is that UK retail and investment banks write off hundreds of millions of pounds each year in software projects which are abandoned because management are too fearful of the consequences if things were to go wrong during or after installation.

A similar view is held by Frances Coppola, a former RBS systems analyst, economist and blogger who told the Sunday Herald that, although the banks have invested heavily in IT systems in recent years, this has been done by wrapping new applications around their legacy core.

"The individual systems are working: the problems occur in the interfaces between the systems which are being held together by glue, strings and sellotape," she said.

"The high street banks generally prefer to put money into front end systems which customers notice and they have not been spending on the infrastructure which underpins it. From the outside it makes the kitchen look pretty but they haven't been investing in the plumbing."

Replacing core systems is highly risky but at some point the banks will inevitably "have to bite the bullet", believes Coppola.

But, despite a call by the Bank of England's chief economist Andy Haldane in 2013 for banks to start migrating away from their legacy systems, Coppola says she has seen little evidence of an appetite among bankers to start the process any time soon.

Part of the growing problem, Coppola believes, is the dramatically different expectations that customers have now compared with 50 years ago.

When the first mainframe systems were installed in the 1960s, banks opened at 9 and closed at half past three. There were no ATMs or card payments and the payment system closed down entirely at weekends and on bank holidays.

Customers now expect access to their accounts via websites and mobile phones 24 hours a day, 365 days a year, and yet, beneath the layers of software added over the decades, the underlying core system remains the same.

Ronan Lynch of the Ireland-based Lafferty Group, which provides research and advisory services for the financial industry worldwide, has some sympathy with the plight of the high street banks as they struggle to balance regulator demands for data (for stress tests and anti-money laundering checks) and new capital requirements with the need to keep up with the new digital services that customers now expect.

"The banks would like to move to new systems but it would be like trying to change the engines on an aeroplane while it is in flight," he says. "Because of the need to back up data continuously while processing payments, calculating interest on a daily basis and so on it is almost impossible to stop everything to migrate to a new system."

Ever increasing competition from new internet start-ups looks set to be the main driver of IT improvements for the UK's dinosaur high street banks in the years to come. If, as the government and regulators hope, that process hastens the break-up of the sector - for too long dependent on a small number of "too big to fail" banks - in favour of a multitude of smaller banks that are more agile and technologically robust then more consumers can live without the background risk of being left high and dry at the check-out.

The move to mobile banking that is leaving High Street banks behind

A report last month found that mobile banking has for the first time eclipsed internet banking and more traditional services such as visits to the High Street branch.

The report, from the British Bankers' Association, found that banking apps were used 10.5 million times a day in March, overtaking the 9.6m daily log-ins to internet banking services. In a typical week, £2.9 billion is now being transferred through banking apps.

Last month, Durham-based Atom Bank - which claims to be the UK's first fully digital bank - was given a licence by regulators and next year is expected to see the launch of Starling Bank, another digital-only branch-less bank.

Anne Boden, Starling's founder and chief executive, says that after a 30-year career in banking - including a stint as head of global transaction banking at RBS - she had come to the conclusion that "the best customer experience can only be delivered by starting from scratch".

"The traditional banks are in a very difficult position whereas we have the luxury of being able to start from scratch with new technology," she said.

"People have changed the way they do their shopping and the way they buy music in the last few years. We now need to change the way they do their banking."

The bank's state of the art app will, according to Boden, allow a wider range of functions than the apps of the established banks and users could opt to receive advice on how to manage their weekly or monthly budgets.

"Somebody who has £300 in their account to last them until the end of the month could get advice on how to manage their spending, including telling them how much they need to set aside for standing orders, direct debits and so forth," says Boden.