News
seven year sunk cost fallacy – 50 foot Blockchain Attack
Enterprise blockchain never made any sense. A slow data store was never going to magically make your business or bureaucracy more efficient. Because magic doesn’t happen.
That didn’t stop the fierce and devoted pursuit of the dream, nor did he burn through millions of dollars in the process.
In 2015, the penultimate CEO of the Australian Securities Exchange, Elmer Funke Kupper, put a blockchain bee in the bonnet. “Every now and then something happens that could change everything. And this is one of those moments,” he told the Sydney Morning Herald. [SMH, 2016]
In 2022, ASX finally shut down its blockchain trading system project, after seven years of no results and A$250 million up in smoke.
The magic of blockchain
When I was writing 50 Foot Blockchain Attack in 2017, Blythe Masters’ Digital Asset Holdings company was in talks with the ASX to replace their aging Electronic Sub-Registry Clearing House System (CHESS) with… something descended from blockchains.
CHESS was launched in 1994. It works and is quite reliable. But this is thirty years of legacy COBOL code, and maintenance has long been trepidatious.
Unfortunately, Digital Asset has sold the ASX on the idea of a “digital ledger”.
The CHESS replacement system – it never had a real name, just “the CHESS replacement system” – was based on Hyperledger. ASX would administer this centrally. The functionality would be smart contracts written in DAML – a functional, formally provable and Turing-complete programming language – although DAML had not actually been completed when ASX accepted it.
In business computing, “smart contracts” are called “database triggers” or “stored procedures”. They are a nightmare, because they are very difficult to reason about or maintain, and they are subject to unexpected and disturbing effects. You only use them if it is absolutely necessary for performance.
Also cryptocurrency exchanges do not use blockchain for their internal systems. They run their trading engines on a conventional database system.
The CHESS replacement system used the wrong approach, with the wrong technology, from a company that did it I’ve never built a real system – not to mention a huge one.
Exhaust tour
Digital Asset spent January to June 2016 introducing a prototype system. ASX invested $17.4 million in Digital Asset in 2016 for 8.5% of the company and appointed them to build the replacement for CHESS in December 2017. [FT, 2016, paywalled; press release, 2017, PDF]
Funke Kupper left the ASX in 2016, but his replacement as CEO, Dominic Stevens, remained committed to the blockchain plan. Stevens said the blockchain system would “put Australia at the forefront of innovation in financial markets”. [BBC, 2017]
The prototype of Digital Asset’s blockchain system has been built and rebuilt, with increasing delays. Perfectly sensible and capable IT people were working on the project – I saw a presentation in early 2018 that impressed me with how sane and sensible it was – but it could barely deliver even the contemporary speeds of CHESS, let alone the promised improvements .
Other market participants hated the new system. It wasn’t working properly, they were spending a fortune building and rebuilding their interfaces, and they complained repeatedly to the regulator that the ASX was ignoring their problems.
Other participants were also unhappy that the project would make the ASX a new centrally controlled octopus, to a degree never achieved before. This is a blockchain project failure mode that I have talked about in detail chapter 11 From the book. Users called for the project to be delayed further, as it simply wasn’t ready to be released. [FT, 2020, paywalled]
The system moved from Hyperledger to VMWare Blockchain in 2019. That didn’t help matters. But ASX has launched Synfini, offering customers DAML on VMWare Blockchain as a Service! ASX has persuaded crypto firms DigitalX and Zerocap to run pilot systems on Synfini. [VMWare, 2022; ASX; AFR, 2022; Cointelegraph, 2022]
And blush
Finally in 2022, the ASX has a new CEO, Helen Lofthouse. She looked at this shaking zombie and killed it in the most effective way possible: she asked Accenture to write a report on it.
Even Accenture couldn’t see a way to profit from this thing and said it had to be killed. [ASX CHESS Replacement Application Delivery Review, PDF, 2022]
Accenture has identified four key issues:
- The blockchain system “introduces higher latency” to the ASX as a whole;
- Concurrent transactions are required, but this can cause conflict problems;
- Batch processing did not resolve the conflict issues;
- DAML was not suitable for batch transactions.
The blockchain replacement initiative has been declared pause on November 17, 2022. The ASX wrote off AU$250 million. ASX users themselves had spent tens of millions of dollars building interfaces to the broken system. [AFR, 2022]
The regulator, ASIC, has launched an investigation into how this all happened and whether ASX had breached the ASIC Act or the Corporations Act. [ACS, 2023]
Parliamentary inquiry
The ASX blockchain was enough of a fiasco to attract the full attention of regulators. A statutory inquiry published its report in April 2024: “Competition in Clearing and Settlement and the ASX CHESS Replacement Project: The CHESS Replacement Project is too important to fail.” [full report, PDF; executive summary]
The investigation found that ASIC and the Reserve Bank of Australia should have been more alert to the risk of the ASX making mistakes in the CHESS update and should have intervened sooner. ASIC has spoken of ASX’s “arrogance”.
To this end, the inquiry recommended that regulators get fully up to speed on the technology underlying market infrastructures so they can meaningfully audit systems such as CHESS as part of their regular ASX compliance checks. It also recommended promoting competition in the clearing and settlement sectors and focusing on the interoperability of CHESS.
The road ahead
A new CHESS replacement system has been planned until 2023 and is currently under consultation. The new system will be built by Tata and is expected to launch between 2026 and 2029. [ASX; Global Treasurer]
ASX took pains to reassure the world that the new system would definitely not involve any type of blockchain. [Reuters, 2023]