In December last year, the ATO’s digital systems experienced what Commissioner Chris Jordan named the “worst unplanned system outage in recent memory”.
PwC was appointed to investigate the outages, and the findings of that report, as well as others including the ATO’s internal review, feedback from stakeholders and an interim root-cause analysis by contractors Hewlett Packard Enterprises (HPE), were released in part today.
The ATO has reached a commercial settlement with HPE, the detailed terms of which are subject to contractual confidentiality.
“The settlement recoups key costs incurred by the ATO, and provides additional and higher grade IT equipment giving the ATO a world-class storage network,” Mr Jordan said.
Importantly for the professional community, Mr Jordan said the tax office is aware it needs to work closer with its partners such as software providers, tax professionals and the superannuation industry.
Mr Jordan said the full reports will, in summary, show that, among other issues, the turnkey service of data storage as per the 3PAR SAN provided by HPE failed. The findings will be released in full next week.
Further, the findings will show the fibre optic cables feeding the SAN were not optimally fitted; disk drives on the SAN had software bugs that made the stored data on the drives inaccessible or unable to be read and some monitoring features were not activated, including a “back-to-base” tool to report operating errors.
“The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience — a relatively small disk drive failure had a large impact — only 12 of some 800 disk drives failed, but they impacted most ATO systems,” Mr Jordan said.
The recovery was slower, Mr Jordan said, because some of the recovery tools required were stored on the same SAN that failed.