Germany is planning to roll out a system of electronic health care smart cards, as already deployed in France (Carte Vitale). The deployment is currently in a first phase of tests.
As usual, this type of system is using a PKI (Public Key Infrastructure). And every PKI is based on the use of a root key pair that signs the certificates and the revocation lists. Thus, the private key of the root Certificate Authority (root CA) is one of the most important secret of the system. Generally, this private key is stored in a Hardware Secure Module (HSM) that makes all the operations of certificate signatures, revocation list signature, … A HSM is a enhanced tamper resistant module that will stop to work when it detects an attempt to tamper.
The German system of course used such a HSM. Unfortunately, a voltage drop was interpreted by the HSM as an attack. It thus erased the private key. The normal procedure is to take the back up HSM, duplicate it and start again. HSM have special strict procedure to make back up of the secret keys on another HSM. Unfortunately, there was no such back up. The consequence is that the trial cannot anymore generate a new smart card.
Fortunately, this is only the test phase. For sure, there will be a backup for real deployment phase.
Root CA management, storage , and handling is an extremely complex task. Some companies (such as Entrust, Verisign…) have made a living of this activity. So if ever you use a root CA, either make a backup (and store ii somewhere in high security) or use a proven operator.
For more details, read here.