Blockchain: consensus protocols

This post is the third post in a series dedicated to demystifying blockchains. The second post described the difference between permissionless and permissioned blockchains. It also introduced the concept of validator. This post will present the notion of consensus.

A consensus mechanism or protocol ensures that all nodes of the system agree to a shared, approved state. In the framework of a distributed ledger or blockchain, a consensus mechanism enforces that all participants use the same state or version of the ledger. In other words, it is the mechanism that ensures that every entity agreed on the same transactions and that all copies of the ledger are identical. Consensus protocols are not new. They at the core of many distributed systems and mirroring systems. Nevertheless, permissionless blockchains introduced a new challenge. Not all participants to a permissionless blockchain may be behaving properly.

Two kinds of consensus

There are mainly two categories of consensus mechanism:

  • Lottery-based
  • Byzantine Fault agreement-based

The first category is sometimes called the Nakamoto-consensus in honor of the pseudonym of Bitcoin’s founder Nakamoto. The consensus mechanism elects the validator, i.e., the node that decides which is the next block to be appended to the ledger. The election is a lottery draw. The winner is the validator. Each new block requires a new draw. The selection through a lottery reduces the likelihood of an ill-behaving node to validate a forged block. The lottery does not necessarily follow an equiprobable distribution. Each mechanism has its own probability distribution favoring one given characteristics of the winner. Thus, each lottery-consensus has a different trust model. The Proof of Work (PoW), used by Bitcoin, is the most well-known mechanism. There are many other types such as Proof of Stake (PoS), Proof of Space, or Proof of Elapsed Time (PoET). Future posts will explore in details PoW and PoS.

The second category is based on Byzantine Fault Tolerant (BFT) system. BFT systems are designed to operate even if some participants in the protocol are failing. Failure may be involuntary (for instance, a participating node is out of order) or voluntary (for instance, an attacker controls the failing node). BFT employs voting mechanisms to decide the consensus. The used mechanism defines the trust model. It is usually well defined. The Practical Byzantine Fault Tolerant (PBFT) mechanism is the most well-known mechanism. A future post will explore PBFT.

Hybrid consensus mechanisms seem to appear mixing lottery with a pinch of BFT. Casper, the next consensus mechanism of Ethereum is a PoS with some BFT in it.

Lottery or Byzantine Fault Tolerant?

Lottery-based mechanisms are more complex and slower than BFT-mechanisms. Lottery-based consensuses are well fitted for permissionless blockchains. There is no control on the validators. Anybody may participate in the validation. Therefore, the lottery reduces the risk. In a permissioned blockchain, the validators are known. Thus, BFT-consensuses are adequate. Depending on the mechanism, the designer knows how many validators must be compromised or must collude to validate a forged block successfully.

Research on consensus is currently an extremely active field. Unfortunately, many consensus mechanisms are (too) young and their security has not been enough studied. The following chart illustrates relative age of major consensus mechanisms.

Conclusions

If you want to manage your blockchain, then you need to understand the corresponding consensus mechanism. It participates to the trust model of your solution.

 

Blockchain: Permissionless versus Permissioned

This post is the second one in a series dedicated to demystifying blockchains. The first post proposed a definition of blockchain. I intended that the topic of this second post would be consensus. The consensus is the cornerstone of blockchain. While starting to write it, I discovered that I needed first to introduce a fundamental characteristic of blockchain: permission.

Entities decide whether a block is valid and appended to the blockchain. They may be called blockchain nodes or validators. Validators are the pieces of software that determine which is the new block on the chain. In Sato’s vision, everybody could/should be a validator. Thus, his blockchain has no central authority. It is claimed that the blockchain is ruled by everybody (or nobody depending on your point of view). Bitcoin is a permissionless blockchain. This is the case for most cryptocurrencies and many other systems. Ethereum is another example of a permissionless blockchain. In a permissionless blockchain, users delegate their trust to uncontrolled, unknown validators under the assumption that the consensus mechanism does not allow a bad acting validator to cheat.

This delegation of trust is not always possible or desirable. Therefore, there is a second breed of blockchains that operate with a different configuration: permissioned blockchains. The validators are a set of finite known servers. A consortium manages this list following some defined governance rules. You may have noticed that the validators were not necessarily trusted. Depending on the chosen consensus mechanism, the level of expected trust may vary. The open source projects of Hyperledger offer many such permissioned architectures.

Which one is the best?

The advantage of the permissionless blockchain is that there is no (at least claimed) central authority. There is not a single point of failure that may be attacked. This advantage comes at a price: the consensus mechanism is complicated and/or extremely power consuming. It will have to be slow. Furthermore, it requires that the nodes have a robust method to validate a transaction. When managing financial ledgers, it is easy. Checking that Alice currently has the number of tokens she asks to transfer to Bob is straightforward. With more complex transactions, it may be less obvious. Would you trust an unknown validator to check whether your land deed belongs to you and to register it on a land registry blockchain? Or a copyright right? Smart contracts are not the golden answer to that issue.

The advantage of permissioned blockchain is that a set of entities that share a common interest in the fulfillment of the transactions can manage it efficiently. The validators have the authority and implement the complex validation rules that some use cases may be requiring. The consensus mechanisms are simpler and faster than the ones used by permissionless blockchains.

Many “purists” claim that permissionless blockchains are more secure than permissioned ones due to the absence of a central authority, arguing that the management of the validators is a weak point. As usual, the answer is more balanced. It mainly depends on the use cases. Some industrial use cases may benefit from permissioned blockchains. Personally, I would argue that the trust model of a permissioned blockchain can usually be more accurately defined than the trust model of permissionless blockchain. I have not yet read a convincing complete
convincing trust model of a permissionless blockchain.

Conclusion

Thus, a hyper-simplified definition: A permissionless blockchain does not trust nor know its validators whereas a permissioned blockchain knows all its validators but does not need to trust all of them.

Blockchain: A Definition

This post is the first one of a series dedicated to the blockchain. In the coming weeks, I will discuss many aspects of the blockchain. As some of my views may be perceived as pessimistic, a cautionary note is mandatory: I am a skeptical blockchain enthusiast. Blockchain has great potential but also many pitfalls. I hope that these posts will shed some lights on the blockchain.

The first step is to propose a definition for blockchain.

A blockchain is a secure distributed ledger.

Let us examine the four elements of this definition.

  • A blockchain is a ledger. It stores the complete chronological records of transactions. The transactions are combined in a data structure called a block. Each block is cryptographically bound to its predecessor, thus creating a chain of blocks. The blockchain is well suited for transactions and time series. For instance, Bitcoin records the exchange of bitcoins. Other types of information, for instance, graphs, are not necessarily well suited for blockchain. Nevertheless, many
    information can be transcribed in a set of transactions.
  • A blockchain is shared. Many entities use the same ledger. They may not all have the same access rights: Some entities may be allowed to submit transactions to the blockchain whereas other entities may only read these transactions. The use case defines the rules for access control. If the ledger is not to be shared, then probably a traditional database is more suitable than a blockchain.
  • A blockchain is distributed. No central server holds all the blockchain. Every node has the same complete copy of the ledger. And the nodes are connected through a peer-to-peer network. Therefore, the blockchain offers high availability and resilience. There is not one point of failure in the system.
  • A blockchain is secure. Each issuer signs its transactions. Each node validates every transaction according to validation rules that are defined by the blockchain governance. For instance, for a cryptocurrency, the validation of a transaction verifies that Alice owned the coins she transfers to Bob. In the case of a land registry or a supply chain tracking, the validation will be most probably more complicated.
    Once all the transactions of the block validated, the nodes engage in a consensus protocol to decide whether the block is to be appended to the blockchain. Once the consensus reached, all nodes add the new block to their copy of the blockchain. The consensus is the most complex element of the blockchain. Many consensus protocols are available. The most famous one is the Proof of Work (PoW) designed by Nakamoto Sato for Bitcoin. Mining is establishing the consensus for Bitcoin. In a next post, we will study in detail the PoW and other types of consensus. The consensus ensures that every node has the same ledger.
    Transactions are immutable. To alter an already recorded transaction, the attacker must modify the block containing the forged transaction, and also adjust all the subsequent blocks to maintain the cryptographic link. Furthermore, the attacker must trick the consensus protocol to vote the forged fork to be the valid one. Thus, it is reasonable to assume that the transactions are carved in stone. As a side note, immutability may become an issue in case of an error or if the “right to forget” is needed. What is in the blockchain stays in the blockchain.

This first post provides a broad definition of the blockchain. Next posts will explore technical elements of a blockchain.

Reference:

Nakamoto, Satoshi. “Bitcoin: A Peer-to-Peer Electronic Cash System,” 2008. http://www.cryptovest.co.uk/resources/Bitcoin%20paper%20Original.pdf.

Symposium on Foundations and Applications of Blockchain 2018

The University of South California (USC) will host on Friday March 9, 2018 the first Symposium on Foundations and Applications of Blockchain 2018.  Its program is available at https://scfab.github.io/2018/schedule.html.   Note the presence of Leonard Adelman at the discussion panel!  I hope to meet some of you there.

Full disclosure:  I am member of its PC.

NIST overview on Blockchain

There are not many excellent available overviews of blockchain technologies. Thus, when NIST issues a draft “Blockchain Technology Overview,” it is interesting to have a look. It is a 57-page document open for public comments.

I like their description:

Blockchains are distributed digital ledgers of cryptographically signed transactions that are grouped into blocks. Each block is cryptographically linked to the previous one after validation and undergoing a consensus decision. As new blocks are added, older blocks become more difficult to modify. New blocks are replicated across all copies of the ledger within the network, and any conflicts are resolved automatically using established rules.

The document provides a high-level overview of blockchain. There are not many detailed technical descriptions. The document uses the bitcoin structure and vocabulary as all blockchains would use them. Thus, a generic block has necessary a nonce (for the Proof of Work) as well as a Merkle Tree. I am sure that many blockchains will not have such elements. Similarly, it uses the terminology of mining nodes for the validators. For consensus mechanisms that are not Proof of Work, it is not suitable. The sections dedicated to consensus (section 4) and Smart Contracts (section 6) are too light. The golden nugget is section 9: Blockchain Limitations and Misconceptions.

Nevertheless, it is worthwhile to read it and potentially to comment. Knowing the NIST, I am confident that the final document will be a reference document.

Meltdown and Spectre

On January 2018, security researchers disclosed two attacks coined Meltdown and Spectre. These attacks bypass the memory isolation of modern CPU by exploiting side-channel attacks on hardware-based optimization features of these CPUs. Thus, Meltdown and Spectre can gain arbitrary access to confidential information in the memory of the computer.

Modern CPUs, so-called superscalar computers, do not execute anymore the instructions sequentially. They implement many hardware-based optimization techniques that modify the normal instruction flow. For instance, the CPU executes multiple instructions concurrently to keep the processor’s sub-units as busy as possible (See Eben Upton’s post). Thus, out-of-order execution speculatively executes instructions further down the instruction flow as soon as all needed resources are available. Thus, the CPU may execute an instruction before it is sure that the instruction is needed. If later the CPU determines the instruction was not needed, it discards the corresponding results from its registers. This mechanism is sound architecturally but not at the microarchitecture level. The cache memory still holds the discarded results. Unfortunately, for many years, security researchers have designed side-channel attacks that leak confidential information from the cache. Modern CPUs’ branch predictors attempt to guess the future control flow and, execute the instructions of the predicted instruction flow preemptively. If the predicted decision is wrong, the CPU discards the “results” of the speculative instructions if the prediction was incorrect. Once more, this mechanism is sound architecturally. Unfortunately, the results remain in the cache memory. Covert-side-channel cache attacks can retrieve them.

The attacks

The goal of Meltdown is to dump the kernel memory space from a user-space process. In a simplified explanation, Meltdown operates in two steps. During the first step, Meltdown entices the CPU to access the kernel space through out-of-order instructions. When the instruction flow reaches this execution point, it detects the violation and triggers an exception handling that blocks actual access to the kernel space. During the second step, Meltdown uses covert-channel cache attacks to retrieve the cached “inaccessible” data. Intel memory management maps privileged kernel memory in the user-space. Thus, kernel memory becomes accessible. The usual security assumption is that kernel memory is secure and not accessible on a computer without root access. Meltdown breaks the hardware-enforced isolation between kernel space and user-space.

Meltdown may affect any CPU using out-of-order mechanism and is OS-independent. Meltdown has been successfully tested on Intel x86, Intel XEON processors, and ARM Cortex A57. Meltdown was mounted on cloud containers, such as Docker, successfully. The software countermeasures use KAISER. KAISER is a software patch that prevents the mapping of kernel memory into the user space, thus thwarting Meltdown. The KAISER patch is available for Windows 10, Linux, MacOS and iOS.

The goal of Spectre is to reach information from another process. Spectre exploits branch prediction and speculative execution. It operates in three steps. During the first step, Spectre mistrains the branch predictor by repeatedly executing a given branching. During the second step, Spectre entices the branch predictor to mispredict the control flow. The CPU then executes the speculative code that should perform the “illegal” operations, such as reading unauthorized memory. As in Meltdown, the third step exfiltrates the cached data using a covert-channel cache attack. Spectre accesses from a given user-space the memory of another user-space. Spectre breaks the hardware-enforced isolation between processes.

Spectre has been successfully implemented on recent Intel processors, AMD Ryzen, AMD FX, and AMD PRO. Spectre was implemented on Windows and Linux-based OS. It was written in C and also in JavaScript. The countermeasure would be to halt predictive execution on sensitive execution paths. This is a difficult task as the current instruction set is not fit for that purpose. The alternative solution is to implement in the code mechanisms that reduce the impact of the leaked information (for instance, combining conditional select and conditional move. In other words, developers must be aware of the covert-channel cache attack and implement adequate countermeasures. Compilers may also implement some tricks.

As Spectre can be mounted with JavasScript, malicious adware may become the first exploits using Spectre in the field. Thus, browsers are receiving patches to mitigate the risk. The exploitability via JavaScript is worrying.

Google’s Project Zero released concurrently three vulnerabilities, coined variant 1 to 3. These three vulnerabilities are identical to Meltdown and Spectre. Variant 1 and 2 correspond to Spectre whereas variant 3 maps to Meltdown.

Conclusion

Meltdown and Spectre are not due to bugs. They are the consequences of a new breed of side-channel attacks exploiting information leaking at the microarchitectural level for speed optimization.

It is interesting to notice that Paul Kocher is one of the researchers disclosing Meltdown and Spectre. In 1996, Paul designed the first side channel attack. His attack disrupted the security of smart cards. Since 1996, side-channel attacks have been among the most prolific, complex fields of research in security.

We want/need the CPUs to be faster. Thus, silicon designer added these optimization features to go faster. Unfortunately, most trivial countermeasures would defeat the benefit. For instance, cache attacks may be defeated by randomizing or equalizing the access time, which would annihilate the purpose of the cache. New hardware architecture, as well as new instruction sets, will help to defend. Nevertheless, we have a new class of side channel attacks to take into account. No doubts that variants will soon flourish.