Post-Mortem Investigation (Feb 2016)

During the 'Turbulent Age' (06 Feb 2016 to 08 Feb 2016) of the King of the Ether Throne, a serious issue caused some monarch compensation payments and over/under payment refunds to fail to be sent. This web page explains the issue, the causes, the response, and the recommended solutions. It is currently in FINAL form.

Important Notice

DO NOT send payments to any contract addresses mentioned on this page - further refunds will NOT be paid. See the King of the Ether Throne home page for the latest ÐApp.

Contents

TL;DR

Sending ether from one contract to another contract - such as from the King of the Ether contract to an Ethereum Mist "contract-based wallet" contract - is quite likely to fail if implemented in the "obvious" way in the Solidity contract language due to insufficient gas. Luckily, all ether was returned to its rightful owners in this case, and a new version of the King of the Ether contract will be launched shortly.

The Issue

But First A Little Background

Ether is stored in accounts. There are two fundamental types of accounts - "externally-owned accounts" and "contract accounts". The "externally-owned accounts" are normally controlled by a human, whereas the "contract accounts" are under the control of a contract. The Ethereum Mist Wallet Client encourages Ethereum users to create "contract-based wallets" (that is, "contract accounts") to hold their ether. All Ethereum transactions such as payments and calls are always started by an "externally-owned account" - if you pay someone from a "contract-based wallet", your "externally-owned account" must have told your "contract-based wallet" to do so.

The King of the Ether Throne contract ("KotET contract" for short) is another example of a "contract account". The normal operation of the KotET contract is (essentially) this:

  • Suppose the current claim price for the throne is 10 ether.
  • You want to be King/Queen, so you send 10 ether to the contract.
  • The contract sends your 10 ether (less a 1% commission) to the previous King/Queen, as a "compensation payment".
  • The contract makes you the new King/Queen of the Ether Throne.
  • The new claim price for the throne goes up by 50%, to 15 ether in this case.
  • If an usurper comes along who is willing to pay 15 ether, they depose you and become King/Queen, and you receive their payment of 15 ether as your "compensation payment".

For more detail, you can look at the original Solidarity source code at KingOfTheEtherThrone.sol (v0.4.0).

In Ethereum, carrying out a "transaction" such as sending a payment to a contract, or calling a contract, costs "gas". The amount of "gas" consumed depends on what sort of operations the contract you call does (and how many). This "gas" is a small payment which goes to the miners and helps pay for providing the Etherum network and block-chain storage. The gas is paid for by the "externally-owned account" which stared the transaction. When making a transaction, you include a little gas with the transaction (unused gas is refunded). Often Ethereum clients do this for you.

So What Went Wrong?

The King of the Ether Throne contract behaved correctly in all cases apart from when it sent a payment to a "contract account" such as an Ethereum Mist "contract-based wallet".

When the King of the Ether Throne contract sent a payment to a "contract account", it inadvertently included only a small amount of gas with the payment - 2300 gas. This was not enough gas for an Ethereum Mist "contract-based wallet" contract to succesfully process a payment - instead the wallet contract failed.

When a wallet contract failed to process the payment sent to it by the KotET contract, the ether paid was returned to the KotET contract. The KotET was not aware that the payment had failed and it continued processing, making the caller King despite the compensation payment not having been sent to the previous monarch.

The specific line of Solidity code used to send payments was: currentMonarch.etherAddress.send(compensation);

Concrete Example

Ethereum Transaction 6d41b1d3e9b01efc0cc63b5c7ee162bccffe5af00fba3940850b09bfcbee0c9e in Block 967395 is a good example of this issue.

Here, 'Major Tom' sent 42.7 ether to the KotET contract from his external account b2afec1da55c15ad57b3310f9008c47f4e028de3. The current monarch at the time was the nameless 0xcb4046e50f71409a3af23da0961b5ce2f769de31 (contract account) - they had paid 28.5 ether in an earlier transaction from their wallet contract to become King. Let's call them "cb..31" for short.

The KotET contract attempted to send "cb..31" his "compensation payment" of 42.273 ether using the Solidity <address>.send(<amount>) technique. You can see this on the live.ether.camp explorer in the 'Produced 1 internal transaction' section.

Sending this compensation payment caused the wallet contract at 0xcb4046e50f71409a3af23da0961b5ce2f769de31 to start executing with 2300 gas available, which is the standard "stipend" included when a contract uses an Ethereum Virtual Machine CALL operation to interact with another contract (sending a payment to a contract is actually just a type of message call).

The wallet contract managed to execute some of its operations, but eventually reached an Ethereum Virtual Machine CALLCODE operation which was too expensive for the amount of gas it had. This caused the Ethereum Virtual Machine (EVM) to undo any work the wallet contract at 0xcb4046e50f71409a3af23da0961b5ce2f769de31 had managed to acheive. EVM experts can see this in the VM Trace tab on live.ether.camp for the transaction - points of interest in the operations list are the CALL at (DEEP = 0, PC = 1015), the CALLCODE at (DEEP = 1, PC = 39), and the SSTORE at (DEEP = 0, PC = 1871).

The KoET contract continued executing after this compensation payment failed (its work was not undone), and it went on to update the block-chain storage to increase the currentClaimPrice to 64 ether and record 'Major Tom' as the new King.

Other Scenarios

During the period 06 Feb 2016 to 08 Feb 2016, this issue affected two monarch compensation payments (including the concrete example above), and also affected one refund payment which should have returned 7.77 ether to a failed usurper who tried to pay less than the current claim price. Details of these are in Appendix A - Transaction History.

Causes

As with most defects, there were a number of underlying causes: (c.f. the 5 Whys)

  1. The stipend of 2300 gas included with a payment from the KotET contract to an Ethereum Mist wallet contract was insufficient for the payment to be accepted by the wallet contract.
  2. KotET contract developer was unaware that only 2300 gas included when sending payment to an address in Soliditity.
  3. KotET contract developer was unaware that part of a transaction could fail and roll-back without the whole transaction "chain" failing and rolling-back.
  4. Insufficient real-world beta testing by KotET contract developer; testing was performed prior to launch but this did not include use of wallet-contracts to interact with the KotET contract.
  5. Many Solidity example contracts (e.g. Simple Open Auction, 30_endowment_retriever.sol) use Solidity <address>.send(<amount>) (where <address> is a msg.sender) to send payment to an address without checking return value, adding extra gas, or otherwise highlighting this issue. There is a note in the Solidity Address section that mentions the possibility of send() failing - but the example code above does not check the return value.
  6. Solidity FAQ section "How do I use Send?" does not mention gas limitation. There is a hint of the problem at the very end of the What is the deal with "function () { ... }" FAQ section.
  7. The fallback function in the wallet contracts used by the Ethereum Mist Wallet requires more gas than the 2300 available during a <address>.send(<amount>) call, which seems like quite a likely use-case. Perhaps the wallet contracts could cope better with a low gas environment? This seems to be a known issue - ethereum/mist github issue #135
  8. (Possibly) false assumption made by KotET contract developer that contracts could be developed at a high-level in Solidity without needing at least some understanding of the low-level Ethereum Virtual Machine behaviour (such as gas stipend included with a CALL operation).
  9. There does not appear to be any easy-to-find detailed Ethereum Virtual Machine (EVM) documentation other than the Ethereum Yellow Paper (PDF), which is written in a academic mathematics style which is quite impenetrable for most contract developers - e.g. try deciphering the behaviour of the CALL operation from the description on page 29.

While not a direct cause of the issue, investigation into the issue was hampered by these factors:

Immediate Response

The possible existence of an issue was spotted when the balance of the contract appeared to be too high. The time-line of the response was:

  • 06 Feb - King of the Ether Throne ÐApp launched.
  • 07 Feb - Warning posted on reddit thread by KotET developer when contract has unexpected balance of approx. 9 Ether.
  • 07 Feb - Disabled 'Claim Throne' button in the DApp and removed the contract address from the website.
  • 08 Feb - Confirmed that funds had definitely failed to be made to at least one monarch.
  • 10 Feb - Added more details to the 'Important Notice' on the KotET website.
  • 19 Feb - Three refunds for a total of 98.5 ether manually sent to the affected wallet contracts.
  • 20 Feb - First draft of this Post-Mortem posted.
  • 21 Feb - Second draft of this Post-Mortem posted.

Recommendations

Based on this issue, the author intends to:

  1. Avoid using the Solidity code <address>.send(<amount>) unless sure the address is an externally-owned address or is a contract whose fallback function can happily cope with only 2300 gas.
  2. Consider using throw to send a full refund back to the current caller rather than trying to use send/call.
  3. Consider using something like <address>.call.value(value).gas(extraGasAmt)() to send a payment to an arbitrary contract, though choosing a sensible extraGasAmt that is high enough for most receiving wallet contracts but low enough for most callers of the current contract might be hard.
  4. Examine the return value of send() and call() and take appropriate action. Throwing on failure might sometimes be appropriate, but see the point below about poison contracts.
  5. Especially for something like the King of the Ether Throne, where person A's payment triggers a payment to person B, be wary of "poison" contracts that need an extremely large amount of gas to call - this could lead to the game getting stuck in a state where no-one can send enough gas to send a payment to such a poison contract.
  6. Find or develop tools that can search for the blockchain for contract-to-contract calls, or work-around by having the contract store tx.origin as well as msg.sender. Logging events would be good (but tool support for logs are also poor).
  7. Do not develop Solidity contracts without a reasonable grasp of the underlying Ethereum Virtual Machine execution model, particularly around gas costs.
  8. Carry out more testing!

Chosen Fix

See our Contract Safety Checklist for details of how we've attempted to avoid vulnerabilities in the new King of the Ether contract.

Appendix A - Transaction History

This section shows relevant transactions that involved the KotET contract.

Block Number Transaction Hash Narrative
963186 c076a813d03be06ad0c0f0b39167860806513edc71363362f5dd202d48b29ab6 King of the Ether Throne contract (v0.4.0) created at contract account b336a86e2feb1e87a328fcb7dd4d04de3df254d0 by wizard b2afec1da55c15ad57b3310f9008c47f4e028de3.
964881 8ce1c76342526038ebefc4d3bfcb997decf1e9cc0f37d07c7c9fbb2f2d6e9474 Received 7.77 Ether from 64589e97f7b3ed412d89daf98839abb080416cf1 (contract account) as a result of origin d63b8ae7f1ccc24b44105452cab9566614d8bb40 (external account), amount was below the claim price, attempted to refund 7.77 by calling back into 64589e97f7b3ed412d89daf98839abb080416cf1 (contract), ran out of gas due to 2300 stipend being too low for wallet contract to operate.
967037 161da349d10ea9aa9b4089706a5409643f62946a0769177aa244077507f25af5 Received 28.5 ether from cb4046e50f71409a3af23da0961b5ce2f769de31 (contract account), as a result of origin 8bb4038aa9923103b0c8edf23eae3dfceb1ac97e. Sent compensation of 28.215 ether to f031f36717cb524b883d440e3837c138180a0289 (external account).
967395 161da349d10ea9aa9b4089706a5409643f62946a0769177aa244077507f25af5 Received 42.7 Ether from 9dec4be08b93838697fba22c3cdd28c1a03ed159 (external account), attempted to send 42.273 ether to cb4046e50f71409a3af23da0961b5ce2f769de31 (contract account) but failed due to insufficient gas within the cb4046e50f71409a3af23da0961b5ce2f769de31 contract, due to only 2300 being supplied by the KotET contract. The KotET contract did not realise this and continued.
967880 926875349a71718687d3cb0f8c3ec3aef60ffebea5b1a561b678983c0518ab5a Received 64 Ether from d585c0c36d09164ab3b54a1ddcc2a26bef055925 (external account), paid compensation of 63.36 ether to 9dec4be08b93838697fba22c3cdd28c1a03ed159 (external account).
968739 72d6a76f5098eb7eb92b5802b35366ad9d17533fc78a426a8ba8382907f53e77 Interesting one, looks like received 96 ether from 107c98584d4b18bc54e1d74f4d7a0bf505ca466f (external account), tried to pay 95.04 ether to d585c0c36d09164ab3b54a1ddcc2a26bef055925 (external account) but it failed due to gas. This time the whole txn failed and was rolled-back, not just the compensation payment.
969205 f79a26ed0d66a4f3d374bb67f2a605bf0b8f69bd764c16ce880fb782ab3b4500 Origin 60cea93e5d7b98027f7e7e433673f9b30448b001 told contract c0e22f23ff54ca58d93a65044a18a3f245552144 to send 144 ether to KotET contract, which was too high (claim price was 96), so it tried to refund 48 ether by calling back into c0e22f23ff54ca58d93a65044a18a3f245552144. Due to low default gas stipend, that failed, KotET contract didn't check return value and continued, succesfully sending compensation of 95.04 ether to d585c0c36d09164ab3b54a1ddcc2a26bef055925 and making c0e22f23ff54ca58d93a65044a18a3f245552144 the King (which so far he remains).
1029675 61674b7648f410056fed0154e685d6166567fabe8436838bd693a9a935e8668c Wizard b2afec1da55c15ad57b3310f9008c47f4e028de3 called sweepComission function on KotET contract to withdraw the entire contract balance of 101.30877 ether.
1029687 b29f1cb2f487ed33ebed5c42386c60ac12837def628f4fdba8e9a7160bb4769a 42.273 ether sent manually back to 0xcb4046e50f71409a3af23da0961b5ce2f769de31 by wizard b2afec1da55c15ad57b3310f9008c47f4e028de3.
1029722 29a1457cb4e267e7fa75ba2b59c0a7db3e21de0805d2a3da9e2e0ceb61106418 7.77 ether sent manually back to 0x64589e97f7b3ed412d89daf98839abb080416cf1 by wizard b2afec1da55c15ad57b3310f9008c47f4e028de3.
1029733 119d79c43b92a1e6cf05991ebd9820ebb1805b30e107376f0dc6e80f4dff9f2e 48 ether sent manually back to 0xc0e22f23ff54ca58d93a65044a18a3f245552144 by wizard b2afec1da55c15ad57b3310f9008c47f4e028de3.