Review of the Filecoin "Double Spending Incident": Inaccurate Accounting by Exchanges

This article is from the Filecoin Network.

On March 18, 2021, reports emerged that a "serious vulnerability" in Filecoin's Remote Procedure Call (RPC) code had led to a "double spend." These claims are incorrect and highly misleading.

The Lotus team conducted a thorough investigation of the report and found no issues related to the Filecoin network or RPC API code. There is no double spend issue with the chain itself, and the API code is error-free. The affected exchanges have corrected the erroneous transaction records within their systems and are reviewing the recharge processing logic in their accounting systems to rectify their API usage.

Event Review

• Incident Report --- Earlier today, the Lotus team received reports that an exchange incorrectly used the Lotus API to calculate transfers/deposits in the Filecoin network. The API misuse occurred because users reported to the exchange that their accounts were incorrectly recorded multiple times by the exchange's accounting system. This issue has been resolved in the exchange's accounting system — there were no duplicate records on the chain itself.

• API Misunderstanding --- The core of the issue lies in the improper use of Lotus's chain state check API, which behaves differently than expected when handling multiple similar messages. Misunderstanding the output of the Lotus API can lead the accounting system to treat both the original and replacement messages as coming from the same sender and receiver. So far, we are aware of only one exchange being affected by this issue.

• False Reports Become Article Headlines --- Incorrect statements regarding the network's "double spend" have spread through social media channels and made their way into article headlines. The content of these reports has been investigated and determined to be misinformation. The team has found no issues with the Filecoin network or RPC API code. After understanding the facts, many teams and media organizations are correcting their reports.

Actions Being Taken

• Affected Exchange --- The affected exchange discovered the misuse of the API and has immediately taken action to stop deposits, withdrawals, and transfers. They have reversed the erroneous transactions (so there was no loss of funds in this incident) and are correcting their usage of the Lotus API to align with recommended practices.

• Isolated Cases --- Other exchanges have been alerted and are reviewing their code logic to ensure they are not affected by the same error. Many of these reviews have been completed — to our knowledge, no other exchanges have misused the API in this manner.

• Lotus Team --- The Lotus team is actively collaborating with all exchanges to ensure proper handling of this behavior and is improving the API documentation (https://github.com/filecoin-project/lotus/pull/5838) to ensure all other exchanges correctly check the Filecoin chain state.

• Community and Media --- Some teams are working together to contact the media, clarifying the details and facts of the alleged incident, and helping to eliminate misinformation.

• Community Teams --- Community members are providing ways to help other community members report issues accurately and thoughtfully, avoiding the accidental spread of misinformation.

Technical Details

• Same Information --- As far as the Lotus team knows, the issue stems from two messages having the same sender/receiver details, the same nonce, but different Gas parameters — included in the same tipset. It is very common to have two similar messages like this, for example, when replacing a message with one that changes the Gas fee. Such situations are handled securely and correctly by the Filecoin network, and do not lead to double transfers: one of the messages will be executed, while the other will be ignored.

• API Misuse --- However, depending on how people check the chain, this can present the appearance of messages being processed twice. Specifically, the affected exchange used an incorrect method for processing chain state — calling ChainGetBlockMessages on each block of the tipset, and then calling StateGetReceipt on those messages.

• Incorrect API Expectations --- A common point of confusion arises when StateGetReceipt is called on two similar messages (one executed, the other skipped); it will provide the same result, giving the impression that both messages were executed. This is indeed a counterintuitive behavior, but it is intentional. The primary use case for StateGetReceipt is in the event handling program used by Lotus miners and during transaction processing. In cases where messages are replaced, these modules do not care whether the returned information corresponds to the original message or the replaced message — they only want to know if the message was successfully executed on the chain. We have added clarifications in the documentation here: https://github.com/filecoin-project/lotus/pull/5838.

• Using the Correct API --- Most exchanges have correctly used ChainGetParentMessages and ChainGetParentReceipts for accounting to determine which messages were executed on the chain and which succeeded. These are the APIs used by Lotus itself during the chain state computation process to ensure that users can accurately reflect the chain state in this way. Executing StateReplay for each message provides the complete call result, allowing users to compare the MsgCid in the returned InvocResult with the CID of the queried message. This is the recommended step for exchanges to correctly check the chain state and keep their internal reporting systems in sync.