A look back: NU5 and network sandblasting

TL;DR

As Electric Coin Co. (ECC) and the Zcash community prepare for public retrospectives of zcashd Network Upgrade 5 (NU5) and what has become known as the sandblasting attack, this blog post details ECC’s motivations, challenges, and accomplishments with regard to each. 

  • The NU5 upgrade was an ambitious undertaking to eliminate the need for trusted setups, increase user confidence, and enhance Zcash’s security and scalability.
  • The implementation of Halo, a cryptographic breakthrough, on Zcash made possible trustless, private, digital-cash transactions for the first time on mobile phones.
  • A malicious attack following the upgrade led to significant wallet performance issues, prompting ECC to enter Emergency Mode. Our response involved multiple technical updates, performance improvements, and the release of new mobile-wallet SDKs.
  • Despite the challenges and delays, ECC succeeded in improving the security and resiliency of Zcash.

Public retrospective: Wednesday, Dec. 13 at 22:00 UTC/17:00 EST

What the FUD?

ECC’s mission is to empower economic freedom, and most organizations that work on Zcash-related projects, like the Zcash Foundation (ZF) and Zcash Community Grants (ZCG), have similar community-minded objectives for the betterment of humanity. 

Nevertheless, since Zcash came into the world in 2016 a stream of misinformation (and some coordinated propaganda) has been circulated on social media, forums, and other community platforms meant to incite FUD (fear, uncertainty, and doubt).

The rumors claimed that Zcash was compromised, or that it was vulnerable to counterfeiting, or that there was a backdoor that would allow third parties to access a user’s transaction information without their consent. 

And unfortunately, until NU5 was released, there was a major obstacle in confronting these attacks: the trusted setup. When Zcash launched, its zero-knowledge proofs required a Ceremony, or trusted setup phase, to produce public parameters that allowed users to construct and verify private transactions.

This technique was pioneered by ECC, and it required multiple parties in different locations using complex security measures.

Regardless of the extra-careful planning and execution plus redundant security, Zcash users and observers had to trust the historical fact that the ceremony participants weren’t conspiring to deceive the public. Trusted setups don’t allow for a mathematically verifiable supply, and as long as these complicated procedures were required for major Zcash upgrades, there would always be doubters and detractors. And the doubters would have a point — mathematics is verifiable, but history isn’t. The reputation and integrity of Zcash was being challenged, and in ECC’s view, this was an issue the community couldn’t afford to ignore.

NU5: A call to action

In 2019, ECC engineers Sean Bowe, Daira Emma Hopwood, and Jack Grigg had been working on scalability design and experimenting with a solution for efficient recursion when Bowe made a discovery.

“I just stumbled upon a technique to do this zero-knowledge proof construction on completely ordinary elliptic curves,” Bowe said. “And so over the course of maybe 24 hours, it went from, ‘Ooh, that’s really exciting, that’ll be neat if we can do that’ to ‘Holy crap, now that we don’t have trusted setups or anything, it’s all really simple. It was a series of steps that turned from a nice insight into a complete paradigm shift.”

Halo, as it would come to be known, is a zero-knowledge proving system that enables recursion without a trusted setup in an efficient way. 

Bowe’s discovery was a cryptographic breakthrough heralded by the industry, and in 2021 ECC committed to implementing Halo in Zcash.

Halo on Zcash would make trustless, private, digital-cash transactions possible for the first time on mobile phones. It would serve as a catalyst for Zcash user confidence and provide a path to much greater scalability, while making the protocol more attractive, faster, and less expensive for others to build on. 

From the ECC blog: 

Halo on Zcash would enable circuit upgrades without the need for trusted setups, making the Zcash shielded protocol more agile for future improvements, such as supporting additional assets like [Zcash Shielded Assets, or ZSAs]. We want to make it easy for other projects and tokens to benefit from Zcash features, such as privacy through encryption. Trusted setup will become a remnant of the past.

In addition, this upgrade would pave the way for shielded Zcash scale through proof aggregation and blockchain succinctness, two scalability improvements. This would improve the user experience by eliminating frustrating synchronization time that plagues all blockchains today, reducing the traditional blockchain bloat, and allowing for non-escalating fees as usage increases. In conversations with large social platforms who expressed interest in native Zcash support, a viable path to scalability was given as a requisite near-term consideration.

In January 2021, after more than a year of Halo R&D, ECC concluded that the benefits of Halo on Zcash outweighed other protocol priorities, and we proposed implementing it in NU5. 

Watch the Zcash Media video about the original Ceremony, its participants (including Edward Snowden), and how Halo makes trusted setups obsolete.

NU5 objectives and obstacles

Our NU5 goals from the outset were to (1) make Zcash more secure, (2) give Zcash users confidence by making the supply mathematically verifiable, (3) make future upgrades easier, and (4) enable future upgrades to benefit from recursive proofs for scalability and programmability improvements.

Our original estimate of 6-7 months turned out to be overly optimistic — as new discoveries were made, opportunities were revealed in the process, and technical complexities arose — and in the end the journey took almost a year and a half.

Along the way, we confronted a number of technical challenges. For example, the complexity of including Orchard support in the zcashd embedded wallet was not properly accounted for in the original estimates for the NU5 timeline. 

ECC faced time-consuming engineering hurdles, like building a complicated circuit for Halo-plus-Orchard to make it work on mobile devices. And midstream, we made the decision to implement unified addresses (UAs) to enable shielded by default in supporting wallets. The design and build-out of UAs were difficult, and if ECC had been practicing product-driven planning and development at the time (like we are now) we believe we would have identified the need for UAs sooner and their implementation would have been smoother.

In addition to battling technical obstacles like these, plus a few surprise bugs, ECC spent significant time and resources advocating for and defending our roadmap decisions against criticism for: our rationale (getting rid of trusted setup wasn’t worth the time and effort, ZSAs were more important), our planning (we kicked off the discussion publicly before planning enough -and- we planned too much before starting the work), the decision to license Orchard under the Bootstrap Open Source Licence (BOSL), the decision to implement Unified Addresses (UAs), and our overall approach to network upgrades (our priorities were arbitrary and out of touch).

It’s important to acknowledge that we had significant community support during the process, too. Overall, community sentiment was positive. We received valuable insight and feedback from our Scientific Advisory Board and crucial security audits from Qedit and NCC group.

NU5 timeline and accomplishments

Not including the pre-implementation research and development, NU5 was a 17-month endeavor that delivered novel technology and improved Zcash user experience. 

Timeline

When Zcash NU5 activated on mainnet May 31, 2022, it was one of the most important milestones for Zcash since the cryptocurrency launched in 2016. As ECC CEO Zooko Wilcox put it, “an historic step forward for human society.” 

In launching the Orchard shielded payment protocol utilizing Halo, we eliminated the trusted setup to improve Zcash security and sustainability, made the supply mathematically verifiable, improved scalability because future network upgrades won’t require a complicated setup ceremony, paved the way for increased interoperability by providing a system that could unlock private cross-chain proofs at scale, and introduced BOSL which has returned value to the Zcash community (e.g., Filecoin Foundation and Ethereum Foundation grants to Brave and Edge).

It was a massive effort, and after almost a year and a half of building and wrestling with the technical challenges of NU5, the ECC Core team was ready for some rest. But almost right away, Zcash was hit with another issue we couldn’t ignore.

The sandblasting attack

In June 2022, almost immediately after ECC launched NU5, a serious problem emerged. The Zcash network began experiencing a huge increase in shielded transaction size and activity. This additional network load caused a “data pileup” that prevented some wallets (Nighthawk, Edge, and Unstoppable) from being able to sync in a reasonable amount of time. Wallets weren’t syncing and some users weren’t able to access their funds.

This was a problem that ECC, as the primary maintainer of zcashd and supplier of mobile-wallet SDKs, was best-positioned to confront.

Sandblasting attack: A call to action

Some parties at ECC did not initially want to make the assumption that this unusual transaction load was the work of a malicious actor. Eventually however, the evidence that it was a malicious actor became overwhelming — and we have reason to believe it may even have been coordinated by the same actor or group of actors who spread Zcash FUD, and that they may have been paid to do so.

But solving that mystery was less important than fixing the problem. 

Sandblasting attack: Objectives and obstacles

Our top priority was ensuring users could regain access to and spend their ZEC (Zcash coins). This is fundamental to our mission of economic freedom and a requirement for real-world, private digital cash.

As an initial line of defense, we released  performance improvements in zcashd 5.1.0 and 5.2.0 to reduce verification time by up to 80%. We also began working on performance upgrades to our mobile wallet SDKs.

In August 2022, with lots of work left to solve these wallet performance issues, ECC went into Emergency Mode. This was our criteria for exiting Emergency Mode:

  • Users of Edge, Nighthawk, and Unstoppable can spend their current funds (funds that are already synced when they open their wallet).
  • Users of those wallets can receive and become able to spend new incoming funds at a rate of a month’s worth of transactions in 1 hour.
  • Users of those wallets see sync updates which are minimally confusing about progress.
  • None of those wallets are impacted by frequent crashes or inconsistent behavior (such as failing to display some already synced transactions), nor do they require work-around behaviors due to the ECC SDK.

The wallet performance issues presented a complicated series of challenges to address, and they required developing and implementing (1) a faster algorithm that does not require a linear sync of all blocks on chain and (2) tooling modifications that would give users the ability to spend funds without having a fully synced chain. The solution required changes to every component in the shielded mobile wallet stack: zcashd, lightwalletd, the ECC wallet SDKs, and the ECC prototype wallet.

Internally, fee changes were debated from the get-go, but for the first few months, the ECC team focused on additional performance improvements and worked to resolve the issue without fee changes. Later, we decided to implement fee changes to throttle the spam attack.

As users became frustrated with their user experience, some community members became vocal and publicly criticized ECC for not being prepared for the attack and slow to respond.

In February 2023, we were notified by blockchain security firm Halborn of vulnerabilities inherited from Bitcoin Core that may have affected more than 280 chains, including Zcash. This was another all-hands-on-deck emergency on top of the wallet performance issues, and as ECC was the only team notified of the vulnerability, we were the only ones who could make the necessary fixes. The coordination of the disclosure and remediation consumed approximately a month of our time.

To make matters worse, ECC as an engineering organization was attempting to do too many things at once: core node maintenance, supporting CEXes like Coinbase, Binance, and Gemini, backports from Bitcoin, mobile SDKs, mobile wallet applications, and future protocol improvements. ECC restructured in May 2023, which further strained resources temporarily.

Additional challenges

  • At the time, ECC internal communication and project management practices weren’t organized for an Emergency Mode situation. This affected our understanding of priorities and workflow across teams.
  • Before the attack, ECC product strategy focused on adoption first, instead of performance and scalability. Users want new features, and working on improving performance slows feature releases. But the attack took Zcash UX to the limits immediately.
  • Technical debt: We had implemented previous network upgrades with only limited performance improvements.
  • ECC vastly underestimated how long it would take to fix outstanding wallet performance issues.
  • Other technical issues arose that caused delays, such as when speed.z.cash bitrotted and then got deleted while ECC was without a devops manager. 
  • The ECC Core team had three different managers during Emergency Mode.
  • Emergency Mode slowed development of our planned wallet product, now known as Zashi. However, we were able to leverage the prototype internally to validate and test SDK improvements.

ECC was lucky to have proactive wallet partners — Edge, Nighthawk, and Unstoppable — who worked with us to test releases and submit bugs, then were quick to implement SDKs 2.0 when they were ready.

Sandblasting attack: Timeline and accomplishments

In all, ECC’s sandblasting response and implementation by third-party wallets consumed about 16 months. By November 2023, Edge, Nighthawk, and Unstoppable were working again (better than ever), and we announced an end to Emergency Mode.

Timeline

  1. June 2022: Network spamming began; shielded outputs jumped from a monthly average of 42,600 to 21,622,590 in June alone
  2. July 2022: Implemented performance improvements in zcashd 5.1.0 and 5.2.0 to reduce verification time by up to 80% and began working on performance improvements to our mobile wallet SDKs
  3. August 2022: ECC officially entered Emergency Mode, although we didn’t use that term in public written communications until March 2023; started researching Spend Before Sync
  4. September 2022: While working on zcashd and improved mobile sync experience, re-opened research into possible fee change mechanisms (ZIP 317) and syncing efficiencies (DAGSync/Spend Before Sync)
  5. October 2022: Released zcashd 5.3.0 with additional performance improvements to reduce concurrent memory utilization during scanning among other memory and performance related optimisations in the zcashd node
  6. October 2022: Reaffirmed our immediate objectives regarding syncing issues with our wallet SDKs and communicated our criteria for exiting Emergency Mode
  7. October 2022: Communicated our intent to have wallet performance issues resolved by the end of February 2023
  8. October 2022: Changes to zcashd increased its robustness against the attack
  9. November 2022: Identified and fixed the last known bugs that were blocking the first phase of SDK releases, continued progress on Spend Before Sync, and began prepping both zcashd and mobile wallets for ZIP 317; squashed the last known librustzcash bugs that were blocking wallet release and completed Android & iOS SDK integration
  10. December 2022: Completed the implementation of zcashd optimizations expected to save memory and to reduce orphan rates for miners
  11. February 2023: Released zcashd 5.4.0 to introduce a number of performance improvements, an update to correct supply reporting, and a clean up of legacy features and support functionality to improve ongoing maintenance of the codebase
  12. March 2023: Published an update and release schedule calling for an end to Emergency Mode by the end of May
  13. April 2023: ECC released 5.5.0 with the completed transaction-fee-structure change (ZIP 317) and infrastructure improvements that laid the groundwork for 5.6.0.
  14. May 2023: (Restructuring at ECC) Pushed the release date for zcashd 5.6.0, lightwalletd, and SDKs to mid-June. As zcashd 5.5.0 began to be adopted by miners on the network, the fee structure changes began to blunt the efficacy of the sandblasting attack, reducing the rate of blockchain growth and wallet scanning load.
  15. June 2023: Released zcashd 5.6.0, introducing the functionality necessary for light wallets to access spendable funds without fully scanning the blockchain; included updates to address issues related to the ZIP-317 implementation and privacy policy features.
  16. July 2023: Lightwalletd 0.4.14 was released, which meant the mobile wallet SDK was the only outstanding piece in ECC’s plan to resolve wallet performance issues and exit Emergency Mode. At this point, we had hoped to release the SDKs in mid-July.
  17. August 2023: By the end of July, the adoption of zcashd 5.5.0 by the network was complete, and network transaction load had returned to pre-NU5 levels. However, the accumulated chain data to this point still presented a problem for wallet scanning, and so to mitigate this issue ECC announced the pre-release availability of the Spend before Sync capability in our mobile SDKs and made this functionality available to wallet developers for testing. Several bugs and UX issues were discovered. Timeline for the SDK releases was shifted to mid-September.
  18. September 2023: New bugs, UX issues, and dependencies arose that pushed the SDKs release date out to the end of September.
  19. Sept. 26, 2023: ECC released the updated mobile SDK 2.0 for both iOS and Android developers. This was the final deliverable for exiting Emergency Mode.
  20. Nov. 1, 2023: After a month in production, no related, major wallet issues were reported and ECC declared an exit from Emergency Mode.

The culmination of our sandblasting response was that third-party Zcash wallets were working again, and future attackers were further disadvantaged. 

During this time, ECC released multiple updates to zcashd and lightwalletd, plus new mobile SDKs that, together, introduced new innovations (and learning) in the world of cryptography and decentralized money. These releases provide massive upgrades to privacy, scalability, and user experience in Zcash, and they have implications for all privacy-focused crypto projects.

The work continues

NU5 and the sandblasting attack consumed nearly three years. But faced with organized adversity, competing emergencies, and community criticism (some of it warranted), ECC did not stray from our commitment to build. With allies along the way, the community overcame unprecedented technical challenges and resource constraints to make Zcash more res

Recent blog posts: