Breakpoint 2024: Workshop: Cranking Up Frankendancer

Solana's Frankendancer project has achieved a groundbreaking milestone, demonstrating the ability to process over 1 million transactions per second (TPS) in a globally distributed testnet. This achievement showcases Solana's commitment to pushing the boundaries of blockchain performance and scalability.

Summary

The Frankendancer project, led by Philip Taffet and his team at Solana, focuses on optimizing the networking stack of the Solana blockchain. Through careful architecture design and performance tuning, they have created a system capable of processing transactions at an unprecedented rate.

The team demonstrated their achievements using a testnet spread across multiple continents, with nodes in London, New York, Frankfurt, and Singapore. By leveraging high-performance hardware and optimizing each component of the transaction pipeline, they were able to achieve over 1 million TPS of simple transfer transactions.

This breakthrough has significant implications for the future of blockchain technology. It demonstrates that decentralized networks can achieve performance levels previously thought impossible, potentially enabling new use cases and applications that require extreme throughput and low latency.

While the demo used simplified conditions compared to a production environment, it nonetheless represents a major step forward in blockchain scalability research. The insights gained from this project could help inform future improvements to the Solana mainnet and push the entire blockchain industry towards higher performance standards.

Key Points:

Frankendancer Architecture

The Frankendancer project utilizes a tile-based architecture, where each tile represents a thread pinned to a CPU core, running specific tasks repeatedly. This design allows for efficient data flow and parallelism, which are crucial for achieving high performance.

The system is composed of various tiles, each responsible for different aspects of transaction processing. These include the net tile for handling network interactions, quick tile for protocol operations, signature verification tiles, TPU tile for deduplication, pack tile for transaction selection, bank tiles for execution, POH tile for ordering, and shred tile for block distribution.

By carefully optimizing each tile and ensuring smooth data flow between them, the team was able to create a highly efficient system capable of processing transactions at an unprecedented rate.

Global Testnet Setup

To demonstrate the capabilities of Frankendancer, the team set up a global testnet mimicking the distribution of nodes on the Solana mainnet. The setup included:

• A load generator and leader node in London
• Follower nodes in New York, Frankfurt, and Singapore
• 10 nodes in total, each with 10 Gigabit network interface cards
• Connections between data centers using Jump's 100 Gigabit WAN
• Nodes equipped with dual 40-core Icelake sockets

This configuration allowed the team to test their optimizations in a realistic, globally distributed environment, demonstrating the system's ability to maintain high performance even with significant network latency between nodes.

Performance Optimizations

The Frankendancer team employed various optimization techniques to achieve their impressive results. Some key optimizations include:

• Streamlining data flow to minimize expensive data copying operations
• Exploiting parallelism at the right granularity and in the right components
• Using AVX-accelerated implementations for signature verification and other computationally intensive tasks
• Implementing non-temporal memory copies and cache-friendly data layouts
• Disabling certain non-essential components to focus on networking stack performance

These optimizations, combined with the tile-based architecture, allowed the system to process transactions at an extremely high rate while maintaining the integrity of the blockchain.

Consensus Limit Adjustments

To fully demonstrate the capabilities of the Frankendancer networking stack, the team adjusted two key consensus parameters:

1. Increased the maximum compute units (CU) per block from 48 million to approximately 600 million
2. Raised the limit on shreds per block from 32,000 to about 1 million

These adjustments allowed the system to pack significantly more transactions into each block, enabling the demonstration of processing over 1 million TPS. While these limits are higher than what would be used in a production environment, they showcase the potential of the optimized networking stack.

Facts + Figures

• Frankendancer demonstrated processing over 1 million transactions per second
• The testnet spanned multiple continents, with nodes in London, New York, Frankfurt, and Singapore
• Initial tests with standard consensus limits achieved 81,517 TPS
• After adjusting consensus limits, the system reached 1.04 million TPS
• The leader node processed and sent out 5.12 Gbps of traffic at peak performance
• Follower nodes retransmitted shreds at about 4.3 Gbps
• The load generator consistently sent transactions at 2.37 Gbps
• Total network bandwidth usage increased from about 6 Gbps to 42 Gbps when processing 1 million TPS
• The testnet used 10 nodes, each with 10 Gigabit network interface cards
• Nodes were equipped with dual 40-core Icelake sockets
• The maximum compute units per block were increased from 48 million to about 600 million
• The limit on shreds per block was raised from 32,000 to approximately 1 million
• The pack tile currently maxes out at about 1.08 million TPS, representing a current bottleneck
• The demo used 7,500 preloaded accounts for sending transactions

Top quotes

1. "How fast can the Frankendancer networking stack go?"
2. "Getting the data flow right for our system is one of the most important decisions in getting good performance."
3. "Core for Core, this roughly doubled our performance."
4. "Parallelism is a bit like fire. It's nice to keep two more, but if it's too much or it's in the wrong place, well, it'll burn you."
5. "We're processing about 1.04 million transactions per second."

Questions Answered

What is Frankendancer?

Frankendancer is a project by the Solana team focused on optimizing the blockchain's networking stack. It aims to achieve extremely high transaction processing rates by carefully designing the system architecture and optimizing each component of the transaction pipeline. The project demonstrates Solana's potential to handle over 1 million transactions per second in a globally distributed testnet environment.

How does Frankendancer achieve such high performance?

Frankendancer achieves high performance through a combination of strategies. It uses a tile-based architecture where each processing stage is assigned to dedicated CPU cores, optimizing data flow and parallelism. The team has implemented various low-level optimizations, such as AVX-accelerated cryptographic operations and non-temporal memory copies. Additionally, they've carefully tuned the system to exploit parallelism effectively and minimize data movement between components.

What was the setup for the Frankendancer demo?

The Frankendancer demo used a globally distributed testnet to simulate real-world conditions. It consisted of 10 nodes spread across London, New York, Frankfurt, and Singapore. Each node was equipped with a 10 Gigabit network interface card and dual 40-core Icelake CPUs. The load generator and leader node were located in London, with follower nodes in the other locations. This setup allowed the team to demonstrate high performance even with significant network latency between nodes.

How many transactions per second did Frankendancer process?

In the final demonstration, Frankendancer processed over 1.04 million transactions per second. This was achieved after adjusting consensus parameters to allow for larger blocks and more compute units per block. Initially, with standard consensus limits, the system processed about 81,517 transactions per second. The million-plus TPS figure represents the raw capabilities of the networking stack when consensus limits are relaxed.

What are the implications of Frankendancer's achievements?

Frankendancer's achievements demonstrate the potential for blockchain networks to handle extremely high transaction volumes. This could enable new use cases that require very high throughput, such as decentralized social media, high-frequency trading, or large-scale IoT applications. While the demo used simplified conditions compared to a production environment, it provides valuable insights that could inform future improvements to Solana's mainnet and push the entire blockchain industry towards higher performance standards.