Breakpoint 2024: Technical Talk: Dyndexer, Indexing Solana On-Chain Data at Scale

Syndicate unveils Dyndexer, a groundbreaking solution for indexing Solana on-chain data at scale, promising to revolutionize how developers interact with blockchain data.

Summary

At Breakpoint 2024, Syndicate, a leading Web3 infrastructure provider, introduced Dyndexer, an innovative tool designed to address the challenges of indexing Solana on-chain data at scale. The presentation, delivered by Syndicate's co-founder and CEO, along with team member Preston, highlighted the complexities of managing real-time streaming infrastructure, processing historical events, and scaling storage and retrieval of indexed data on the Solana blockchain.

Dyndexer emerges as a flexible, dynamic indexing platform that leverages WebAssembly (Wasm) runtime to allow developers to specify how they want to observe and react to on-chain events. This solution promises to simplify the indexing process, offering verifiability, ensuring no missed events, and providing robust observability features.

The tool's architecture is designed to handle the high-throughput nature of Solana, processing approximately one gigabit per second of data changes. By utilizing Syndicate's existing infrastructure, Dyndexer offers strong guarantees in terms of availability, scalability, and data integrity, positioning itself as a potential game-changer for Solana developers and projects requiring efficient on-chain data indexing.

Key Points:

Challenges in Solana Data Indexing

Indexing data on the Solana blockchain presents several significant challenges. First and foremost is the issue of availability – developers need to run a validator to access on-chain data, which can be resource-intensive. Scalability is another major concern, as the infrastructure must be able to grow in tandem with the increasing volume of on-chain data. Lastly, ensuring data integrity is crucial, meaning that all relevant data must be captured accurately and completely.

These challenges are compounded by the high-speed nature of the Solana blockchain. With around one gigabit per second of data changes, managing a real-time streaming infrastructure becomes a formidable task. Additionally, the need to process historical events and scale storage and retrieval of indexed data as the number of on-chain events grows adds layers of complexity to the indexing process.

Dyndexer: A Flexible Indexing Solution

Dyndexer, developed by Syndicate, addresses these challenges head-on with its innovative approach to indexing. At its core, Dyndexer is a WebAssembly (Wasm) runtime that allows for dynamic specification of how to observe and react to on-chain events. This flexibility is crucial in the rapidly evolving Solana ecosystem, where new protocols and data structures are constantly emerging.

The platform operates as a one-time solution, calling the developer's code and passing in every relevant event. This approach allows for both real-time indexing and historical data processing, enabling developers to replay history and restate their databases as needed. Dyndexer takes care of the complex streaming infrastructure, leveraging Syndicate's experience as an RPC provider to ensure data verifiability and completeness.

Dyndexer Architecture and Functionality

Dyndexer's architecture is designed for simplicity and efficiency. Developers create an indexer by specifying a filter that defines the subset of Solana stream data they're interested in. This filtered data is then passed to the indexer runtime, where it can be processed according to the developer's specifications.

The indexer program itself is typically written in Rust, although the use of WebAssembly allows for flexibility in programming languages. The program defines how to process incoming events, which can be either transactions or account updates. Developers can use anchor IDLs to parse events from transactions, transform the data as needed, and then write the results to a supported database such as PostgreSQL or ClickHouse.

Availability, Scalability, and Integrity Guarantees

Syndicate leverages its existing infrastructure to provide strong guarantees for Dyndexer users. In terms of availability, the company runs multiple validators and consumes data streams from multiple nodes across different regions. This redundancy ensures high uptime and resilience for indexers.

Scalability is addressed through optimized hardware and improved networking capabilities. Syndicate has invested in powerful infrastructure to handle the increasing demands of data indexing as more protocols are added to the Solana ecosystem. The storage layer is also designed to scale, with support for various database systems and messaging protocols.

Data integrity is ensured by consuming multiple streams from different validators. This approach allows Dyndexer to fill in any missing updates that may occur from a single validator, ensuring that indexers don't miss any critical data. The ability to restart from Genesis and process all historical events further enhances data completeness and accuracy.

Facts + Figures

• Syndicate is building a Web3 native cloud, primarily providing RPC infrastructure and developer tools for Solana.
• The team behind Syndicate is also developing the SIG Validator.
• Solana on-chain data changes occur at a rate of approximately one gigabit per second.
• Dyndexer uses a WebAssembly (Wasm) runtime for flexibility in programming languages.
• The tool supports PostgreSQL and ClickHouse for data storage, with plans to support more messaging protocols.
• Dyndexer allows indexing from Genesis, enabling complete historical data processing.
• The platform currently supports transaction indexing, with account indexing in development.
• Syndicate runs multiple validators and consumes data streams from multiple nodes across different regions for high availability.
• The company has invested in custom hardware and improved networking to handle increasing indexing demands.
• Dyndexer is currently in closed beta, with the team actively seeking feedback from developers.

Top quotes

1. "Indexing data on chain is not easy. The first issue today is availability. So you have to run a validator to actually get that data from on chain into your infrastructure."

2. "We developed a solution. It's called the indexer. And I'm going to pass it over to Preston to talk about it."

3. "Dyndexer is a one-time platform. We call your code and we pass in every single event that you care about."

4. "We guarantee all those things by leveraging our existing infrastructure. On the point of availability, we run multiple validators. We have a lot of experience doing this. We're globally distributed."

5. "We consume multiple streams from multiple validators. Make sure that data reported on one, but now we're on the other. It gets it filled in. So that way your indexer isn't missing any updates."

Questions Answered

What is Dyndexer and how does it solve Solana indexing challenges?

Dyndexer is a flexible indexing solution developed by Syndicate to address the challenges of indexing Solana on-chain data at scale. It uses a WebAssembly runtime that allows developers to dynamically specify how to observe and react to on-chain events. Dyndexer solves indexing challenges by providing a streamlined way to filter, process, and store relevant blockchain data, while ensuring availability, scalability, and data integrity through Syndicate's robust infrastructure.

How does Dyndexer ensure data integrity and completeness?

Dyndexer ensures data integrity and completeness through several mechanisms. It consumes multiple data streams from different validators, allowing it to fill in any missing updates that might occur from a single validator. This redundancy ensures that indexers don't miss critical data. Additionally, Dyndexer allows indexing from Genesis, enabling complete historical data processing. This approach guarantees that all relevant data, from the beginning of the blockchain to the latest transactions, can be accurately captured and indexed.

What programming languages can be used with Dyndexer?

While the primary example provided in the presentation uses Rust, Dyndexer's use of WebAssembly (Wasm) runtime allows for flexibility in programming languages. Developers can write their indexer programs in any language that compiles to WebAssembly. This flexibility enables developers to use their preferred programming language, making Dyndexer accessible to a wider range of developers and potentially speeding up adoption and integration into existing projects.

How does Dyndexer handle the high throughput of Solana data?

Dyndexer is designed to handle Solana's high-throughput nature, which generates approximately one gigabit per second of data changes. It achieves this through Syndicate's optimized infrastructure, including custom hardware in data centers and improved networking capabilities. The platform's architecture is built to scale as more protocols are added and data volume increases. By leveraging multiple validators and data streams, Dyndexer can efficiently process and index the large volume of data generated on the Solana blockchain.

What storage options does Dyndexer support for indexed data?

Currently, Dyndexer supports PostgreSQL and ClickHouse as primary storage options for indexed data. These databases are known for their performance and scalability, making them suitable for handling the large volumes of data generated through blockchain indexing. The team at Syndicate is also planning to expand support to include more messaging protocols and storage solutions in the future, potentially including options like Redis and Kafka to meet diverse developer needs.