Step Finance: Revolutionizing Solana DeFi Data Tracking and Portfolio Management

In the rapidly evolving world of decentralized finance (DeFi), keeping track of one's portfolio across multiple protocols and platforms can be a daunting task. Enter Step Finance, a pioneering dashboard and analytics platform built on Solana that aims to simplify this process for users. In a recent episode of the Solfate podcast, hosts Nick and James sat down with Dana (also known as @TheSoftwareJedi), the lead backend developer at Step Finance, to discuss the intricate workings of their data aggregation system and the challenges they face in the ever-changing DeFi landscape.

Dana's Journey into Crypto

Dana's entry into the world of cryptocurrency is a testament to the transformative power of blockchain technology. In 2013, while struggling with alcohol addiction and experiencing homelessness, Dana discovered Bitcoin. This discovery came at a time when he was grappling with the realities of being unbanked in the United States.

"I was discovering this very real problem of being unbanked," Dana recalls. "Now, mind you, this is being unbanked in the US, right? It was very short-lived. I got a bank account. I got things cleaned up. But this philosophy of money thing really stuck with me."

This experience sparked a deep interest in the potential of cryptocurrencies to provide financial solutions for those excluded from traditional banking systems. Despite his challenging circumstances, Dana began investing in Bitcoin, setting up a bi-weekly dollar-cost averaging (DCA) strategy that he has maintained to this day.

The Solana Revelation

While Dana's journey into crypto began with Bitcoin, his involvement with Solana came later, in December 2020. A friend introduced him to the blockchain, and Dana was immediately impressed by its architecture and design. Unlike his skepticism towards Ethereum, which he had previously dismissed due to its slow block times, Solana's high-speed capabilities resonated with him.

"I looked into Solana and was like, wow, this is legit," Dana explains. "Like this is like Ethereum, but the architecture and design while incredibly complex makes sense. Like this does high-speed stuff."

Driven by his newfound passion for Solana, Dana began tinkering with on-chain programs and became an active participant in the Solana Discord community. His helpfulness and technical knowledge caught the attention of Jordan Prince, a prominent figure in the Solana ecosystem, who connected Dana with a recruiter. This connection ultimately led to his role at Step Finance.

Step Finance: A Comprehensive DeFi Dashboard

Step Finance emerged as a solution to a common problem faced by DeFi users – the difficulty of tracking positions across multiple protocols. As one of the first employees at Step Finance, Dana was drawn to the project's mission of creating a comprehensive dashboard for Solana-based DeFi activities.

The core product of Step Finance is its dashboard, which aggregates data from various DeFi protocols and presents it in a user-friendly interface. This allows users to view their positions, track historical performance, and manage their portfolios more effectively.

The Engineering Challenges of DeFi Data Aggregation

Creating a dashboard that can seamlessly integrate data from multiple DeFi protocols is no small feat. Dana shed light on the complex engineering challenges that the Step Finance team faces in maintaining and improving their product.

The Old Way: Protocol-Specific Integrations

Initially, Step Finance approached data aggregation by integrating each protocol individually. This method involved using protocol-specific SDKs or recreating lightweight versions of these SDKs to read data from each protocol. However, this approach quickly became unsustainable as the number of protocols grew and evolved.

"You have like your Orca positions showing on a page and then along comes Radium and then along comes, let's say 50 other portfolio or products that you want to integrate into this dashboard," Dana explains. "And that's when the problems really come up because as everyone knows that tries to integrate 50 different SDKs into the same program, you have versioning problems right off the bat."

The team faced additional challenges with optimizing RPC calls and keeping up with protocol upgrades and retirements. This constant maintenance became a significant burden on the engineering team.

The New Way: Real-Time State Tracking

To address these challenges, Step Finance is transitioning to a new approach that involves tracking the state of every account on Solana in real-time. This ambitious project aims to create a comprehensive database of DeFi-related information, parsed and indexed in a way that supports both the dashboard and analytics products.

Dana explains the shift: "What would it take to just receive and record state updates for every account on all of Solana in real time and index it in a very specific way, not just, you know, account data, you see a lot of indexers out there that are just, you know, put all the binary account data in a table. Instead, what if we parsed the DeFi related information out of there and sorted in very specific DeFi tables?"

This new approach allows Step Finance to provide more comprehensive and up-to-date information, including historical data for various DeFi positions. It also enables powerful analytics capabilities, such as tracking the number of users in each farm across different protocols.

Technical Infrastructure and Scaling Challenges

The shift to real-time state tracking has brought its own set of technical challenges, particularly in terms of data storage and processing. Step Finance initially used PostgreSQL with TimescaleDB for their database needs, but the rapid growth of data volume pushed them to explore more scalable solutions.

Transitioning to ClickHouse

To handle the massive amount of data generated by tracking every Solana account in real-time, Step Finance has begun migrating to ClickHouse, a column-oriented database management system designed for high-performance analytics.

"We started with Postgres TimescaleDB. We have officially just outgrown that," Dana reveals. "I think our drive is now up to four terabytes running on AWS and EBS drive of four terabytes cost a fortune."

The transition to ClickHouse presents its own challenges, requiring a different approach to data modeling and querying. Dana emphasizes the importance of thoroughly understanding ClickHouse's architecture and capabilities before diving in:

"Don't go into it just plowing along. Even if you're a DBMS expert, just running in there and thinking, you know, how it works and you know how to do it. And you'll figure the details out later, you got to dive into the docs. You got to understand how this thing's being stored on disk, how indexes work because they're not the same."

The Importance of Low-Level Understanding

Throughout the conversation, Dana repeatedly emphasized the importance of having a deep, low-level understanding of Solana's architecture and account model. This knowledge is crucial for developers working on Solana-based projects, especially when dealing with complex data aggregation and real-time processing.

"If you're getting started out as an engineer, spend the time to really learn this account model on Solana. What a PDA is, what get program accounts means, what's a discriminator," Dana advises. "You're going to have to go low level. The answers are all down there in the low level code."

He draws a parallel between programming on Solana and working with Arduino microcontrollers, highlighting the similarities in dealing with constrained environments and the need for efficient resource management.

Step Finance's Broader Ecosystem

While the dashboard remains the core focus of Step Finance, the company has expanded its offerings to include several other initiatives under the Step Foundation umbrella. These include:

1. Solana All Stars: An underbanked country onboarding program that holds weekly events in countries like Venezuela, Argentina, Turkey, and Nigeria to spread awareness about cryptocurrency.

2. Solana Floor: A news team that covers developments in the Solana ecosystem.

3. Step Staking: A staking program that allows users to earn revenue generated from Step Finance's various business lines.

These initiatives demonstrate Step Finance's commitment to not only providing valuable tools for DeFi users but also contributing to the broader growth and adoption of the Solana ecosystem.

The Business Model of a Free Dashboard

One intriguing aspect of Step Finance's business model is that their primary product – the dashboard – is free for users. This raises questions about how the company generates revenue to sustain its operations and development.

Dana explains that Step Finance has explored various revenue streams over time, including fees on token swaps. However, they've recently shifted their focus to validator staking as a primary source of income:

"We started running our own validator. We're using the sanctum program. So we have steps old, encouraging our users to stake with us makes us more money than, you know, a point 1% on a swap."

This approach leverages Step Finance's large user base and strong brand recognition in the Solana ecosystem. By encouraging users to stake with their validator, Step Finance can generate revenue while also contributing to the network's security and decentralization.

The Future of DeFi Data Aggregation

As the DeFi landscape continues to evolve at a rapid pace, the need for comprehensive and user-friendly data aggregation tools like Step Finance will only grow. The challenges faced by the Step Finance team in tracking and presenting this data highlight the complexity of the DeFi ecosystem and the innovative solutions required to make it accessible to a broader audience.

The transition to real-time state tracking and the use of advanced database technologies like ClickHouse represent a significant leap forward in DeFi data aggregation capabilities. These advancements promise to deliver more accurate, up-to-date, and comprehensive insights to users, potentially revolutionizing how people interact with and manage their DeFi portfolios.

Moreover, Step Finance's commitment to education and onboarding through initiatives like Solana All Stars demonstrates a holistic approach to ecosystem growth. By combining powerful tools with educational efforts, Step Finance is playing a crucial role in driving adoption and understanding of DeFi technologies, particularly in underbanked regions.

Conclusion

Step Finance's journey from a simple dashboard to a comprehensive DeFi data aggregation and analytics platform showcases the rapid evolution of the Solana ecosystem. Through innovative engineering solutions and a deep understanding of the blockchain's underlying architecture, the team is tackling some of the most complex challenges in DeFi data management.

As the DeFi space continues to grow and mature, tools like Step Finance will play an increasingly important role in making these complex financial systems accessible and manageable for users. By providing clear, real-time insights into users' DeFi activities across multiple protocols, Step Finance is not just simplifying portfolio management – it's paving the way for broader adoption and understanding of decentralized finance.

The story of Step Finance and Dana's journey in the crypto space serves as an inspiring example of how blockchain technology can create opportunities and drive innovation. From discovering Bitcoin in a homeless shelter to leading the backend development of a cutting-edge DeFi platform, Dana's experience underscores the transformative potential of this technology and the exciting possibilities that lie ahead in the world of decentralized finance.

Facts + Figures

• Dana discovered Bitcoin in 2013 while in a homeless shelter, leading to his passion for cryptocurrency.
• Step Finance was one of the first tokens to launch on Solana, participating in early Raydium fusion pools.
• The Step Finance dashboard initially made about 400 RPC calls for a single page load.
• Step Finance's database has grown to four terabytes, running on AWS EBS drives.
• The company has transitioned from PostgreSQL with TimescaleDB to ClickHouse for better performance and cost-efficiency.
• Step Finance runs its own validator, which has the highest number of distinct stakers on the Solana network.
• The company has expanded to include six different business lines under the Step Foundation umbrella.
• Solana All Stars, a Step Finance initiative, holds weekly events in countries like Venezuela, Argentina, Turkey, and Nigeria.
• Step Finance is transitioning to tracking the state of every account on Solana in real-time.
• The team processes around 4,000 DeFi operations per second in their real-time data stream.
• Dana joined Step Finance as employee number five or six in late 2020 or early 2021.
• Step Finance recently made swap fees free for users, focusing on validator staking as a primary revenue source.
• The company faced a technical challenge when they maxed out a 32-bit integer space for storing account IDs in their database.
• Dana's experience with Arduino microcontrollers has been helpful in understanding and working with Solana's constrained environment.
• Step Finance's transition to ClickHouse is expected to significantly improve query performance for their analytics and dashboard products.

Questions Answered

How did Dana first get involved with cryptocurrency?

Dana discovered Bitcoin in 2013 while staying in a homeless shelter. During this time, he was experiencing the challenges of being unbanked in the US, which made him realize the potential of cryptocurrencies as an alternative financial system. This experience sparked his interest in Bitcoin, leading him to set up a bi-weekly investment strategy that he has maintained to this day.

What attracted Dana to Solana specifically?

Dana was introduced to Solana by a friend in December 2020. He was immediately impressed by Solana's architecture and design, particularly its ability to handle high-speed transactions. Unlike Ethereum, which he had previously dismissed due to slow block times, Solana's capabilities resonated with Dana's understanding of what a blockchain should be able to achieve in terms of performance.

What is Step Finance's core product?

Step Finance's core product is a comprehensive dashboard for Solana-based DeFi activities. The dashboard aggregates data from various DeFi protocols, allowing users to view their positions, track historical performance, and manage their portfolios more effectively. This solution addresses the common problem of DeFi users struggling to keep track of their positions across multiple protocols.

How is Step Finance changing its approach to data aggregation?

Step Finance is transitioning from protocol-specific integrations to a new approach that involves tracking the state of every account on Solana in real-time. This new method aims to create a comprehensive database of DeFi-related information, parsed and indexed in a way that supports both the dashboard and analytics products. This approach allows for more comprehensive and up-to-date information, including historical data for various DeFi positions.

What technical challenges has Step Finance faced in scaling its data infrastructure?

Step Finance has faced significant challenges in scaling its data infrastructure to handle the massive amount of data generated by tracking every Solana account in real-time. They initially used PostgreSQL with TimescaleDB but outgrew this solution as their database reached four terabytes. To address this, they are migrating to ClickHouse, a column-oriented database management system designed for high-performance analytics. This transition requires a different approach to data modeling and querying.

How does Step Finance generate revenue if its dashboard is free?

Step Finance has explored various revenue streams, including fees on token swaps. However, they've recently shifted their focus to validator staking as a primary source of income. By running their own validator and encouraging users to stake with them, Step Finance can generate revenue while also contributing to the Solana network's security and decentralization. This approach leverages their large user base and strong brand recognition in the Solana ecosystem.

What advice does Dana give to aspiring Solana developers?

Dana emphasizes the importance of gaining a deep, low-level understanding of Solana's architecture and account model. He advises new developers to spend time learning about PDAs (Programmed Derived Addresses), get program accounts, and discriminators. Dana stresses that many answers lie in the low-level code, and this knowledge is crucial for effectively developing on Solana, especially when dealing with complex data aggregation and real-time processing.

How is Step Finance contributing to the broader Solana ecosystem?

Beyond its core dashboard product, Step Finance has expanded its offerings under the Step Foundation umbrella. This includes initiatives like Solana All Stars, an underbanked country onboarding program that holds weekly events in countries like Venezuela, Argentina, Turkey, and Nigeria to spread awareness about cryptocurrency. They also run Solana Floor, a news team covering developments in the Solana ecosystem. These initiatives demonstrate Step Finance's commitment to contributing to the growth and adoption of the Solana ecosystem beyond just providing tools for DeFi users.