We regularly encounter the challenge of collecting TVL and APY from DeFi protocols. On the surface, it seems simple: pull the data, store it in a database, serve it via an API. But in practice, each protocol uses its own calculation logic — part of the data lives on-chain, part in subgraphs with delays, and APY is recalculated every block. Many protocols deploy multiple versions on different chains with incompatible ABIs. Moreover, RPC providers throttle request frequency, and The Graph subgraphs can become stale (The Graph Protocol). Without a well-thought-out scraping architecture, instead of clean data, you end up with a mess of missing data points, inflated TVL due to manipulated prices, and incorrect APY. Our team has 5+ years of experience in DeFi data engineering, having delivered 50+ scraping projects for leading protocols.
Data Sources and Their Peculiarities
The Graph: Primary Source for Aggregated Data
Most major protocols have official subgraphs: Uniswap, Curve, Aave, Compound, Balancer, Yearn. The Graph Studio allows querying historical and current data via GraphQL.
Problems we encounter:
- Latency. Subgraphs update with a 1–10 minute delay after on-chain events. Not suitable for real-time monitoring, but fine for historical data and dashboards.
- Stale subgraphs. The Uniswap v2 subgraph hasn't been maintained by the team for a long time; data may be incomplete. For Uniswap v3, the official subgraph periodically lags during high volume.
-
Pagination. The Graph returns a maximum of 1000 records per query. To fetch all Uniswap v3 pools (over 50,000), you need pagination using
skiporid_gtpattern.
query GetPools($lastId: String) {
pools(first: 1000, where: { id_gt: $lastId }, orderBy: id) {
id
token0 { symbol, decimals }
token1 { symbol, decimals }
totalValueLockedUSD
volumeUSD
feeTier
}
}
- TVL nuance in The Graph. The Uniswap v3 subgraph calculates TVL as the sum of token values in USD using an internal price feed. This price feed sometimes gives incorrect values for low-liquidity tokens — a pool with a real TVL of $500k might show as $50M due to a manipulated price of one token. This needs to be cross-checked with an external source.
On-Chain Queries for Accurate Data
For data that needs to be exact and current — direct eth_call to contracts:
- Aave v3 TVL:
Pool.getReserveData(asset)returnsaToken.totalSupply() * liquidityIndex. For each asset in each market. - Curve APY:
Minter.minted(gauge, user)for CRV emission,gauge.inflation_rate()for current rate. Real APY =(crv_per_year * crv_price) / gauge_tvl_usd. - Uniswap v3 fee APY:
positions.tokensOwed0/1— accumulated fees. For general pool APY:pool.feeGrowthGlobal0X128— delta over period / liquidity.
Multicall3 (0xcA11bde05977b3631167028862bE2a173976CA11) is deployed on all major chains, allowing batching hundreds of eth_call into one transaction. Instead of 100 individual RPC requests — one batch. For scraping, this is critical for performance. Multicall3 is 10 times more efficient than sequential eth_call.
DeFi Llama API
https://api.llama.fi — public API without key for TVL data of most protocols. Data structure:
GET /tvl/{protocol} → current TVL
GET /protocol/{protocol} → historical TVL + breakdown
GET /pools → APY for all pools (~10k records)
/pools is a goldmine: it already calculates APY for thousands of pools across all chains. However, DeFi Llama updates data every few minutes — for real-time tasks, you need your own calculation.
| Source | Latency | TVL Accuracy | Request Cost |
|---|---|---|---|
| The Graph | 1–10 min | Medium (manipulation risk) | Free (1000 requests/day) |
| On-chain (eth_call) | Real-time | High | Gas, RPC limits |
| DeFi Llama | few min | Medium | Free, no key |
How to Normalize Data from Different Protocols?
Each protocol returns data in its own format. Normalization is key. We convert everything to a unified schema: { protocolId, chainId, poolAddress, tvlUsd, apy, timestamp }. A unified schema enables cross-protocol comparisons. For this, we use Node.js scripts with TypeScript and the ethers.js library. Our normalizer employs a schema-agnostic adapter pattern to handle protocol-specific quirks.
Scraping System Architecture
Data Collection Layers
Scheduler (cron / event-driven)
├── GraphQL Fetcher (The Graph subgraphs)
├── On-chain Fetcher (Multicall3 + ethers.js)
├── HTTP Fetcher (DeFi Llama, CoinGecko)
└── WebSocket Listener (real-time events)
↓
Normalizer (unified format)
↓
TimescaleDB / PostgreSQL
↓
API (REST/GraphQL)
The Normalizer is the key component. Each protocol returns data in its own format. Normalization: { protocolId, chainId, poolAddress, tvlUsd, apy, timestamp }. A unified schema enables cross-protocol comparisons.
APY Calculation
APY = Annual Percentage Yield with compounding. For most DeFi protocols, raw data is APR (without compounding), which needs to be converted:
APY = (1 + APR/n)^n - 1, where n is the number of compounding periods per year.
For lending protocols, APR usually already includes compounding (Aave v3 uses liquidityRate). For LP positions, it does not: fees are accrued without reinvestment.
Real APY components for a Uniswap v3 LP position:
- Trading fees APR (depends on volume and position range)
- Liquidity mining rewards (if incentives exist)
- Minus impermanent loss (historical estimate)
Why APY Without Subtracting IL Is Misleading?
An honest APY without subtracting impermanent loss shows inflated returns. In reality, with a wide range LP position, IL can eat up to 80% of profits. We show both numbers: fee APY and fee APY minus IL.
Error Handling and Rate Limiting
Alchemy free tier: 300 CUPS (compute units per second). One eth_call = 10–40 CU, Multicall3 batch = 20 CU regardless of the number of queries inside. We batch as much as possible.
The Graph: 1000 requests per day on the free plan. We use a cache with TTL — most data doesn't need to be refreshed more often than every 5 minutes.
Retry with exponential backoff on all HTTP requests. Dead letter queue for failed fetches — we don't lose data during temporary RPC failures.
| Component | Source | Peculiarities |
|---|---|---|
| Trading fees | On-chain / The Graph | Depends on volume and range |
| Rewards | Merkle drop / incentives | Requires monitoring |
| IL | On-chain price | Estimated historically |
Tech Stack
TypeScript + Node.js for scrapers. PostgreSQL + TimescaleDB for time-series storage. Redis for caching intermediate data. Docker Compose for local development.
ethers.js v6 for on-chain interactions. graphql-request for The Graph queries. p-limit for concurrency control (don't hammer RPC providers). TimescaleDB hypertables enable efficient time-series queries over millions of records.
Typical Mistakes When Scraping DeFi Data
- Using only one source without validation. TVL from an unverified subgraph can be 10x higher than the real value due to manipulated prices.
- Not accounting for impermanent loss when calculating APY for LP positions.
- Ignoring rate limits — the scraper fails with errors, data is lost.
- Storing all data in a single table without partitioning — historical queries become slow.
What the Work Includes
- Analysis of protocols and data sources for your task.
- Development of a scraper with normalization and validation.
- Setup of TimescaleDB with TTL and partitioning.
- REST/GraphQL API with caching.
- Documentation of data structure and endpoints.
- One month of support after delivery.
- Deployment scripts and access to a private dashboard.
- Guaranteed data accuracy with cross-validation from multiple sources.
Timeline Estimates
A scraper for 2–3 protocols on a single chain with a basic API: 2–3 days. A multi-protocol, multi-chain system with historical data and normalization: 1–2 weeks, depending on the number of sources and APY accuracy requirements. Typical project cost ranges from $2,000 to $10,000.
We will evaluate your project and offer a turnkey solution. Contact us to discuss the details. Order development of a DeFi data scraper.