Unmarshaling Blockchain Data
A deep dive into how raw data is converted into useful data
Data from blockchains, in itself, is not much useful and almost nothing is directly useable for developers building DeFi applications. A transformation process is needed to convert data into information that once decoded, is valuable for decentralized applications to make use of.
How is Data stored in Blockchains?
Blockchain is a list of blocks and each block is attached to a previous and next block. A Block contains an array of transactions. Each transaction is a function call for an operation on the blockchain. Eg: Transfer, Mint, AddLiquidity, etc. Transactions have internal functions called to achieve an operation and emit log events. Every operation performed on a blockchain is using a hash code.
Let’s explore this with an example of a random Transaction ID on BSC
0xca629c082f180c57180fbc6c030fcd23dd3982a614a1a9d6f127569a5461c976
The above transaction is to add liquidity of two tokens in PancakeSwap. This triggers the PancakeSwap “addLiquidity” function in BSC.
The input data which is sent to process the transaction is
The first 8 characters e8e33700 excluding 0x, represents the function name. Similarly, the data can be passed through the abi-decoder to find the other details.
Pancake swap addLiquidity ABI is
addLiquidity(address,address,uint256,uint256,uint256,uint256,address,uint256)
The function definition is hashed using Keccak-256 which will result in e8e33700.
For complete execution of the function, there are a set of internal function calls. In the above case approve, transfer and mint functions are called, which can be seen here
How to retrieve data from Blockchain
There are a variety of challenges while indexing the data, the first step is to index the data from the Genesis Block. Ethereum chain has 11960866 blocks, close to 1.5M transactions a day(at the time of writing). The main net is slow so there is a limit as to how much we can fetch and index.
There are multiple ways to extract data from a node running the Mainnet. Using JSON RPC, WebSocket.
Likewise, there are a wide range of API support can be found here https://eth.wiki/json-rpc/API
While we can extract the data from the above methods, we can only get the response in a bunch of hashcodes and hexadecimal, which is hard to make sense of.
At Unmarshal
We fetch blockchain data in the raw format, run all the data through our in-house decoder and transformation tools to make the data meaningful and rich. We constantly sync with nodes and store it in our indexers in the best way possible, so as to retrieve it instantaneously with the scale that our users need. Unmarshal has multiple nodes running, to fetch the data parallelly, store the data, validate each block, and track the errors on failure.
Below is the difference between data that is fetched from blockchain directly and Data produced by Unmarshal.
Consistency, Isolation, and Durability of the data are very important which Unmarshal makes sure of. We constantly strive to keep our indexers updated to the latest block with no anomalies. Consumer applications can directly integrate our end-points without spending any effort on decoding and transforming the data.
Conclusion:
Blockchains only understand the language of Hash. Relevant data like the address transaction history, interactions with a contract, existing liquidity positions, etc. cannot be fetched from blockchains. The article is only a gist of intricacies in fetching data from blockchains. There is a lot more to it in terms of receipts, logs, events, etc which happens for a particular transaction. At Unmarshal, we do all the heavy-lifting to make data relatable to a wide variety of DeFi use cases.
For more information about Unmarshal, connect with us
In case of queries, reach us out at “info@unmarshal.io”
