README
¶
Rosetta DeFiChain
ROSETTA-DEFICHAIN IS CONSIDERED ALPHA SOFTWARE. USE AT YOUR OWN RISK! COINBASE ASSUMES NO RESPONSIBILITY NOR LIABILITY IF THERE IS A BUG IN THIS IMPLEMENTATION.
Overview
rosetta-defichain provides a reference implementation of the Rosetta API for
DeFiChain in Golang. If you haven't heard of the Rosetta API, you can find more
information here.
Features
- Rosetta API implementation (both Data API and Construction API)
- UTXO cache for all accounts (accessible using
/account/balance) - Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address
Usage
As specified in the Rosetta API Principles,
all Rosetta implementations must be deployable via Docker and support running via either an
online or offline mode.
YOU MUST INSTALL DOCKER FOR THE FOLLOWING INSTRUCTIONS TO WORK. YOU CAN DOWNLOAD DOCKER HERE.
Install
Running the following commands will create a Docker image called rosetta-defichain:latest.
From GitHub
To download the pre-built Docker image from the latest release, run:
curl -sSfL https://raw.githubusercontent.com/DeFiCh/rosetta-defichain/master/install.sh | sh -s
Do not try to install rosetta-defichain using GitHub Packages!
From Source
After cloning this repository, run:
make build-local
Run
Running the following commands will start a Docker container in
detached mode with
a data directory at <working directory>/data and the Rosetta API accessible.
Mainnet:Online
docker run -d --ulimit "nofile=100000:100000" -v "$(pwd)/data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 rosetta-defichain:latest
If you cloned the repository, you can run make run-mainnet-online.
Mainnet:Offline
docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 rosetta-defichain:latest
If you cloned the repository, you can run make run-mainnet-offline.
Testnet:Online
docker run -d --ulimit "nofile=100000:100000" -v "$(pwd)/data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 rosetta-defichain:latest
If you cloned the repository, you can run make run-testnet-online.
Testnet:Offline
docker run -d -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 rosetta-defichain:latest
If you cloned the repository, you can run make run-testnet-offline.
General information about ports
- Online API port is 8080
- Offline API port is 8081
Requirements
Machine
The following results may be expected by running rosetta-defichain on a machine with characteristics comparable to the following:
- x8 CPU i5-8250U @ 1.60GHz
- ~8 GB RAM & ~4-12 SWAP
Mainnet:
- about 40 GB of disk space usage
- about 4 hours to sync
Testnet:
- about 20 GB of disk space usage
- about 2 hours to sync
NOTE: presented values may change, since blockchain data is growing up continuously
Network Settings
To increase the load rosetta-defichain can handle, it is recommended to tune your OS
settings to allow for more connections. On a linux-based OS, you can run the following
commands (source):
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_max_syn_backlog=10000
sysctl -w net.core.somaxconn=10000
sysctl -p (when done)
We have not tested rosetta-defichain with net.ipv4.tcp_tw_recycle and do not recommend
enabling it.
You should also modify your open file settings to 100000. This can be done on a linux-based OS
with the command: ulimit -n 100000.
Memory-Mapped Files
rosetta-defichain uses memory-mapped files to
persist data in the indexer. As a result, you must run rosetta-defichain on a 64-bit
architecture.
If you receive a kernel OOM, you may need to increase the allocated size of swap space on your OS. There is a great tutorial for how to do this on Linux here.
Architecture
rosetta-defichain uses the syncer, storage, parser, and server package
from rosetta-sdk-go instead
of a new DeFiChain-specific implementation of packages of similar functionality. Below
you can find a high-level overview of how everything fits together:
+------------------------------------------------------------------+
| |
| +--------------------------------------+ |
| | | |
| | indexer | |
| | | |
| | +--------+ | |
+-------------------+ pruner <----------+ | |
| | +--------+ | | |
+-----v----+ | | | |
| defid | | +------+--------+ | |
+-----+----+ | +--------> block_storage <----+ | |
| | | +---------------+ | | |
| | +---+----+ | | |
+-------------------> syncer | | | |
| +---+----+ | | |
| | +--------------+ | | |
| +--------> coin_storage | | | |
| +------^-------+ | | |
| | | | |
+--------------------------------------+ |
| | |
+-------------------------------------------------------------------------------------------+ |
| | | | |
| +------------------------------------------------------------+ | | |
| | | | |
| | +---------------------+-----------------------+------+ | |
| | | | | | |
| +-------+---------+ +-------+---------+ +-------+-------+ +-----------+----------+ | |
| | account_service | | network_service | | block_service | | construction_service +--------+
| +-----------------+ +-----------------+ +---------------+ +----------------------+ |
| |
| server |
| |
+-------------------------------------------------------------------------------------------+
Optimizations
- Automatically prune defid while indexing blocks
- Reduce sync time with concurrent block indexing
- Use Zstandard compression to reduce the size of data stored on disk without needing to write a manual byte-level encoding
Concurrent Block Syncing
To speed up indexing, rosetta-defichain uses concurrent block processing
with a "wait free" design (using channels instead of sleeps to signal
which threads are unblocked). This allows rosetta-defichain to fetch
multiple inputs from disk while it waits for inputs that appeared
in recently processed blocks to save to disk.
+----------+
| defid |
+-----+----+
|
|
+---------+ fetch block data / unpopulated txs |
| block 1 <------------------------------------+
+---------+ |
+--> tx 1 | |
| +---------+ |
| | tx 2 | |
| +----+----+ |
| | |
| | +---------+ |
| | | block 2 <-------------------+
| | +---------+ |
| +-----------> tx 3 +--+ |
| +---------+ | |
+-------------------> tx 4 | | |
| +---------+ | |
| | |
| retrieve previously synced | +---------+ |
| inputs needed for future | | block 3 <--+
| blocks while waiting for | +---------+
| populated blocks to save to +---> tx 5 |
| disk +---------+
+------------------------------------> tx 6 |
| +---------+
|
|
+------+--------+
| coin_storage |
+---------------+
Testing with rosetta-cli
To validate rosetta-defichain, install rosetta-cli
and run one of the following commands:
rosetta-cli check:data --configuration-file rosetta-cli-conf/testnet/config.jsonrosetta-cli check:construction --configuration-file rosetta-cli-conf/testnet/config.jsonrosetta-cli check:data --configuration-file rosetta-cli-conf/mainnet/config.json
NOTE: as you may notice, we suggests to use our fork instead of an original rosetta-cli implementation. This is because current version of rosetta-cli doesn't call /account/coins endpoint to fetch UTXOs for every prefunded account. That's why check for construction API with prefunded account doesn't work. For the issue details, please, see the following description
If you cloned the repository, you can run the following make rules respectively:
make check-testnet-data-apimake check-testnet-construction-apimake check-mainnet-data-api
Future Work
- Add CI test using
rosetta-clito run on each PR
Development
make depsto install dependenciesmake testto run testsmake lintto lint the source codemake salusto check for security concernsmake buildto build a Docker image from the remote repositorymake build-localto build a Docker image from the local contextmake coverage-localto generate a coverage report
License
This project is available open source under the terms of the Apache 2.0 License.
Documentation
¶
There is no documentation for this package.
Source Files
Directories