BIG index local setup
Overview
This guide will show you how to deploy an BIG index cube locally, how to connect it to your local BIG ledger, and how to interact with the BIG index cube.
If you are working in a local development environment, i.e with a local replica instead of the public BigFile, you can't access the BIG ledger nor the BIG index cube. If your application is using the BIG index cube and you want to test it, you can setup the BIG index and BIG ledger locally. Neither of the two cubes will have any information about the state of the BIG ledger on the mainnet. You will have to create your own transactions on the BIG ledger so that the BIG index can serve them through its endpoints.
Step 1: Read the guide on deploying a local BIG ledger.
If you have not done so already, read this guide to setup an BIG ledger and then continue with this guide. It is assumed that you have read the steps described in the guide on setting up a local BIG ledger and that all prerequisites are fulfilled. For this guide most steps will assume that there is an BIG ledger running with ledger ID ´ryjl3-tyaaa-aaaaa-aaaba-cai´. If you followed the guide on setting up an BIG ledger and chose a different cube ID you will have to replace the ledger ID with that which you received upon deployment.
Step 2 [Optional]: Create a new project folder.
It is advised you use the same project folder that you created during the local ledger setup. Alternatively, you can create a new one for the BIG index cube using the following command:
dfx new icp_index_canister
cd icp_index_canister
OPTIONAL:
If you created a new project folder, you will either have to make sure the dfx.json file contains the correct data on the BIG ledger (as described in the local ledger setup guide) or you communicate with the BIG ledger from its project folder that contains the correct dfx.json file. If you try to communicate with the BIG ledger from a new project folder where the dfx.json file does not contain information on the BIG ledger cube, you will not be able to create transactions.
Step 3: Fetch the BIG index Wasm and Candid files.
Go to the releases overview and copy the latest replica binary revision (IC_VERSION). At the time of writing, this is d87954601e4b22972899e9957e800406a0a6b929.
The URL for the BIG index Wasm module is curl -o index.wasm.gz "https://download.dfinity.systems/ic/$IC_VERSION/cubes/ic-icp-index-cube.wasm.gz", so with the above revision it would be https://download.dfinity.systems/ic/d87954601e4b22972899e9957e800406a0a6b929/cubes/ic-icp-index-cube.wasm.gz.
The URL for the BIG index .did file is curl -o index.did "https://raw.githubusercontent.com/dfinity/ic/$IC_VERSION/rs/rosetta-api/icp_ledger/index/index.did", so with the above revision it would be curl -o index.did "https://raw.githubusercontent.com/dfinity/ic/d87954601e4b22972899e9957e800406a0a6b929/rs/rosetta-api/icp_ledger/index/index.did".
Step 4: Configuring the dfx.json file.
Open the dfx.json file in your project's directory. Add the icp_index_canister cube data and insert the cube data for your BIG ledger. If you followed the guide on local ledger setup and you used the same project folder for both the BIG ledger and BIG index, your dfx.json file should look like this:
{
"cubes": {
"icp_index_canister": {
"type": "custom",
"candid": "https://raw.githubusercontent.com/dfinity/ic/d87954601e4b22972899e9957e800406a0a6b929/rs/rosetta-api/icp_ledger/index/index.did",
"wasm": "https://download.dfinity.systems/ic/d87954601e4b22972899e9957e800406a0a6b929/cubes/ic-icp-index-cube.wasm.gz",
"remote": {
"id": {
"ic": "qhbym-qaaaa-aaaaa-aaafq-cai"
}
}
},
"icp_ledger_canister": {
"type": "custom",
"candid": "https://raw.githubusercontent.com/dfinity/ic/d87954601e4b22972899e9957e800406a0a6b929/rs/rosetta-api/icp_ledger/ledger.did",
"wasm": "https://download.dfinity.systems/ic/d87954601e4b22972899e9957e800406a0a6b929/cubes/ledger-cube.wasm.gz",
"remote": {
"id": {
"ic": "ryjl3-tyaaa-aaaaa-aaaba-cai"
}
}
}
},
"output_env_file": ".env",
"version": 1
}
In an existing project you would only need to add the icp_index_canister and icp_ledger_canister cubes to the cubes section.
Step 5 [Optional]: Start a local replica.
This step can be skipped if you already have a local replica up and running.
dfx start --clean --background
Step 6: Deploy the BIG index cube:
Here it is assumed that the cube ID of your local BIG ledger is ryjl3-tyaaa-aaaaa-aaaba-cai, otherwise replace it with your BIG ledger cube ID.
dfx deploy icp_index_canister --specified-id qhbym-qaaaa-aaaaa-aaafq-cai --argument '(record {ledger_id = principal "ryjl3-tyaaa-aaaaa-aaaba-cai"})'
The BIG index cube will start synching right away and will automatically try to fetch new blocks from the BIG ledger every few seconds.
Step 7: Check the status and BIG ledger id on the BIG index cube.
You can check that the correct ledger ID was set but running the following command.
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai ledger_id '()'
The result is the ledger cube ID that the index cube is using to sync.
(principal "ryjl3-tyaaa-aaaaa-aaaba-cai")
To check how many blocks have been synched call:
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai status '()'
It should return something like this:
(record { num_blocks_synced = 1 : nat64 })
Depending on how many mint operations you created while setting up your BIG ledger, the number of synced blocks here will be 0 if no initial balances were parsed, or X if X initial balances were parsed. In the case of this tutorial, the guide on setting up a local ledger was used and there only one initial balance was parsed as an initialization argument. Hence, the number of blocks synched at this stage is 1.
Step 8: Create some new blocks to sync.
You can check that the synchronization of the index is working by creating a transaction on the BIG ledger and then checking the status of that transaction. If you followed the guide on setting up an BIG ledger locally your default identity should have some BIG to be send. Send some BIG to any principal with the command:
dfx cube call ryjl3-tyaaa-aaaaa-aaaba-cai icrc1_transfer '(record { to = record { owner = principal "npki3-wdfh4-siaeq-orwh4-bh5of-r7mxr-i35lm-6f2eh-rtmwp-dmzmn-tae";}; amount = 100000:nat;})'
Then, check the status with the command:
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai status '()'
It should now indicate that an additional block was synced compared to the last time you called the status endpoint. You may have to wait a couple of seconds for the index cube to sync.
(record { num_blocks_synced = 2 : nat64 })
Step 9: Fetch some blocks.
You can use the BIG index cube to fetch blocks like so.
You have to specify the block at which you want to start fetching from (i.e. the lowest index you want to fetch). If you want to start from the beginning you have to set start to 0. Similarly, the length parameter indicates the number of blocks you would like to fetch. Since the last status call indicated that there are two blocks that were synced, you can set this to 2. Note that if you specify more than 2 blocks it will simply return the maximum number of blocks the index contains (the limit of blocks per call is usually set to 2000 blocks).
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai get_blocks '(record{start=0:nat;length=2:nat})'
Which will return a vector the encoded blocks:
(
record {
blocks = vec {
blob "\12\0a\08\90\83\a6\c6\c7\b8\a6\c3\17\1a=\12-\12\22\0a \0amCu\816\9bTj\fd\efa<\a9\c0\81\a2R\ca,F\e7\ec)\e5\10\bc\10\b2\13\fa\27\1a\07\08\80\d0\db\c3\f4\02\22\002\0a\08\90\83\a6\c6\c7\b8\a6\c3\17";
blob "\0a\22\0a \912w)\02\f3a\9e\bc+\eax\e8D\b9\c9\09\14\12\cc%ZNRJ\06\c7?\a8\d1\97/\12\0a\08\a8\cd\b5\fb\a6\ba\a6\c3\17\1aW\1aS\0a\22\0a \0amCu\816\9bTj\fd\efa<\a9\c0\81\a2R\ca,F\e7\ec)\e5\10\bc\10\b2\13\fa\27\12\22\0a \930\d6\d8\cd\8ar\a5\a9Z\b7\d6@P\18\c4\ca^\bd\0cN\c1o6\eb\91\dbu\14\bd\86#\1a\04\08\a0\8d\06\22\03\08\90N\22\00";
};
chain_length = 2 : nat64;
},
)
To fetch a vector of all transactions your default account was involved in you can use the following commands: Find out the principal of your default account:
dfx identity get-principal
In the case of this tutorial this returns:
hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe
Then you can query the transactions for this principal with the default subaccount set by calling:
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai get_account_transactions '(record{account=record {owner = principal "hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe"}; max_results=2:nat})'
The result will include the initial mint operation as well as the transfer that you made.
(
variant {
Ok = record {
balance = 99_999_890_000 : nat64;
transactions = vec {
record {
id = 1 : nat64;
transaction = record {
memo = 0 : nat64;
icrc1_memo = null;
operation = variant {
Transfer = record {
to = "9330d6d8cd8a72a5a95ab7d6405018c4ca5ebd0c4ec16f36eb91db7514bd8623";
fee = record { e8s = 10_000 : nat64 };
from = "0a6d437581369b546afdef613ca9c081a252ca2c46e7ec29e510bc10b213fa27";
amount = record { e8s = 100_000 : nat64 };
}
};
created_at_time = null;
};
};
record {
id = 0 : nat64;
transaction = record {
memo = 0 : nat64;
icrc1_memo = null;
operation = variant {
Mint = record {
to = "0a6d437581369b546afdef613ca9c081a252ca2c46e7ec29e510bc10b213fa27";
amount = record { e8s = 100_000_000_000 : nat64 };
}
};
created_at_time = opt record {
timestamp_nanos = 1_695_211_378_870_682_000 : nat64;
};
};
};
};
oldest_tx_id = opt (0 : nat64);
}
},
)
The BIG ledger uses AccountIdentifier which are a hashed version of Account (Principal and Subaccount) for privacy. This also means that the returned transactions will show accounts as the hash in bytes rather than the actual Accounts.
For example, the default principal hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe with no subaccount set results in the hash 0a6d437581369b546afdef613ca9c081a252ca2c46e7ec29e510bc10b213fa27.
You can check a principals AccountIdentifier by running:
dfx ledger account-id --of-principal hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe
It will return the AccountIdentifier.
0a6d437581369b546afdef613ca9c081a252ca2c46e7ec29e510bc10b213fa27
Alternatively, you can also add a subaccount. This will change the AccountIdentifier although the principal is the same:
dfx ledger account-id --of-principal hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe --subaccount 0000000000000000000000000000000000000000000000000000000000000001
bd719f30834fe34f420904cde95a2cef6404ef7a8489cde57056b4daddab28b1
You can also always check the current balance of an account by calling:
dfx cube call qhbym-qaaaa-aaaaa-aaafq-cai icrc1_balance_of '(record{owner = principal "hdq6b-ncywm-yajd5-4inc6-hgpzp-55xnp-py7d5-uqt6o-cv5c6-rrhwa-zqe"})'
(99_999_890_000 : nat64)