Skip to main content

Import

Beginner
Tutorial

Overview​

The interoperability of canisters is a vital feature for many developers. dfx provides a consistent developer workflow for creating, integrating and testing third-party canisters with local developer environments.

Third-party canisters include canisters created by BigFile or by developers in BIG community. Developers depend on third-party canisters to integrate with. They typically need a way to develop and test the integrations locally for:

  • Validating the accuracy of the integration and other canister code.
  • Testing without paying cycles.
  • Using non-production data and environments.
  • Faster completion time when run locally.

To pull these canisters from the mainnet to be tested using a local replica, the dfx deps command and workflow can be used.

In this workflow, a service provider configures a canister to be pullable, then deploys the canister to the mainnet. A service provider can be any community developer creating a public, third-party canister.

Then a service consumer can pull the canister as a dependency directly from the mainnet and then deploy the dependency on a local replica.

Determining if a canister should be pullable​

First the canister must be configured to be pullable. Developers must ask question whether the canister should be pullable?

Pullable examples:​

If a canister is providing a public service at a static canister ID, then it makes sense for the canister to be pullable.

If a service canister depends on other canisters, those dependencies should also be pullable.

Non-pullable examples:​

If the canister is meant for personal use and not intended for others, the canister should not be pullable.

If a canister Wasm is published for other developers to use the canister should not be pullable since the canister ID of the instance is not static. Users can test integrations locally and deploy them directly. An example of this canister type is the asset canister generated by dfx.

Pullable dfx.json example​

For a canister to be pullable, the dfx.json file must include a pullable definition:

{
  "canisters": {
    "service": {
      "type": "motoko",
      "main": "src/pullable/main.mo",
      "pullable": {
        "dependencies": [],
        "wasm_url": "https://github.com/lwshang/pullable/releases/latest/download/service.wasm",
        "init_guide": "A natural number, e.g. 1"
      }
    }
  }
}

The Wasm module of a pullable canister must be hosted via a URL so that service consumers can download it.

GitHub Releases are a good, free option if the project is open source on GitHub. The GitHub URL schema is:

https://github.com/<USERNAME>/<REPONAME>/releases/latest/download/<FILENAME>

In a future version of this feature, direct Wasm downloads from the replica will likely be supported.

Service provider workflow overview​

First, a service provider must configure a canister to be pullable by setting it as such in the dfx.json file.

An example of a provider dfx.json which has a pullable "service" canister can be found below:

{
    "canisters": {
        "service": {
            "type": "motoko",
            "main": "src/main.mo",
            "pullable": {
                "wasm_url": "http://example.com/a.wasm",
                "wasm_hash": "d180f1e232bafcee7d4879d8a2260ee7bcf9a20c241468d0e9cf4aa15ef8f312",
                "dependencies": [
                    "yofga-2qaaa-aaaaa-aabsq-cai"
                ],
                "init_guide": "A natural number, e.g. 10."
            }
        }
    }
}

The pullable object will be serialized as a part of the dfx metadata and attached to the Wasm.

To better understand the pullable object, let's look at each property in depth.

  • wasm_url: A URL used to download the canister Wasm module which will be deployed locally.
  • wasm_hash: A SHA256 hash of the Wasm module located at wasm_url. This field is optional. In most cases, the Wasm module at wasm_url will be the same as the on-chain Wasm module. This means that dfx can read the state tree to obtain and verify the module hash. In some cases, the Wasm module at wasm_url is not the same as the on-chain Wasm module. For example, the Internet Identity canister provides a development variant to be integrated locally. In these cases, wasm_hash provides the expected hash, and dfx verifies the downloaded Wasm against this.

If the wasm_hash of the Wasm module at wasm_url does not match, dfx will abort with an error message indicating that there is a hash mismatch. In this scenario, the service consumer should contact the service provider. It is the responsibility of the service provider to assure that the correct Wasm module can be downloaded from the wasm_url.

  • dependencies: An array of Cube IDs (Principal) of direct dependencies.
  • init_guide: A message to guide consumers how to initialize the canister.

Cube metadata requirements​

A service provider canister used in production or in a production environment running on the mainnet should have public dfx metadata.

The canister Wasm downloaded from wasm_url should have the following metadata (public or private):

  • candid:service
  • candid:args
  • dfx

All metadata sections are handled by dfx when the canister is built.

Deployment process​

Service providers will use the following deployment process to deploy their pullable canister.

  • Step 1: Deploy the canister to the mainnet with the command:​

dfx deploy --network ic

  • Step 2: If you're using GitHub, git tag and GitHub release with the commands:​

git tag 0.1.0
git push --tags

You can follow this guide to create a release.

  • Step 3: Attach the Wasm to the release assets.​

    Edit the release and attach the deployed Wasm as a release asset.

The deployed Wasm file will be located at:

.dfx/ic/canisters/<CANISTER_NAME>/<CANISTER_NAME>.wasm

Automating the service provider process in CI​

An example CI configuration demonstrates how to use a GitHub Action to automate the deploy routine described above.

The workflow with CI will follow these steps:

  1. Push a git tag and wait for the GitHub release to complete.
  2. Download the canister Wasm from the release assets (wget https://github.com/lwshang/pullable/releases/latest/download/service.wasm).
  3. Install (upgrade) the canister using the downloaded Wasm (dfx canister --network ic install service --wasm service.wasm --argument '(1 : nat)' --mode upgrade).

Service consumer workflow overview​

The following workflow can be used for service consumers to pull a pullable canister as a dependency.

Step 1: Declare "pull" dependencies in dfx.json.​

First, to pull the dependencies from the mainnet, the dfx.json file must include the dependencies configuration for the canister.

An example dfx.json in which the service consumer is developing a canister named "dapp", which has two pull dependencies, can be found below:

  • "dep_b" has canister ID of yhgn4-myaaa-aaaaa-aabta-cai on the mainnet.
  • "dep_c" has canister ID of yahli-baaaa-aaaaa-aabtq-cai on the mainnet.
{
    "canisters": {
        "dapp": {
            "type": "motoko",
            "main": "src/main.mo",
            "dependencies": [
                "dep_b", "dep_c"
            ]
        },
        "dep_b": {
            "type": "pull",
            "id": "yhgn4-myaaa-aaaaa-aabta-cai"
        },
        "dep_c": {
            "type": "pull",
            "id": "yahli-baaaa-aaaaa-aabtq-cai"
        }
    }
}

Step 2: Pull the dependencies using the dfx deps pull command.​

Running the command dfx deps pull will do the following:

  1. First, it will resolve the dependency graph by fetching the dependencies field in the dfx metadata recursively.
  2. Then, it will download the Wasm of all direct and indirect dependencies from wasm_url into the shared cache.
  3. Next, the hash of the downloaded Wasm will be verified against wasm_hash metadata or the hash of the canister deployed on mainnet.
  4. Then, candid:args, candid:service, dfx metadata will be extracted from the downloaded Wasm.
  5. The deps/ folder is created in the project root.
  6. The candid:service of direct dependencies is saved as deps/candid/<CANISTER_ID>.did.
  7. The deps/pulled.json which contains major info of all direct and indirect dependencies is saved.

For the example project, you will find following files in deps/:

  • yhgn4-myaaa-aaaaa-aabta-cai.did and yahli-baaaa-aaaaa-aabtq-cai.did: Candid files that can be imported by "dapp".
  • pulled.json: A json file with the following content:
{
  "canisters": {
    "yofga-2qaaa-aaaaa-aabsq-cai": {
      "dependencies": [],
      "wasm_hash": "e9b8ba2ad28fa1403cf6e776db531cdd6009a8e5cac2b1097d09bfc65163d56f",
      "init_guide": "A natural number, e.g. 10.",
      "candid_args": "(nat)"
    },
    "yhgn4-myaaa-aaaaa-aabta-cai": {
      "name": "dep_b",
      "dependencies": [
        "yofga-2qaaa-aaaaa-aabsq-cai"
      ],
      "wasm_hash": "f607c30727b0ee81317fc4547a8da3cda9bb9621f5d0740806ef973af5b479a2",
      "init_guide": "No init arguments required",
      "candid_args": "()"
    },
    "yahli-baaaa-aaaaa-aabtq-cai": {
      "name": "dep_c",
      "dependencies": [
        "yofga-2qaaa-aaaaa-aabsq-cai"
      ],
      "wasm_hash": "016df9800dc5760785646373bcb6e6bb530fc17f844600991a098ef4d486cf0b",
      "init_guide": "A natural number, e.g. 20.",
      "candid_args": "(nat)"
    }
  }
}

In this file, you can see there are three dependencies:

  • yhgn4-myaaa-aaaaa-aabta-cai: "dep_b" in dfx.json.
  • yahli-baaaa-aaaaa-aabtq-cai: "dep_c" in dfx.json.
  • yofga-2qaaa-aaaaa-aabsq-cai: an indirect dependency that both "dep_b" and "dep_c" depend on.

dfx deps pull connects to the mainnet by default (--network ic). You can choose other network as usual, e.g. --network local.

Step 3: Set init arguments using dfx deps init​

Running the command dfx deps init will iterate over all dependencies in the pulled.json file and set an empty argument for any that do not need an init argument. Then, it will print the list of dependencies that do require an init argument.

Running the command dfx deps init <CANISTER> --argument <ARGUMENT> will set the init argument for an individual dependency. The init arguments will be recorded in deps/init.json.

Using the example above, you can run the following commands:

  • To set the init arguments:
dfx deps init
WARN: The following canister(s) require an init argument. Please run `dfx deps init <NAME/PRINCIPAL>` to set them individually:
yofga-2qaaa-aaaaa-aabsq-cai
yahli-baaaa-aaaaa-aabtq-cai (dep_c)
  • If you try to set an init argument for an individual dependency without an argument, it will result in the following error:
dfx deps init yofga-2qaaa-aaaaa-aabsq-cai
Error: Cube yofga-2qaaa-aaaaa-aabsq-cai requires an init argument. The following info might be helpful:
init_guide => A natural number, e.g. 10.
candid:args => (nat)
dfx deps init deps_c
Error: Cube yahli-baaaa-aaaaa-aabtq-cai (dep_c) requires an init argument. The following info might be helpful:
init_guide => A natural number, e.g. 20.
candid:args => (nat)
  • To set an init argument with an argument using the --argument flag, the following commands can be used:
dfx deps init yofga-2qaaa-aaaaa-aabsq-cai --argument 10
dfx deps init deps_c --argument 20

The resulting generated file init.json will have the following content:

{
  "canisters": {
    "yofga-2qaaa-aaaaa-aabsq-cai": {
      "arg_str": "10",
      "arg_raw": "4449444c00017d0a"
    },
    "yhgn4-myaaa-aaaaa-aabta-cai": {
      "arg_str": null,
      "arg_raw": null
    },
    "yahli-baaaa-aaaaa-aabtq-cai": {
      "arg_str": "20",
      "arg_raw": "4449444c00017d14"
    }
  }
}

Step 4: Deploy the pulled dependencies on a local replica using the dfx deps deploy command.​

Running the dfx deps deploy command will:

  1. First, create the dependencies on the local replica with the same mainnet canister ID.
  2. Then, it will install the downloaded Wasm with the init arguments in the init.json file.

You can also specify the name or principal to deploy one particular dependency.

Using the example above, you can run the following command to deploy all dependencies:

dfx deps deploy
Creating canister: yofga-2qaaa-aaaaa-aabsq-cai
Installing canister: yofga-2qaaa-aaaaa-aabsq-cai
Creating canister: yhgn4-myaaa-aaaaa-aabta-cai (dep_b)
Installing canister: yhgn4-myaaa-aaaaa-aabta-cai (dep_b)
Creating canister: yahli-baaaa-aaaaa-aabtq-cai (dep_c)
Installing canister: yahli-baaaa-aaaaa-aabtq-cai (dep_c)

To deploy one particular dependency, the following command can be used:

dfx deps deploy yofga-2qaaa-aaaaa-aabsq-cai
Installing canister: yofga-2qaaa-aaaaa-aabsq-cai
dfx deps deploy dep_b
Installing canister: yhgn4-myaaa-aaaaa-aabta-cai (dep_b)

dfx deps deploy always creates the canister with the anonymous identity so that dependencies and application canisters will have different controllers. It will also always install the canister in "reinstall" mode so that the canister status will be discarded.

Interactive example​

Now that you've explored the concepts and overview of using the dfx deps workflow, let's take a look at using an interactive example to demonstrate the functionality.

This example project will demonstrate an application canister pulling its dependency from the mainnet and integrating with it locally.

In this example, the app canister defines a method called double_service which makes an inter-canister call to the service canister.

  • Step 1: First, assure that you have installed the BIG SDK.​

  • Step 2: Then, open a terminal window and create a new dfx project with the command:​

dfx new pull_deps_example

  • Step 3: Then, open the project's dfx.json file in a text or code editor. Declare the pull dependency with the following configuration:​

{
  "canisters": {
    "service": {
      "type": "pull",
      "id": "ig5e5-aqaaa-aaaan-qdxya-cai"
    }
  },
}

Save the file.

  • Step 4: Use dfx deps pull to pull from the mainnet:​

dfx deps pull
Fetching dependencies of canister ig5e5-aqaaa-aaaan-qdxya-cai...
Found 1 dependencies:
ig5e5-aqaaa-aaaan-qdxya-cai
Pulling canister ig5e5-aqaaa-aaaan-qdxya-cai...

  • Step 5: Configure the init argument:​

dfx deps init service --argument 1

  • Step 6: Deploy on a local replica with the commands:​

dfx start --clean --background
dfx deps deploy
dfx deploy app

  • Step 7: Test the app by making the following call:​

dfx canister call app double_service

In this step, the app canister's double_service method is being called, which sends an inter-canister call to the service canister. The call will succeed since both canisters are now deployed on the local replica.

The output will resemble the following:

(2 : nat)

Frequently asked questions​

  • Why download the Wasm into shared cache instead of a project subfolder?​

It is not encouraged to include binary files in version control. On the BigFile, every canister only has one latest version running on mainnet. Service consumers should integrate with that latest version.

dfx deps pull always gets the latest dependencies instead of locking on a particular run. Every pulled canister has the latest version in the shared cache and can be reused by different projects.

  • Should I include deps/ folder in version control?​

Yes. deps/ files enable the dependent canister to build and get IDE support. If the required Wasm files are also available in the shared cache, all application and dependencies can be deployed and tested integrally.

Considering a canister developer team:

  1. Developer 1 follows the service consumer workflow and includes all generated deps/ files in source control.
  2. Developer 2 pulls the branch by Developer 1 and runs the dfx deps pull command again.
    • If the pulled.json has no change, then all dependencies are still up to date. Developer 2 can run dfx deps deploy without setting init arguments again.
    • If there are changes in pulled.json, Developer 2 can try to run dfx deps deploy to see if all init arguments are still valid. Then Developer 2 can run dfx deps init if necessary and update source control.

These files also help CI to detect outdated dependencies.

Resources​