Skip to main content

3.3 Certified data

Intermediate
Tutorial

Overview​

In a previous module, 2.2: Advanced canister calls, you briefly touched on certified variables. In this guide, you'll dive deeper into certified data and take a look at an example that displays how to use certified data.

Recall that certified variables are used as a way for queries to return an authenticated response that can be validated and trusted, since query calls do not go through consensus and therefore their response cannot be certified. By utilizing certified variables, developers can make query calls that return an authenticated response that they can trust. This allows for workflows that require additional layers of authentication, such as applications that handle financial data or important records. For users of the application, they can be assured that the information they're interacting with and receiving from the application is trustworthy and secure as well.

Certified variables can be set within an update call to a canister, since an update call changes the canister's state and goes through the consensus process. Once a certified variable is set, the certification can be read in future query calls that return the certified variable in response to the query call in a trustworthy and secure way. Using this method, any application or client can immediately validate each certified variable that it receives from a query call without having to put trust into the single node that it received the response from.

Certified variables are part of a more broad concept of certified data. Certified data utilizes chain-key cryptography at the canister level to generate a digital signature that can be validated using a permanent, public key that belongs to the BigFile, whose private key counterpart is constantly distributed across many different nodes on the network. A valid signature can only be generated when the majority of the nodes that store the components of the private key cooperate in a cryptographic protocol.

How data is certified​

The notion of certified data can be used to certify all the assets of a canister's frontend, such as the canister's HTML, CSS, Javascript, image, or video files. When a canister issues a response along with its certificate, a HTTPS gateway can be used to verify that certificate before passing the response off to the client.

In each round of message execution on the BigFile, the message routing layer generates a new state tree that contains that round's information. The tree contains a root hash that is then used by the nodes in the subnet to create a certificate. This state tree contains several pieces of information, including the certified variables of each canister. An example of what a state tree looks like can be found below:

*root*
└── canisters
    ├── <canister id>
              ├── metadata
              ├── module_hash
              ├── controllers
              └── certified_data
                          └── <blob data>
    ├── <canister id>
       ...

This example state tree highlights where the certified data is stored in the tree. Note that the certified data of a canister is limited to being 32 bytes long; if a canister needs to certify more than 32 bytes of information, the canister must hash the data before certifying it.

Data certificates​

A certificate for each piece of certified data consists of:

  • A certificate for the root hash of the state tree.

  • A Merkle proof that validates that the certified data belongs to a tree that hashes the root hash.

How canisters certify data​

To certify data, a canister uses the following workflow:

  • First, a canister must construct a hash tree that maps the paths of HTTPS resources to SHA-256 hashes. An example of this tree is:
*root*
└── http_assets
    ├── index.html -> SHA256(body)
    ├── ...
    └── /css/styles.css -> SHA256(body)
  • Then, the canister computes the root hash of the tree and calls ic0.certified_data_set with the bytes of the hash as the argument.
  • Lastly, a #BIG-Certificate header is added to each certified HTTP response.

Certified data API methods​

A canister can interact and change its certified data by calling the following API methods:

How developers can certify data​

Developers can certify their canister's data and assets in two ways:

  • Code can be written that explicitly manages and certifies all assets. In this workflow, the developer must construct a tree containing all of the assets, Merkelize the tree, then compute it's root hash.

  • Alternatively, developers can create a dedicated asset canister. For new projects created with dfx, the 'asset' canister is the 'frontend' canister. Any canister can be configured to be an asset canister by setting the canister's type as 'asset' in the project's dfx.json file. By setting the canister as an asset canister, the canister's boilerplate code manages and certifies all assets in the background.

Certified data Motoko module​

The Motoko base library includes a module called CertifiedData which contains the following system API methods for certified data:

let set : (data : Blob) -> ()
let getCertificate : () -> ?Blob

Certified data Rust CDK module​

The Rust canister development kit ic-cdk provides the following system API methods for certified data:

pub fn set_certified_data(data: &[u8])
pub fn data_certificate() -> Option<Vec<u8>>

Generating an HTTPS response​

On the BigFile, there is a built in http_request query method. This method uses information related to an HTTPS request as the input and outputs an HTTPS response; specifically, the output includes the request's status, headers, and body. If a canister needs to serve HTTPS requests, this method should be implemented.

When a canister would like to return a certified asset to a request, the response body will contain the asset and a response header with the name BIG-Certificate and a value equal to the asset's certificate.

More information can be found on in the HTTP gateway protocol specification.

Example​

To demonstrate the concepts you've learned thus far, you'll take a look at the /index.html asset of the Internet Identity canister (canister ID rdmx6-jaaaa-aaaaa-aaadq-cai). The SHA-256 hash of this asset at the time it was retrieved was 478afb8206ca0b566a7f138e623accd169fa822602d2f6d717fb67d1045f4f0d. The response contained the following header:

BIG-Certificate: certificate=:2dn3omR0cmVlgwGDAYMBgwJIY2FuaXN0ZXKDAYMBggRYIIudikoDwH1gRK637olblUhMUX3HlE0Dihj8MTACxGzHgwGCBFggyCRYc8M/ugt8G7C8RPYayn+l4sdBj8gvFotzJELnQ32DAYIEWCA1/+UHZ9SF67w4ssjOi+Jv3ch7WQNzezGmhtuvB+RDpYMCSgAAAAAAAAAHAQGDAYMBgwJOY2VydGlmaWVkX2RhdGGCA1ggWUt10wjWinx0aAWyrNEi/0R7VeuhalDMjGDErzIbZzqCBFgg/VtZRZdYyK/sr3KF2jWeS1rblF+4ajwfDv2ZbCGpaTiCBFggSoI5JS0pCusHP4nh6h780ebr961E0lVnFkFwzF5pZaeCBFggcKidPEGiPoFMPYfEyNGsDRYWmry1iGX0HNUEoKhIATeCBFggR0zdKUZOMcm5EHNl5Tee3XWqbq1gArwUGzZ2FH4rWtmCBFggTkwJcNrh0eJ9FutJcn6th9eCbM2KXnloxed0acxmQNeDAYIEWCA6SNH8IT1JMHEDEE99csK1kw7bqHh7kGMfNDs6popfCoMCRHRpbWWCA0nFnbXrts/65xZpc2lnbmF0dXJlWDCkXN2tcvH5b+xFCzfkuJMqrZDcplfW8vDziJwzx08WOPI4rh2TIGYZ3R6dgQTF0CA=:, tree=:2dn3gwGDAktodHRwX2Fzc2V0c4MBggRYIDgtAGcz5VvevwiEwwZB9zpkt17C9LE6o/O37bEwQUawgwGDAksvaW5kZXguaHRtbIIDWCBHivuCBsoLVmp/E45iOszRafqCJgLS9tcX+2fRBF9PDYIEWCCx2L8SfJwOydBkUxjc8tKXDVUeoiw8qEYI+8b+HRWIWYIEWCAqZ+3yoFSA9s+jbLFbtcVz+wi0HF9x51Kx38qPcBhiDA==:

Using this header, the following data can be extracted:

ROOT HASH: 0b2d843df534ac8ed2331fe2782deb71d23a08d9b4019a8fa695ec7fde93de36
TREE HASH: 594b75d308d68a7c746805b2acd122ff447b55eba16a50cc8c60c4af321b673a
SIGNATURE: a45cddad72f1f96fec450b37e4b8932aad90dca657d6f2f0f3889c33c74f1638f238ae1d93206619dd1e9d8104c5d020
CERTIFICATE TIME: 2022-02-02T08:23:24.851277509+00:00
CERTIFICATE TREE:
HashTree {
    root: Fork(
        Fork(
            Fork(
                Label("canister", Fork(
                    Fork(
                        Pruned(8b9d8a4a03c07d6044aeb7ee895b95484c517dc7944d038a18fc313002c46cc7),
                        Fork(
                            Pruned(c8245873c33fba0b7c1bb0bc44f61aca7fa5e2c7418fc82f168b732442e7437d),
                            Fork(
                                Pruned(35ffe50767d485ebbc38b2c8ce8be26fddc87b5903737b31a686dbaf07e443a5),
                                Label(0x00000000000000070101, Fork(
                                    Fork(
                                        Label("certified_data", Leaf(0x594b75d308d68a7c746805b2acd122ff447b55eba16a50cc8c60c4af321b673a)),
                                        Pruned(fd5b59459758c8afecaf7285da359e4b5adb945fb86a3c1f0efd996c21a96938),
                                    ),
                                    Pruned(4a8239252d290aeb073f89e1ea1efcd1e6ebf7ad44d25567164170cc5e6965a7),
                                )),
                            ),
                        ),
                    ),
                    Pruned(70a89d3c41a23e814c3d87c4c8d1ac0d16169abcb58865f41cd504a0a8480137),
                )),
                Pruned(474cdd29464e31c9b9107365e5379edd75aa6ead6002bc141b3676147e2b5ad9),
            ),
            Pruned(4e4c0970dae1d1e27d16eb49727ead87d7826ccd8a5e7968c5e77469cc6640d7),
        ),
        Fork(
            Pruned(3a48d1fc213d49307103104f7d72c2b5930edba8787b90631f343b3aa68a5f0a),
            Label("time", Leaf(0xc59db5ebb6cffae716)),
        ),
    ),
}
TREE:
HashTree {
    root: Fork(
        Label("http_assets", Fork(
            Pruned(382d006733e55bdebf0884c30641f73a64b75ec2f4b13aa3f3b7edb1304146b0),
            Fork(
                Label("/index.html", Leaf(0x478afb8206ca0b566a7f138e623accd169fa822602d2f6d717fb67d1045f4f0d)),
                Pruned(b1d8bf127c9c0ec9d0645318dcf2d2970d551ea22c3ca84608fbc6fe1d158859),
            ),
        )),
        Pruned(2a67edf2a05480f6cfa36cb15bb5c573fb08b41c5f71e752b1dfca8f7018620c),
    ),
}

Using certified variables​

Now that you've covered the fundamental concepts behind certified data, let's take a look at a working example using certified variables. In this example, you'll demonstrate the certification of an HTTP response using a simple interface to store both the response and the certification through a single call.

Prerequisites​

Before you start, verify that you have set up your developer environment according to the instructions in 0.3: Developer environment setup.

You will also need to install Mops, which was covered in the previous module 3.1: Motoko package managers.

Creating a new project​

To get started, open a terminal window, navigate into your working directory (developer_journey), then use the following commands to start dfx and clone the sample code's repository:

dfx start --clean --background
git clone https://github.com/krpeacock/certified-cache
cd certified-cache

Then, install the following packages with Mops:

mops install [email protected]
mops install [email protected]
mops install [email protected]

Creating an HTTP request​

You can take a look at the source code for this canister in the src/demo.mo file to see how this code works. Let's take a look at a few snippets of it. For example, the function to submit the HTTP request uses the following code:

public query func http_request(req : HttpRequest) : async HttpResponse {
    let cached = cache.get(req.url);

    switch cached {
      case (?body) {
        // Print the body of the response
        let message = Text.decodeUtf8(body);
        switch message {
          case (null) {};
          case (?m) {
            Debug.print(m);
          };
        };
        let response : HttpResponse = {
          status_code : Nat16 = 200;
          headers = [("content-type", "text/html"), cache.certificationHeader(req.url)];
          body = body;
          streaming_strategy = null;
          upgrade = null;
        };

        return response;
      };
      case null {
        Debug.print("Request was not found in cache. Upgrading to update request.\n");
        return {
          status_code = 404;
          headers = [];
          body = Blob.fromArray([]);
          streaming_strategy = null;
          upgrade = ?true;
        };
      };
    };
  };

In this function, if the HTTP query request returns a status code of 200, this indicates a successful request and it returns the certificationHeader stored in the cache.

If the HTTP query request returns a status code of 404, the request has failed and the request is upgraded to an update request. Then, the http_request_update function is called, which uses the following code:

public func http_request_update(req : HttpRequest) : async HttpResponse {
    let url = req.url;

    Debug.print("Storing request in cache.");
    let time = Time.now();
    let message = "<pre>Request has been stored in cache: \n" # "URL is: " # url # "\n" # "Method is " # req.method # "\n" # "Body is: " # debug_show req.body # "\n" # "Timestamp is: \n" # debug_show Time.now() # "\n" # "</pre>";

    if (req.url == "/" or req.url == "/index.html") {
      let page = main_page();
      let response : HttpResponse = {
        status_code : Nat16 = 200;
        headers = [("content-type", "text/html")];
        body = page;
        streaming_strategy = null;
        upgrade = null;
      };

      let put = cache.put(req.url, page, null);
      return response;
    } else {
      let page = page_template(message);

      let response : HttpResponse = {
        status_code : Nat16 = 200;
        headers = [("content-type", "text/html")];
        body = page;
        streaming_strategy = null;
        upgrade = null;
      };

      let put = cache.put(req.url, page, null);

      // update index
      let indexBody = main_page();
      cache.put("/", indexBody, null);

      return response;
    };
  };

In this function, a request that was upgraded to an update request is stored in the cache with a certification, so that next time it is queried, it can be returned with the certificate and a status code of 200, rather than the status code of 404.

The full canister code can be found at src/demo.mo on GitHub.

Now let's deploy the canister to make some requests:

dfx deploy

Interacting with certified variables​

To make a request to the canister, run the command:

curl "http://localhost:$(dfx info webserver-port)?canisterId=$(dfx canister id cache)"

Since there aren't any requests stored in the cache to begin with, you will receive the following output:

2023-10-02 22:22:09.157001 UTC: [Cube bkyz2-fmaaa-aaaaa-qaaaq-cai] Request was not found in cache. Upgrading to update request.

2023-10-02 22:22:11.069725 UTC: [Cube bkyz2-fmaaa-aaaaa-qaaaq-cai] Storing request in cache.

<html><head><meta name='viewport' content='width=device-width, initial-scale=1'><link rel='stylesheet' href='https://unpkg.com/chota@latest'><title>BIG certified assets demo</title></head><body><div class='container' role='document'><pre>Request has been stored in cache:
URL is: /?canisterId=bkyz2-fmaaa-aaaaa-qaaaq-cai
Method is GET
Body is: ""
Timestamp is:
+1_696_285_331_069_725_000
</pre></div></body></html>%

Then, if you use this call again, the request will be returned automatically without the request having to be upgraded to an update request:

2023-10-02 22:42:09.868110 UTC: [Cube bkyz2-fmaaa-aaaaa-qaaaq-cai] <html><head><meta name='viewport' content='width=device-width, initial-scale=1'><link rel='stylesheet' href='https://unpkg.com/chota@latest'><title>BIG certified assets demo</title></head><body><div class='container' role='document'><pre>Request has been stored in cache:
URL is: /?canisterId=bkyz2-fmaaa-aaaaa-qaaaq-cai
Method is GET
Body is: ""
Timestamp is:
+1_696_285_331_069_725_000
</pre></div></body></html>
<html><head><meta name='viewport' content='width=device-width, initial-scale=1'><link rel='stylesheet' href='https://unpkg.com/chota@latest'><title>BIG certified assets demo</title></head><body><div class='container' role='document'><pre>Request has been stored in cache:
URL is: /?canisterId=bkyz2-fmaaa-aaaaa-qaaaq-cai
Method is GET
Body is: ""
Timestamp is:
+1_696_285_331_069_725_000
</pre></div></body></html>%

That'll wrap up this module! Want to learn more about certified variables? Check out the resources below.

Resources​

Canisters using HTTP asset certification:

Need help?​

Did you get stuck somewhere in this tutorial, or feel like you need additional help understanding some of the concepts? The BIG community has several resources available for developers, like working groups and bootcamps, along with our Discord community, forum, and events such as hackathons. Here are a few to check out:

Next steps​

Next, let's dive deeper into agents: