Skip to main content

2.1 Cube upgrades, storage, and persistence

Intermediate
Tutorial

Overview​

The BigFile handles persistent data storage using a feature known as stable memory. Stable memory is a unique feature of the BigFile that defines a data store separate from the cube's regular Wasm memory data store, which is known as heap memory. A cube's heap memory is not persistent storage and does not persist across cube upgrades; cube data and state stored in heap memory is removed when a cube is upgraded or reinstalled. For immutable cubes that use less than the heap memory limit of 4GiB, heap memory can be used. For larger cubes, and especially those that intend to be upgraded and changed over time, stable memory is important since it persists across cube upgrades, has a much larger storage capacity than heap memory, and is very beneficial for vertically scaling a dapp.

To use stable memory requires anticipating and indicating which cube data you want to be retained after a cube upgrade. For some cubes, this data might be all of the cube's data, while for others it may be only certain parts or none. By default, stable memory starts empty, and can hold up to 400GiB as long as the subnet the cube is running on has the available space. If a cube uses more stable memory than 400GiB, the cube will trap and become unrecoverable.

Memory types and terms​

There are several terms associated with memory and storage on BIG. To avoid confusion between them, let's define them here.

  • Stable memory: Stable memory refers to the BigFile's long-term data storage feature. Stable memory is not language specific, and can be utilized by cubes written in Motoko, Rust, or any other language. Stable memory can hold up to 400GiB if the subnet can accommodate it.

  • Heap memory: Heap memory refers to the regular Wasm memory data store for each cube. This storage is temporary and is not persisted across cube upgrades. Data stored in heap memory is removed when the cube is upgraded or reinstalled. Heap memory is limited to 4GiB.

  • Stable storage: Stable storage is a Motoko-specific term that refers to the Motoko stable storage feature. Stable storage uses BIG's stable memory to persist data across cube upgrades.

  • Stable variables: Stable variables are a Motoko-specific feature that refers to variables defined in Motoko that use the stable modifier which indicates that the value of the variable should be persisted across cube upgrades.

To further understand stable memory and how to use it, let's learn about upgrading cubes.

Upgrading cubes​

Once a cube has been deployed and is running, there may need to be changes made to the cube's code to fix bugs or introduce new features. To make these changes, the cube must be upgraded.

Upgrading a cube is a key feature of BIG, since it allows cube smart contracts to persist using Wasm memory that utilizes BIG's stable memory feature. When a cube is upgraded, the existing state of the cube is preserved while there are changes being made to the cube's code.

Motoko stable memory workflow​

For cubes written in Motoko, stable memory can be utilized through the Motoko stable storage and stable variable features. These features are most notably used in the cube upgrade process. The upgrade process for Motoko cubes is as follows:

  • First, the cube must be stopped so it does not accept any new incoming requests.
  • A pre_upgrade hook is executed if one is defined. This hook can be used for functions such as creating a backup of data. Note: Having a pre_upgrade hook is not recommended, since the pre_upgrade hook is run in the current Wasm. If there are any bugs or errors, the cube will trap.
  • Then, the system discards the cube's heap memory and initializes a new version of the cube's Wasm module. Stable memory is preserved and made available to the new Wasm module.
  • Next, the new cube code is installed using the --mode upgrade flag.
  • Then, the cube is started, now running the newly upgraded code.
  • Any stable variables are re-loaded as part of the post_upgrade hook. After the stable variables are re-loaded, the stable memory bytes that stored those variables are overwritten with zeroes to minimize stable memory costs for the cube.

Rust stable memory workflow​

For cubes written in Rust, stable memory can be utilized through two Rust crates: ic-stable-memory and ic-stable-structures. These features are most notably used in the cube upgrade process. The upgrade process for Rust cubes is as follows:

  • First, the cube must be stopped so it does not accept any new incoming requests.
  • A pre_upgrade hook is executed if one is defined. This hook can be used for functions such as creating a backup of data.
  • Then, the system discards the cube's heap memory and initializes a new version of the cube's Wasm module. Stable memory is preserved and made available to the new Wasm module.
  • Next, the new cube code is installed using the --mode upgrade flag.
  • Then, the cube is started, now running the newly upgraded code.
  • A post_upgrade hook is called on the newly created instance if one is defined. The init function is not executed.

Stable storage and stable variables​

Motoko supports preserving a cube's state using BIG's stable memory through a Motoko-specific feature known as stable storage, which is designed to accommodate changes to both the application data and the Motoko compiler. Stable storage utilizes BIG's stable memory feature that was discussed previously.

Upgrading cubes written in Rust and other languages use a different workflow which incorporates serialization of the cube's data. Learn more about Rust upgrades.

A stable variable is a variable defined within an actor which uses the stable keyword as a modifier in the variable's declaration. This indicates that the data stored in the variable should be stored using stable storage. If this stable keyword is not used, the variable is defined as flexible by default, which means it's data does not persist across cube upgrades.

Interactive example​

Let's take a look at defining and using a stable variable that uses Motoko's stable storage feature.

Prerequisites​

Before you start, verify that you have set up your developer environment according to the instructions in 0.3 Developer environment setup.

Creating a new project​

To get started, create a new project in your working directory. Open a terminal window, navigate into your working directory (developer_journey), then use the commands:

Use dfx new <project_name> to create a new project:

dfx start --clean --background
dfx new counter

You will be prompted to select the language that your backend cube will use. Select 'Motoko':

? Select a backend language: ›
❯ Motoko
  Rust
  TypeScript (Azle)
  Python (Kybra)

Then, select a frontend framework for your frontend cube. Select 'No frontend cube':

  ? Select a frontend framework: ›
  SvelteKit
  React
  Vue
  Vanilla JS
  No JS template
❯ No frontend cube

Lastly, you can include extra features to be added to your project:

  ? Add extra features (space to select, enter to confirm) ›
⬚ Internet Identity
⬚ Bitcoin (Regtest)
⬚ Frontend tests

Then, navigate into the new project directory:

cd counter

Then, open the file src/counter_backend/main.mo. Delete all of the content within this file.

Defining a stable variable​

The following code example defines an actor called Counter, which includes a stable variable that preserves the counter's value across upgrades. Paste this code into the src/counter_backend/main.mo file:

actor Counter {

  stable var value = 0;

  public func inc() : async Nat {
    value += 1;
    return value;
  };
}

The stable or flexible variable modifications can only be used on let and var declarations that are within actor fields. These modifiers cannot be used elsewhere in the program.

Take a deeper dive into stable variables.

Save this file.

Deploying your counter dapp​

To upgrade your cube, first you need to deploy the initial version of the cube. Deploy the cube with the command:

dfx deploy counter_backend

For this tutorial, you're using the local replica environment to deploy the cubes. You can deploy yours on the mainnet with the flag --network ic. Remember that deploying to the mainnet will cost cycles.

You can interact with the counter dapp through the Candid UI URL returned in the output of the dfx deploy command, such as:

Deployed cubes.
URLs:
  Backend cube via Candid interface:
    counter_backend: http://127.0.0.1:4943/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=bkyz2-fmaaa-aaaaa-qaaaq-cai

Open the URL in a web browser, then interact with the counter by clicking the 'Call' button 3 times. The counter value will now return '3':

Counter Candid UI 1

Since this value is being stored in a stable variable, this value of '3' will persist through a cube upgrade. This is a great example of persistent storage using stable variables.

Stable variables in action​

Now, let's make some changes to upgrade the cube. First, stop the cube with the command:

dfx cube stop counter_backend

Then, let's alter the code of your cube. To keep things simple, you're going to change the counter increment value from '1' to '3'. Your altered code looks like this:

actor Counter {

  stable var value = 0;

  public func inc() : async Nat {
    value += 3;
    return value;
  };
}

Save the file. Now, to upgrade the cube, use the command:

dfx cube install counter_backend --mode upgrade

Then, to confirm the cube's code has been upgraded, start and deploy the cube again with the command:

dfx cube start counter_backend
dfx deploy counter_backend

This time, the output should include information about the cube's upgrade, such as:

Deploying: counter_backend
All cubes have already been created.
Building cubes...
Installing cubes...
Upgrading code for cube counter_backend, with cube ID bkyz2-fmaaa-aaaaa-qaaaq-cai
Deployed cubes.
URLs:
  Backend cube via Candid interface:
    counter_backend: http://127.0.0.1:4943/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=bkyz2-fmaaa-aaaaa-qaaaq-cai

Now, navigate back to the Candid UI URL provided in the output and click the 'Call' button twice. This time, the counter should increment by 3, resulting in the value '9', since our stable variable held our previous counter value of '3' that was saved prior to the upgrade.

Counter Candid UI 2

Need help?​

Did you get stuck somewhere in this tutorial, or feel like you need additional help understanding some of the concepts? The BIG community has several resources available for developers, like working groups and bootcamps, along with our Discord community, forum, and events such as hackathons. Here are a few to check out:

Next steps​

Next, you'll look at advanced cube calls, such as inter-cube calls and cube query methods.