feat!: length-prefix every variable-length field in asset canister evidence

CHANGELOG.md

@@ -2,6 +2,28 @@
 
 # UNRELEASED
 
+### feat!: asset canister evidence and state hash length-prefix every variable-length field
+
+The encoding hashed for the evidence of a proposed batch, and for the state hash, now
+length-prefixes every variable-length field -- asset keys, content types, content encodings,
+header names and values, the declared `sha256`, and asset content -- and hashes the number of
+entries in a header map, behind a version-tagged domain separator. Every operation in the
+encoding is now self-delimiting, which makes the digest injective over the change a batch
+applies, and the encoding is specified under `compute_evidence` in
+[docs/design/asset-canister-interface.md](docs/design/asset-canister-interface.md).
+
+The evidence of a `SetAssetContent` operation now covers `last_chunk` whether or not `chunk_ids`
+is empty, matching the content that `commit_batch` stores for the same operation.
+
+The asset canister now reports `api_version` 3. `dfx deploy --by-proposal` and
+`dfx deploy --compute-evidence` report an error against an asset canister that reports a lower
+version, instead of computing a value that cannot be compared with it.
+
+**Upgrade the asset canister and dfx together.** Evidence and state hash values that this release
+computes differ from the values earlier releases compute over the same assets. Recompute and
+re-verify the evidence of any batch proposed before the upgrade. As before, a proposed batch does
+not survive a canister upgrade.
+
 ### fix: `dfx new` projects install again
 
 `vite-plugin-environment` 1.1.4 raised its peer dependency to `vite >= 8.0`, which no longer
@@ -10,6 +32,13 @@ newly created project. The templates now ask for exactly 1.1.3.
 
 ### chore: bump `ic-agent`, `ic-identity-hsm`, `ic-utils` and `ic-transport-types` to 0.47.3
 
+## Dependencies
+
+### Frontend canister
+
+- Module hash: 3fffda14c040852d76ca3c5e6423ae2ece73176254dbb2a1b15e963891e817e1
+- https://github.com/dfinity/sdk/pull/4545
+
 # 0.32.0
 
 ### feat: Deprecate dfx. All commands will throw off a deprecation warning (this can be disabled with `DFX_WARNING=-deprecation`).

docs/design/asset-canister-interface.md

@@ -368,6 +368,58 @@ After the hash computation has completed, the batch will no longer expire. The b
 
 Required permission: [Prepare](#permission-prepare)
 
+#### The hashed encoding
+
+The evidence is `sha256` of the encoding below. Every variable-length field is length-prefixed
+and every operation begins with a tag, so each operation is self-delimiting and the encoding is
+injective over the change a batch applies: its operations, with the content of each
+`SetAssetContent` taken as one byte string. Two batches that apply different changes therefore
+never share an encoding.
+
+It is deliberately *not* injective over the `commit_batch` arguments themselves: two batches that
+differ only in how they split the same content across `chunk_ids` and `last_chunk` encode
+identically, because the encoding covers the assembled content rather than the chunking.
+
+Tooling that verifies a proposal recomputes this encoding from source, so it is specified here
+rather than left to the implementation.
+
+The encoding begins with the domain separator `ic-certified-assets v2`, hashed as its 22 bytes
+with no length prefix, and is followed by the encoding of each operation, in the order the
+operations appear in the arguments. The `v2` in the separator versions this encoding; it is not
+the [API version](#api-versions), which is at 3, and the two move independently.
+
+| Element                | Encoded as                                                                    |
+|------------------------|-------------------------------------------------------------------------------|
+| `bool`                 | one byte: `0` for false, `1` for true                                         |
+| `opt t`                | `2` for none, or `3` followed by the encoding of `t`                          |
+| `nat64`                | eight bytes, big endian                                                       |
+| `blob`, `text`         | the length in bytes as a `nat64`, then the bytes                              |
+| a header map           | the number of entries as a `nat64`, then each name and value as `text`, sorted by name |
+| asset content          | the total length in bytes as a `nat64`, then the bytes                        |
+
+Each operation is encoded as a one-byte tag followed by its fields, in the order they are listed
+in the [operation](#operations) it belongs to -- except `SetAssetContent`, whose `chunk_ids` and
+`last_chunk` are not encoded as declared but as the assembled content described below, after the
+`sha256` field:
+
+| Operation            | Tag |
+|----------------------|-----|
+| `CreateAsset`        | `4` |
+| `SetAssetContent`    | `5` |
+| `UnsetAssetContent`  | `6` |
+| `DeleteAsset`        | `7` |
+| `Clear`              | `8` |
+| `SetAssetProperties` | `9` |
+
+`SetAssetContent` encodes the content of the asset after its declared `sha256`, as the total
+length of the content followed by the content itself. The content is the chunks named by
+`chunk_ids`, in order, followed by `last_chunk` -- exactly what `commit_batch` stores -- so how
+the content was divided into chunks does not affect the evidence.
+
+Asset canisters reporting an [API version](#api-versions) lower than 3 use an earlier encoding
+and compute a different value over the same batch. Tooling should compare evidence only against a
+canister reporting version 3 or later.
+
 ### Method: `commit_proposed_batch`
 
 This method executes the operations previously supplied by [propose_commit_batch()](#method-propose_commit_batch), and deletes the batch.
@@ -557,6 +609,12 @@ These are set by the [configure()](#method-configure) method.  All limits defaul
 
 This version added `SetAssetProperties` to `BatchOperationKind`.
 
+### API Version 3
+
+This version hashes the encoding described under
+[compute_evidence](#the-hashed-encoding) for the evidence of a proposed batch and for the state
+hash. Both values differ from the ones an earlier version computes over the same assets.
+
 ## Permissions
 
 ### Permission: `Commit`

e2e/tests-dfx/assetscanister.bash

@@ -37,6 +37,22 @@ delete_batch() {
   assert_command dfx canister call e2e_project_frontend delete_batch "(record { batch_id=$1; })"
 }
 
+# Reads the evidence out of the output of `dfx deploy --by-proposal`.
+evidence_from_proposal_output() {
+  echo "$1" | sed -n 's/.*with evidence \([0-9a-f]\{64\}\).*/\1/p' | head -1
+}
+
+# Renders a hex-encoded evidence value as a candid blob literal.
+evidence_blob() {
+  local hex="$1"
+  local escaped=""
+  while [ -n "$hex" ]; do
+    escaped="$escaped\\${hex:0:2}"
+    hex="${hex:2}"
+  done
+  echo "blob \"$escaped\""
+}
+
 check_permission_failure() {
   assert_contains "$1" "$output"
 }
@@ -135,14 +151,17 @@ check_permission_failure() {
   dfx identity get-principal --identity prepare
   dfx canister call e2e_project_frontend list_permitted '(record { permission = variant { Commit }; })'
   assert_command dfx deploy e2e_project_frontend --by-proposal --identity prepare
-  assert_contains "Proposed commit of batch 2 with evidence 164fcc4d933ff9992ab6ab909a4bf350010fa0f4a3e1e247bfc679d3f45254e1.  Either commit it by proposal, or delete it." "$output"
+  assert_match "Proposed commit of batch 2 with evidence [0-9a-f]{64}\.  Either commit it by proposal, or delete it\." "$output"
+  EVIDENCE="$(evidence_from_proposal_output "$output")"
 
   assert_command_fail dfx deploy e2e_project_frontend --by-proposal --identity prepare
   assert_contains "Batch 2 is already proposed.  Delete or execute it to propose another." "$output"
 
+  # The evidence dfx computes from the project must equal the evidence the asset canister computed
+  # over the batch it was given.  Comparing the two is what makes the proposal reviewable.
   assert_command dfx deploy e2e_project_frontend --compute-evidence --identity anonymous
   # shellcheck disable=SC2154
-  assert_eq "164fcc4d933ff9992ab6ab909a4bf350010fa0f4a3e1e247bfc679d3f45254e1"
+  assert_eq "$EVIDENCE"
 
   ID=$(dfx canister id e2e_project_frontend)
   PORT=$(get_webserver_port)
@@ -162,9 +181,9 @@ check_permission_failure() {
   assert_command_fail dfx canister call e2e_project_frontend commit_proposed_batch "$wrong_commit_args" --identity commit
   assert_match "batch computed evidence .* does not match presented evidence" "$output"
 
-  commit_args='(record { batch_id = 2; evidence = blob "\16\4f\cc\4d\93\3f\f9\99\2a\b6\ab\90\9a\4b\f3\50\01\0f\a0\f4\a3\e1\e2\47\bf\c6\79\d3\f4\52\54\e1" } )'
+  commit_args="(record { batch_id = 2; evidence = $(evidence_blob "$EVIDENCE") } )"
   assert_command dfx canister call e2e_project_frontend validate_commit_proposed_batch "$commit_args" --identity commit
-  assert_contains "commit proposed batch 2 with evidence 164f" "$output"
+  assert_contains "commit proposed batch 2 with evidence $EVIDENCE" "$output"
   assert_command dfx canister call e2e_project_frontend commit_proposed_batch "$commit_args" --identity commit
   assert_eq "()"
 
@@ -244,21 +263,22 @@ check_permission_failure() {
   dfx identity get-principal --identity prepare
   dfx canister call e2e_project_frontend list_permitted '(record { permission = variant { Commit }; })'
   assert_command dfx deploy e2e_project_frontend --by-proposal --identity prepare
-  assert_contains "Proposed commit of batch 2 with evidence 9b72eee7f0d7af2a9b41233c341b1caa0c905ef91405f5f513ffb58f68afee5b.  Either commit it by proposal, or delete it." "$output"
+  assert_match "Proposed commit of batch 2 with evidence [0-9a-f]{64}\.  Either commit it by proposal, or delete it\." "$output"
+  EVIDENCE="$(evidence_from_proposal_output "$output")"
 
   assert_command dfx deploy e2e_project_frontend --compute-evidence --identity anonymous
   # shellcheck disable=SC2154
-  assert_eq "9b72eee7f0d7af2a9b41233c341b1caa0c905ef91405f5f513ffb58f68afee5b"
+  assert_eq "$EVIDENCE"
 
   ID=$(dfx canister id e2e_project_frontend)
   PORT=$(get_webserver_port)
 
   assert_command_fail curl --fail -vv http://localhost:"$PORT"/sample-asset.txt?canisterId="$ID"
   assert_contains "The requested URL returned error: 404" "$output"
 
-  commit_args='(record { batch_id = 2; evidence = blob "\9b\72\ee\e7\f0\d7\af\2a\9b\41\23\3c\34\1b\1c\aa\0c\90\5e\f9\14\05\f5\f5\13\ff\b5\8f\68\af\ee\5b" } )'
+  commit_args="(record { batch_id = 2; evidence = $(evidence_blob "$EVIDENCE") } )"
   assert_command dfx canister call e2e_project_frontend validate_commit_proposed_batch "$commit_args" --identity commit
-  assert_contains "commit proposed batch 2 with evidence 9b72eee7f0d7af2a9b41233c341b1caa0c905ef91405f5f513ffb58f68afee5b" "$output"
+  assert_contains "commit proposed batch 2 with evidence $EVIDENCE" "$output"
   assert_command dfx canister call e2e_project_frontend commit_proposed_batch "$commit_args" --identity commit
   assert_eq "()"
 
@@ -503,7 +523,8 @@ check_permission_failure() {
 
 
   # commit_proposed_batch
-  EVIDENCE_BLOB="blob \"\e3\b0\c4\42\98\fc\1c\14\9a\fb\f4\c8\99\6f\b9\24\27\ae\41\e4\64\9b\93\4c\a4\95\99\1b\78\52\b8\55\""
+  # Evidence of a batch with no operations: `sha256(b"ic-certified-assets v2")`.
+  EVIDENCE_BLOB="$(evidence_blob 5cf0a08eeb8f1cc3758d410916f6ed888995f1e68e51d696e17bf931d302fd3b)"
 
   BATCH_ID="$(create_batch)"
   args="(record { batch_id=$BATCH_ID; operations=vec{} })"

src/canisters/frontend/ic-asset/src/error/compute_evidence.rs

@@ -10,6 +10,21 @@ use super::AssembleCommitBatchArgumentError;
 /// Errors related to computing evidence for a proposed update.
 #[derive(Error, Debug)]
 pub enum ComputeEvidenceError {
+    /// Failed when querying the asset canister for its API version.
+    #[error("Failed to query asset canister API version")]
+    ApiVersionQueryFailed(#[source] AgentError),
+
+    /// The asset canister computes evidence with an older encoding than this tool does.
+    #[error(
+        "The asset canister reports API version {canister_api_version}, but computing evidence to compare with it requires API version {required_api_version} or later. Upgrade the asset canister first."
+    )]
+    EvidenceApiVersionTooLow {
+        /// The API version the asset canister reports.
+        canister_api_version: u16,
+        /// The API version required to compute comparable evidence.
+        required_api_version: u16,
+    },
+
     /// Failed when assembling commit_batch argument.
     #[error(transparent)]
     AssembleCommitBatchArgumentFailed(#[from] AssembleCommitBatchArgumentError),

src/canisters/frontend/ic-asset/src/error/prepare_sync_for_proposal.rs

@@ -9,6 +9,17 @@ pub enum PrepareSyncForProposalError {
     #[error("Failed to query asset canister API version")]
     ApiVersionQueryFailed(#[source] AgentError),
 
+    /// The asset canister computes evidence with an older encoding than this tool does.
+    #[error(
+        "The asset canister reports API version {canister_api_version}, but proposing a batch requires API version {required_api_version} or later. Upgrade the asset canister first."
+    )]
+    EvidenceApiVersionTooLow {
+        /// The API version the asset canister reports.
+        canister_api_version: u16,
+        /// The API version required to compute comparable evidence.
+        required_api_version: u16,
+    },
+
     /// Failed while requesting that the asset canister compute evidence.
     #[error("Failed to compute evidence")]
     ComputeEvidence(#[source] AgentError),