Decide whether @polkadot/types needs patching for DilithiumSignatureScheme, or whether registry overrides suffice #1

Closed
opened 2026-09-10 10:13:57 +00:00 by grenade · 3 comments
Owner

Investigation first, code only if the answer says so. Context in quantus/extension#1.

The question

packages/types/src/interfaces/extrinsics/definitions.ts hardcodes:

ExtrinsicSignature: 'MultiSignature',

which is 64/65 bytes. Quantus needs DilithiumSignatureScheme — a two-variant enum carrying 5261 or 7219 bytes. But polkadot-js resolves ExtrinsicSignature through the registry, and chains have historically overridden it from userland. So this may need no fork change at all:

new ApiPromise({
  provider,
  types: {
    ExtrinsicSignature: 'DilithiumSignatureScheme',
    DilithiumSignatureScheme: {
      _enum: { Dilithium87: '[u8;7219]', Dilithium65: '[u8;5261]' }
    }
  }
})

Test that before writing anything. If it works, this repo stays a plain mirror of upstream and the override moves into whatever dapp-side helper the extension ships — a much better outcome than carrying a patch.

What to actually check

  1. Does U8aFixed accept a 7219-byte length? Its bit-length handling is well outside normal use at this size; there may be an assertion or a practical limit.
  2. Does the extension's own path still work? packages/extension-base/src/background/handlers/Extension.ts builds a TypeRegistry per request and calls registry.register(types) from the stored MetadataDef, so the same override may be injectable there through metadata registration rather than code.
  3. ExtrinsicPayload.sign picks withType from registry.createTypeUnsafe('ExtrinsicSignature', []) instanceof Enum. MultiSignature is an Enum and so is DilithiumSignatureScheme, so withType stays true either way — meaning the signing side is already correct without any override, and only extrinsic assembly is at issue. Confirm this, because it decides whether the extension needs this repo at all.
  4. Round-trip: build a signed extrinsic with a real Quantus signature, encode it, and check the bytes against what the node accepts — preamble 0x84, MultiAddress::Id, variant byte, then sig ‖ pk as a fixed array with no compact length prefix.

That last point is where a decoder most plausibly goes wrong. qsafe.af's hand-rolled parser reads the signature as SCALE bytes with a compact length prefix, which is not what the runtime encodes; worth confirming which of the two is mistaken before trusting either as a reference.

The precedent worth reading first

quantus-apps hit this exact class of problem in polkadart and pinned a fork over it. Their note:

Upstream emits a compact integer for the MultiAddress::Index variant where the metadata declares the field zero-width. A codec that disagrees with the metadata about a field's width re-frames every byte after it, so a crafted payload can display one call and sign another — a clearsigning bypass on the cold wallet.

A codec/metadata width disagreement is reachable here too, and the consequence is that a user approves what they did not sign. Whatever conclusion this issue reaches, pin it with a test that encodes a full signed extrinsic and compares bytes — not just one that checks the type constructs.

Outcome

Record the answer here either way. If no patch is needed, say so and leave this repo as a mirror; that is a result, not a non-result.

Investigation first, code only if the answer says so. Context in quantus/extension#1. ## The question `packages/types/src/interfaces/extrinsics/definitions.ts` hardcodes: ```js ExtrinsicSignature: 'MultiSignature', ``` which is 64/65 bytes. Quantus needs `DilithiumSignatureScheme` — a two-variant enum carrying 5261 or 7219 bytes. But polkadot-js resolves `ExtrinsicSignature` *through the registry*, and chains have historically overridden it from userland. So this may need no fork change at all: ```js new ApiPromise({ provider, types: { ExtrinsicSignature: 'DilithiumSignatureScheme', DilithiumSignatureScheme: { _enum: { Dilithium87: '[u8;7219]', Dilithium65: '[u8;5261]' } } } }) ``` **Test that before writing anything.** If it works, this repo stays a plain mirror of upstream and the override moves into whatever dapp-side helper the extension ships — a much better outcome than carrying a patch. ## What to actually check 1. Does `U8aFixed` accept a 7219-byte length? Its bit-length handling is well outside normal use at this size; there may be an assertion or a practical limit. 2. Does the extension's own path still work? `packages/extension-base/src/background/handlers/Extension.ts` builds a `TypeRegistry` per request and calls `registry.register(types)` from the stored `MetadataDef`, so the same override may be injectable there through metadata registration rather than code. 3. `ExtrinsicPayload.sign` picks `withType` from `registry.createTypeUnsafe('ExtrinsicSignature', []) instanceof Enum`. `MultiSignature` is an `Enum` and so is `DilithiumSignatureScheme`, so `withType` stays `true` either way — meaning the **signing** side is already correct without any override, and only extrinsic *assembly* is at issue. Confirm this, because it decides whether the extension needs this repo at all. 4. Round-trip: build a signed extrinsic with a real Quantus signature, encode it, and check the bytes against what the node accepts — preamble `0x84`, `MultiAddress::Id`, variant byte, then `sig ‖ pk` as a fixed array with **no compact length prefix**. That last point is where a decoder most plausibly goes wrong. `qsafe.af`'s hand-rolled parser reads the signature as SCALE bytes *with* a compact length prefix, which is not what the runtime encodes; worth confirming which of the two is mistaken before trusting either as a reference. ## The precedent worth reading first `quantus-apps` hit this exact class of problem in polkadart and pinned a fork over it. Their note: > Upstream emits a compact integer for the `MultiAddress::Index` variant where the metadata declares the field zero-width. A codec that disagrees with the metadata about a field's width re-frames every byte after it, so a crafted payload can display one call and sign another — a clearsigning bypass on the cold wallet. A codec/metadata width disagreement is reachable here too, and the consequence is that a user approves what they did not sign. Whatever conclusion this issue reaches, pin it with a test that encodes a full signed extrinsic and compares bytes — not just one that checks the type constructs. ## Outcome Record the answer here either way. If no patch is needed, say so and leave this repo as a mirror; that is a result, not a non-result.
Author
Owner

Narrowing this: the workaround that looked most promising is ruled out.

signedTransaction does not avoid the problem

SignerPayloadJSON carries a withSignedTransaction flag, and a signer may return a fully assembled extrinsic instead of a bare signature. That looked like a way to sidestep ExtrinsicSignature entirely — let the extension concatenate the bytes itself, hand back the finished transaction, and never ask @polkadot/types to encode a 5261-byte signature.

It does not work. @polkadot/api's submittable/createClass.js accepts the field and then immediately re-decodes it:

if (result.signedTransaction && options.withSignedTransaction) {
  const ext = this.registry.createTypeUnsafe('Extrinsic', [result.signedTransaction]);
  
  if (!ext.isSigned) { throw new Error(); }
  if (!allowCallDataAlteration) { this.#validateSignedTransaction(payload, ext); }
  super.addSignature(address, result.signature, newSignerPayload.toPayload());
}

That createTypeUnsafe('Extrinsic', …) goes straight through MultiSignature, and addSignature on the line below does too. So the registry still has to know the real signature type — which is this issue. The re-decode is a deliberate anti-tampering check (it re-derives a SignerPayload from the decoded extrinsic and compares), so it is not something to disable.

Worth noting the extension's own signedTransaction path has the same dependency: LedgerSign.tsx builds it with extrinsic.addSignature(...), i.e. via @polkadot/types, not by concatenation.

Consequence

Anything on the @polkadot/api stack — including polkadot-js/apps — needs the answer to this issue. There is no signer-side trick that avoids it.

Contrast, and why it is a useful comparison

polkadot-api (papi) reaches the same goal without any of this. getSignerType reads the Address and Signature types from metadata rather than from a hardcoded definition, and createV4Tx assembles by concatenation with no length assertion anywhere, so a 5261-byte signature needs no accommodation at all. Its only Quantus blocker is a variant-name whitelist bolted on top of the metadata lookup — quantus/papi-console#1, a two-line pnpm patch.

That is the same doctrine blackbeard.observer settled on: decode the chain's data using the chain's own description of itself. Where this issue lands, prefer the solution that reads the type from metadata over one that registers another hardcoded name — the runtime already said what the type is, and transactionVersion has moved 2 → 3 → 6 across four Quantus upgrades, so anything pinned by name will rot.

Practical effect on priority

quantus/papi-console gives a working signing path without waiting for this issue, so this is no longer on the critical path for "can we sign a real extrinsic". It stays required for polkadot-js/apps and for any dapp on @polkadot/api — which is most of the existing ecosystem, so it is still worth doing, just not first.

Narrowing this: the workaround that looked most promising is ruled out. ## `signedTransaction` does not avoid the problem `SignerPayloadJSON` carries a `withSignedTransaction` flag, and a signer may return a fully assembled extrinsic instead of a bare signature. That looked like a way to sidestep `ExtrinsicSignature` entirely — let the extension concatenate the bytes itself, hand back the finished transaction, and never ask `@polkadot/types` to encode a 5261-byte signature. It does not work. `@polkadot/api`'s `submittable/createClass.js` accepts the field and then immediately **re-decodes it**: ```js if (result.signedTransaction && options.withSignedTransaction) { const ext = this.registry.createTypeUnsafe('Extrinsic', [result.signedTransaction]); … if (!ext.isSigned) { throw new Error(…); } if (!allowCallDataAlteration) { this.#validateSignedTransaction(payload, ext); } super.addSignature(address, result.signature, newSignerPayload.toPayload()); } ``` That `createTypeUnsafe('Extrinsic', …)` goes straight through `MultiSignature`, and `addSignature` on the line below does too. So the registry still has to know the real signature type — which is this issue. The re-decode is a deliberate anti-tampering check (it re-derives a `SignerPayload` from the decoded extrinsic and compares), so it is not something to disable. Worth noting the extension's own `signedTransaction` path has the same dependency: `LedgerSign.tsx` builds it with `extrinsic.addSignature(...)`, i.e. via `@polkadot/types`, not by concatenation. ## Consequence Anything on the `@polkadot/api` stack — including polkadot-js/apps — needs the answer to this issue. There is no signer-side trick that avoids it. ## Contrast, and why it is a useful comparison polkadot-api (papi) reaches the same goal without any of this. `getSignerType` reads the `Address` and `Signature` types **from metadata** rather than from a hardcoded definition, and `createV4Tx` assembles by concatenation with no length assertion anywhere, so a 5261-byte signature needs no accommodation at all. Its only Quantus blocker is a variant-*name* whitelist bolted on top of the metadata lookup — quantus/papi-console#1, a two-line pnpm patch. That is the same doctrine blackbeard.observer settled on: decode the chain's data using the chain's own description of itself. Where this issue lands, prefer the solution that reads the type from metadata over one that registers another hardcoded name — the runtime already said what the type is, and `transactionVersion` has moved 2 → 3 → 6 across four Quantus upgrades, so anything pinned by name will rot. ## Practical effect on priority quantus/papi-console gives a working signing path without waiting for this issue, so this is no longer on the critical path for "can we sign a real extrinsic". It stays required for polkadot-js/apps and for any dapp on `@polkadot/api` — which is most of the existing ecosystem, so it is still worth doing, just not first.
Author
Owner

Answer: neither. @polkadot/api's codec cannot be the encode/decode path at all.

Tested directly against Heisenberg (spec 148, genesis 0xa5aa9e5c…c8d3b4f) with a tier-1 harness that signs with the forked keyring and submits. The registry-override hypothesis in the title is falsified, and so is the fallback ("patch @polkadot/types"). Recording the evidence because the failures are ordered, and only the first two are the ones you would predict.

1. Registry overrides lose to metadata

types: { ExtrinsicSignature: 'DilithiumSignatureScheme',  }

User types are registered before setMetadata, so the metadata-derived lookup type wins. The override is dead code.

2. @polkadot/types refuses fixed arrays longer than 2048

createType(ExtrinsicV4):: Struct: failed on signature:
  PortableRegistry: 45 (qp_dilithium_crypto::types::DilithiumSignatureScheme):
  PortableRegistry: 47: {"array":{"len":7219,"type":3}}:
  Only support for [Type; <length>], where length <= 2048

Two hard-coded guards, both in this repo's upstream:

  • packages/types/src/metadata/PortableRegistry/PortableRegistry.ts#extractArray
  • packages/types-create/src/util/getTypeDef.ts_decodeFixedVec

ML-DSA-65 is [u8;5261] and ML-DSA-87 is [u8;7219], so every Quantus signature trips it. The cap is arbitrary rather than structural: [u8;N] resolves to U8aFixed.with(N * 8), a single Uint8Array with no per-element codec, so nothing gets slower. Raising it locally to 65536 in node_modules let the extrinsic encode and reach the node.

3. …and even then, polkadot-js cannot decode a Quantus block

VEC: Unable to decode on index 0
  Signed Extrinsics are currently only available for ExtrinsicV4
RPC-CORE: getBlock(hash?: BlockHash): SignedBlock:: … failed on extrinsics

Index 0 is the timestamp inherent. Its preamble byte is 0x05: the top two bits are a type tag (0b00 bare, 0b10 signed, 0b01 general) and the low six are the version, so that is bare, v5, while signed transactions in the same block are 0x84 — signed, v4 — and the metadata declares extrinsic version 4. Three different numbers, all correct. blackbeard.observer's runtime.rs documents this and has the regression test; polkadot-js reads the byte as a version and rejects the block.

So api.rpc.chain.getBlock throws on every block of this chain, at the first extrinsic, before any Quantus-specific type is reached.

4. The submission is still rejected, and that is the decisive part

With the cap raised, the extrinsic encodes (7220 bytes: variant 0x00 + 7219) and submits:

payload     118 bytes -> signing 118
context     QUANTUS_EXTRINSIC
signature   7220 bytes, variant 0x00
1010: Invalid Transaction: Transaction has a bad signature

Ruled out as causes:

  • Account derivation. The signer resolves to qzk1Nxai…2vSn7, which holds 627410384707496 (627.41 HEI) at nonce 2760 on Heisenberg. Our Poseidon2 matches the chain's hash_bytes, and Verify checks into_account() == signer first.
  • Context. primitives/dilithium-crypto/src/signing_context.rsEXTRINSIC = b"QUANTUS_EXTRINSIC", and pair.rs's test_extrinsic_signature_rejects_other_contexts pins it. We sign under exactly that.
  • Signature layout. scheme_macro.rs stores sig ‖ pk (bytes[..SIGNATURE_LEN] is the signature). Ours matches.
  • Payload length rule. 118 < 256, so no BLAKE2b — same as the runtime.

What is left is the signed payload's bytes, and here is the point: polkadot-js logs

REGISTRY: Unknown signed extensions ReversibleTransactionExtension,
          WormholeProofRecorderExtension found, treating them as no-effect

"Treating them as no-effect" is an assumption, not a reading. The runtime's TxExtension tuple (runtime/src/lib.rs) has twelve members, two of which polkadot-js has never heard of, and it guesses zero bytes for both their extra and their implicit. That guess happens to be right today — both are PhantomData with type Implicit = () — but the metadata declares those types, and a signer that guesses is wrong the moment a runtime upgrade gives either of them a field.

That is not a hypothetical here. Testnet encoding has already changed between runtimes on this chain, and transactionVersion has gone 2 → 3 → 6 across four upgrades — each one an extrinsic-format change that silently breaks a signer written against the previous one. A wallet that produces a valid signature over the wrong payload fails as BadProof, indistinguishable from a wrong key.

Decision

Do not fork @polkadot/api. Encode and decode against the runtime's own metadata instead, the way blackbeard.observer already does for blocks — state_getMetadata at a block hash makes the node run Metadata_metadata against that block's runtime, so the runtime WASM is the oracle and the signed extensions are read, in order, with their declared types, rather than assumed.

Filed as quantus/wasm#3 (@quantus/codec). This issue stays open only to track removing @polkadot/api's codec from the extension's path; the "patch or override" question is closed — the answer is neither.

## Answer: neither. `@polkadot/api`'s codec cannot be the encode/decode path at all. Tested directly against Heisenberg (spec 148, genesis `0xa5aa9e5c…c8d3b4f`) with a tier-1 harness that signs with the forked keyring and submits. The registry-override hypothesis in the title is falsified, and so is the fallback ("patch `@polkadot/types`"). Recording the evidence because the failures are ordered, and only the first two are the ones you would predict. ### 1. Registry overrides lose to metadata ```js types: { ExtrinsicSignature: 'DilithiumSignatureScheme', … } ``` User types are registered before `setMetadata`, so the metadata-derived lookup type wins. The override is dead code. ### 2. `@polkadot/types` refuses fixed arrays longer than 2048 ``` createType(ExtrinsicV4):: Struct: failed on signature: PortableRegistry: 45 (qp_dilithium_crypto::types::DilithiumSignatureScheme): PortableRegistry: 47: {"array":{"len":7219,"type":3}}: Only support for [Type; <length>], where length <= 2048 ``` Two hard-coded guards, both in this repo's upstream: - `packages/types/src/metadata/PortableRegistry/PortableRegistry.ts` → `#extractArray` - `packages/types-create/src/util/getTypeDef.ts` → `_decodeFixedVec` ML-DSA-65 is `[u8;5261]` and ML-DSA-87 is `[u8;7219]`, so every Quantus signature trips it. The cap is arbitrary rather than structural: `[u8;N]` resolves to `U8aFixed.with(N * 8)`, a single `Uint8Array` with no per-element codec, so nothing gets slower. Raising it locally to 65536 in `node_modules` let the extrinsic encode and reach the node. ### 3. …and even then, polkadot-js cannot decode a Quantus *block* ``` VEC: Unable to decode on index 0 Signed Extrinsics are currently only available for ExtrinsicV4 RPC-CORE: getBlock(hash?: BlockHash): SignedBlock:: … failed on extrinsics ``` Index 0 is the timestamp inherent. Its preamble byte is `0x05`: the top two bits are a **type tag** (`0b00` bare, `0b10` signed, `0b01` general) and the low six are the version, so that is *bare, v5*, while signed transactions in the same block are `0x84` — signed, v4 — and the metadata declares extrinsic version 4. Three different numbers, all correct. `blackbeard.observer`'s `runtime.rs` documents this and has the regression test; polkadot-js reads the byte as a version and rejects the block. So `api.rpc.chain.getBlock` throws on **every block of this chain**, at the first extrinsic, before any Quantus-specific type is reached. ### 4. The submission is still rejected, and *that* is the decisive part With the cap raised, the extrinsic encodes (7220 bytes: variant `0x00` + 7219) and submits: ``` payload 118 bytes -> signing 118 context QUANTUS_EXTRINSIC signature 7220 bytes, variant 0x00 1010: Invalid Transaction: Transaction has a bad signature ``` Ruled out as causes: - **Account derivation.** The signer resolves to `qzk1Nxai…2vSn7`, which holds 627410384707496 (627.41 HEI) at nonce 2760 on Heisenberg. Our Poseidon2 matches the chain's `hash_bytes`, and `Verify` checks `into_account() == signer` first. - **Context.** `primitives/dilithium-crypto/src/signing_context.rs` — `EXTRINSIC = b"QUANTUS_EXTRINSIC"`, and `pair.rs`'s `test_extrinsic_signature_rejects_other_contexts` pins it. We sign under exactly that. - **Signature layout.** `scheme_macro.rs` stores `sig ‖ pk` (`bytes[..SIGNATURE_LEN]` is the signature). Ours matches. - **Payload length rule.** 118 < 256, so no BLAKE2b — same as the runtime. What is left is the **signed payload's bytes**, and here is the point: polkadot-js logs ``` REGISTRY: Unknown signed extensions ReversibleTransactionExtension, WormholeProofRecorderExtension found, treating them as no-effect ``` "Treating them as no-effect" is an *assumption*, not a reading. The runtime's `TxExtension` tuple (`runtime/src/lib.rs`) has twelve members, two of which polkadot-js has never heard of, and it guesses zero bytes for both their `extra` and their implicit. That guess happens to be right today — both are `PhantomData` with `type Implicit = ()` — but the metadata *declares* those types, and a signer that guesses is wrong the moment a runtime upgrade gives either of them a field. That is not a hypothetical here. Testnet encoding has already changed between runtimes on this chain, and `transactionVersion` has gone 2 → 3 → 6 across four upgrades — each one an extrinsic-format change that silently breaks a signer written against the previous one. A wallet that produces a valid signature over the wrong payload fails as `BadProof`, indistinguishable from a wrong key. ### Decision Do not fork `@polkadot/api`. Encode and decode against the runtime's own metadata instead, the way `blackbeard.observer` already does for blocks — `state_getMetadata` at a block hash makes the node run `Metadata_metadata` against that block's runtime, so the runtime WASM is the oracle and the signed extensions are *read*, in order, with their declared types, rather than assumed. Filed as quantus/wasm#3 (`@quantus/codec`). This issue stays open only to track removing `@polkadot/api`'s codec from the extension's path; the "patch or override" question is closed — the answer is neither.
Author
Owner

Resolved, by removal

@quantus/codec (quantus/wasm#3) landed and quantus/extension#7 tier 1 now signs a balances.transfer_keep_alive that Heisenberg accepts and includes — block 1050475, nonce 2761 → 2762 — with @polkadot/api entirely out of the path. WsProvider remains only as a JSON-RPC transport; no type is decoded by polkadot-js.

One correction to the analysis above, for the record. Under §4 I listed the two unrecognised extensions as the remaining suspect for the BadProof. Having read the metadata with the new codec, the registry says they are innocent:

CheckNonZeroSender                       extra=false additional=false
CheckSpecVersion                         extra=false additional=true
CheckTxVersion                           extra=false additional=true
CheckGenesis                             extra=false additional=true
CheckMortality                           extra=true  additional=true
CheckNonce                               extra=true  additional=false
CheckWeight                              extra=false additional=false
ReversibleTransactionExtension           extra=false additional=false
WormholeProofRecorderExtension           extra=false additional=false
ChargeTransactionPayment                 extra=true  additional=false
CheckMetadataHash                        extra=true  additional=true
WeightReclaim                            extra=false additional=false

Both are empty on both halves, so polkadot-js's guess of zero bytes was correct here. It was correct by luck rather than by reading, which is the argument that stands — but it was not the defect.

I did not isolate what the defect was. The working submission differs from the failed one in using an immortal era where polkadot-js chose a mortal one, which makes CheckMortality's implicit the genesis hash instead of a birth block both sides have to agree on; that is the obvious candidate and it is not evidence. Chasing it further would mean debugging a code path that is being removed, so it stays unknown and is recorded as unknown.

Closing. The question in the title — patch @polkadot/types, or override the registry — has the answer neither, and the reasons are §1–§3 above rather than the BadProof. Those three stand on their own: the 2048-byte cap, the preamble byte read as a version, and guessing at extensions the metadata declares.

## Resolved, by removal `@quantus/codec` (quantus/wasm#3) landed and quantus/extension#7 tier 1 now signs a `balances.transfer_keep_alive` that Heisenberg **accepts and includes** — block 1050475, nonce 2761 → 2762 — with `@polkadot/api` entirely out of the path. `WsProvider` remains only as a JSON-RPC transport; no type is decoded by polkadot-js. One correction to the analysis above, for the record. Under §4 I listed the two unrecognised extensions as the remaining suspect for the `BadProof`. Having read the metadata with the new codec, the registry says they are innocent: ``` CheckNonZeroSender extra=false additional=false CheckSpecVersion extra=false additional=true CheckTxVersion extra=false additional=true CheckGenesis extra=false additional=true CheckMortality extra=true additional=true CheckNonce extra=true additional=false CheckWeight extra=false additional=false ReversibleTransactionExtension extra=false additional=false WormholeProofRecorderExtension extra=false additional=false ChargeTransactionPayment extra=true additional=false CheckMetadataHash extra=true additional=true WeightReclaim extra=false additional=false ``` Both are empty on both halves, so polkadot-js's guess of zero bytes was correct here. It was correct by luck rather than by reading, which is the argument that stands — but it was not the defect. **I did not isolate what the defect was.** The working submission differs from the failed one in using an immortal era where polkadot-js chose a mortal one, which makes `CheckMortality`'s implicit the genesis hash instead of a birth block both sides have to agree on; that is the obvious candidate and it is not evidence. Chasing it further would mean debugging a code path that is being removed, so it stays unknown and is recorded as unknown. Closing. The question in the title — patch `@polkadot/types`, or override the registry — has the answer *neither*, and the reasons are §1–§3 above rather than the `BadProof`. Those three stand on their own: the 2048-byte cap, the preamble byte read as a version, and guessing at extensions the metadata declares.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantus/api#1