Support persistent storage in Containers (#493)

Extend the work of https://github.com/0xPARC/pod2/pull/487 to the Containers (Dictionary, Set, Array).

The merkle tree only stores `RawValue` for both the key and the value, so it is the responsibility of the Container to store the rich value.

In order to handle containers with persistent storage efficiently (which means, cloning them or updating them should not cause an O(n) data copy) I figured we need to have a database of `Value`s indexed by their raw value; as this gives us deduplication and free cloning of containers.
The issue with this approach is that in the current design we have collisions between Value's of different types: https://github.com/0xPARC/pod2/issues/426 and the current API relies on the single type of values.

To resolve this issue I decided to change the API, instead of assuming that a Value has a fixed type, let the value be possibly multiple compatible types and let the user of the library try casting the Value to a particular type.
For this I deprecated the public access of everything related to `TypedValue` and I propose for it to be considered an implementation detail and a blackbox from the external developer point of view.  The `Value` type is now used like this:
- To create a new Value use `Value::from(...)` where you can pass any compatible type (the same types as before)
- To access the Value in typed form you cast it like `value.as_foo()` which returns `Option<Foo>`.

Previously we had a collision between `true` and `1` (and `false` and `0`).  Now it doesn't matter whether a value holds a `true` or a `1`, both should be seen as the same and both return `Some` when doing `as_int` and `as_bool`.

Similarly we had collisions with containers.  For example `set(0, 1, 2) == array[0, 1, 2]` and `set("a", "b") = dict("a": "a", "b": "b")`.  Now any container can be casted to any of `set, array, dict`.  There's a caveat here: each of these types expects a particular encoding of keys, so casting to the wrong type will return errors on some operations.

With this design it no longer matters what is being stored and recovered because the API requires the user to express the expected type and any type with collisions for particular values can be casted to the right type.

There's only one case where it's not desirable to swap one `TypedValue` for another: the `TypedValue::Raw`.  If a non-`RawValue` in the DB is replaced by the corresponding `RawValue` we erase the required information to recover the rich value.  For this reason the implementations of the database treat the `RawValue` as a special case: if an value is stored in non-`RawValue`, the corresponding `RawValue` can never overwrite it.  If a value is stored in `RawValue`, a matching non-`RawValue` will overwrite it (promoting it to a rich value).  This way we never lose data.

A consequence of this is that the serialization, `Display` and `Debug` of a container is not stable.  At any point any of the entries can be swapped for a "compatible" one if they share the storage with other containers that introduce collisions.

I rewrote all containers as wrapper to a generic `Container` which holds a `Map` from `Value` to `Value`.  The serialization of each container now uses the single implementation of the generic `Container`.
This commit is contained in:
Eduard S. 2026-03-23 12:31:28 +01:00 committed by GitHub
parent 32f45872d7
commit 13cabdb511
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
22 changed files with 1187 additions and 621 deletions

View file

@ -61,8 +61,8 @@ macro_rules! new {
}
use InnerError::*;
impl Error {
pub fn custom(s: String) -> Self {
new!(Custom(s))
pub fn custom(s: impl Into<String>) -> Self {
new!(Custom(s.into()))
}
pub fn plonky2_proof_fail(context: impl Into<String>, e: anyhow::Error) -> Self {
Self::Plonky2ProofFail(context.into(), e)

View file

@ -225,11 +225,10 @@ pub(crate) fn extract_public_key_of(
) = (op, st)
{
let deduction_err = || MiddlewareError::invalid_deduction(op.clone(), st.clone());
let sk = SecretKey::try_from(
value_from_op(sk_s, sk_ref)
.ok_or_else(deduction_err)?
.typed(),
)?;
let value = value_from_op(sk_s, sk_ref).ok_or_else(deduction_err)?;
let sk = value
.as_secret_key()
.ok_or_else(|| Error::custom("{value} not SecretKey"))?;
aux_list[i] = OperationAux::PublicKeyOfIndex(table.len());
table.push(sk);
}
@ -283,7 +282,9 @@ pub(crate) fn extract_signatures(
aux_list[i] = OperationAux::SignedByIndex(table.len());
table.push(SignedBy {
msg: msg.raw(),
pk: PublicKey::try_from(pk.typed())?,
pk: pk
.as_public_key()
.ok_or_else(|| Error::custom(format!("{pk} is not PublicKey")))?,
sig: sig.clone(),
});
}

View file

@ -207,7 +207,7 @@ impl Point {
u: *u,
});
points.find(|p| p.is_in_subgroup()).ok_or(Error::custom(
"One of the points must lie in the EC subgroup.".into(),
"One of the points must lie in the EC subgroup.",
))
}
pub fn as_bytes_from_subgroup(&self) -> Result<Vec<u8>, Error> {

View file

@ -6,19 +6,20 @@ use std::{
sync::{Arc, Mutex},
};
use anyhow::{anyhow, bail, Result};
use anyhow::{anyhow, Result};
use dyn_clone::DynClone;
use crate::{
backends::plonky2::primitives::merkletree::{Leaf, Node},
middleware::{RawValue, EMPTY_VALUE},
backends::plonky2::primitives::merkletree::{Intermediate, Node},
middleware::{Hash, EMPTY_HASH},
};
#[cfg(feature = "db_rocksdb")]
pub mod rocks;
pub trait DB: Debug + DynClone + Sync + Send {
fn load_node(&self, hash: RawValue) -> Result<Node>;
/// Must always return the empty intermediate node when hash is EMPTY_HASH
fn load_node(&self, hash: Hash) -> Result<Option<Node>>;
fn store_node(&mut self, node: Node) -> Result<()>;
}
dyn_clone::clone_trait_object!(DB);
@ -26,7 +27,7 @@ dyn_clone::clone_trait_object!(DB);
/// MemDB implements the DB trait in a in-memory HashMap.
#[derive(Clone, Debug, Default)]
pub(crate) struct MemDB {
inner: Arc<Mutex<HashMap<RawValue, Node>>>,
inner: Arc<Mutex<HashMap<Hash, Node>>>,
}
impl MemDB {
@ -36,21 +37,18 @@ impl MemDB {
}
impl DB for MemDB {
fn load_node(&self, hash: RawValue) -> Result<Node> {
fn load_node(&self, hash: Hash) -> Result<Option<Node>> {
let db = self
.inner
.lock()
.map_err(|e| anyhow!("failed to acquire memdb lock for read: {}", e))?;
if let Some(node) = db.get(&hash) {
return Ok(node.clone());
if hash == EMPTY_HASH {
return Ok(Some(Node::Intermediate(Intermediate::new(
EMPTY_HASH, EMPTY_HASH,
))));
}
if hash == EMPTY_VALUE {
return Ok(Node::Leaf(Leaf::new(hash, EMPTY_VALUE)));
}
bail!("MemDB error: node not found: {}", hash);
Ok(db.get(&hash).cloned())
}
fn store_node(&mut self, node: Node) -> Result<()> {
@ -58,25 +56,15 @@ impl DB for MemDB {
.inner
.lock()
.map_err(|e| anyhow!("failed to acquire memdb lock for write: {}", e))?;
db.insert(node.hash().into(), node);
db.insert(node.hash(), node);
Ok(())
}
}
// NOTE: this can be replaced by `.to_bytes` & `from_bytes` optimized methods at `Node`
#[allow(dead_code)]
fn encode_node(node: &Node) -> Result<Vec<u8>> {
serde_json::to_vec(node).map_err(|e| anyhow!("failed to serialize node: {e}"))
}
#[allow(dead_code)]
fn decode_node(bytes: &[u8]) -> Result<Node> {
serde_json::from_slice(bytes).map_err(|e| anyhow!("failed to deserialize node: {e}"))
}
#[cfg(test)]
pub mod tests {
use super::*;
use super::{super::Leaf, *};
#[test]
fn test_db() -> Result<()> {
@ -97,7 +85,7 @@ pub mod tests {
let node = Leaf::new(1.into(), 1.into());
db.store_node(Node::Leaf(node.clone()))?;
let obtained_node = db.load_node(node.hash.into())?;
let obtained_node = db.load_node(node.hash)?.unwrap();
let leaf = match obtained_node {
Node::Leaf(l) => l,
_ => panic!("expected a leaf"),

View file

@ -3,10 +3,9 @@ use std::{fmt, path::Path, sync::Arc};
use anyhow::{anyhow, Result};
use rocksdb::{Options, TransactionDB, TransactionDBOptions};
use super::DB;
use crate::{
backends::plonky2::primitives::merkletree::{Leaf, Node},
middleware::{RawValue, EMPTY_VALUE},
backends::plonky2::primitives::merkletree::{self, db},
middleware::{Hash, RawValue, EMPTY_HASH},
};
#[derive(Clone)]
@ -30,29 +29,27 @@ impl fmt::Debug for RocksDB {
}
}
impl DB for RocksDB {
fn load_node(&self, hash: RawValue) -> Result<Node> {
if hash == EMPTY_VALUE {
return Ok(Node::Leaf(Leaf::new(hash, EMPTY_VALUE)));
impl db::DB for RocksDB {
fn load_node(&self, hash: Hash) -> Result<Option<merkletree::Node>> {
if hash == EMPTY_HASH {
return Ok(Some(merkletree::Node::Intermediate(
merkletree::Intermediate::new(EMPTY_HASH, EMPTY_HASH),
)));
}
let maybe_node_bytes = self
match self
.0
.get(hash.to_bytes())
.map_err(|e| anyhow!("rocksdb transaction get failed: {e}"))?;
match maybe_node_bytes {
Some(bytes) => super::decode_node(&bytes),
None => Err(anyhow!("rocksdb: node not found")),
.get(RawValue::from(hash).to_bytes())
.map_err(|e| anyhow!("rocksdb: get failed: {e}"))?
{
None => Ok(None),
Some(bytes) => Ok(Some(merkletree::Node::decode(bytes.as_ref())?)),
}
}
fn store_node(&mut self, node: Node) -> Result<()> {
fn store_node(&mut self, node: merkletree::Node) -> Result<()> {
self.0
.put(
RawValue::from(node.hash()).to_bytes(),
super::encode_node(&node)?,
)
.put(RawValue::from(node.hash()).to_bytes(), node.encode()?)
.map_err(|e| anyhow!("rocksdb transaction put failed: {e}"))
}
}

View file

@ -2,12 +2,16 @@
use std::{backtrace::Backtrace, fmt::Debug};
use crate::middleware::Hash;
pub type TreeResult<T, E = TreeError> = core::result::Result<T, E>;
#[derive(Debug, thiserror::Error)]
pub enum TreeInnerError {
#[error("key not found")]
KeyNotFound,
#[error("node with hash {0} not found")]
NodeNotFound(Hash),
#[error("key already exists")]
KeyExists,
#[error("max depth reached")]
@ -22,6 +26,9 @@ pub enum TreeInnerError {
StateTransitionProofFail(String),
#[error("circuit max_depth {0} is smaller than proof depth {1}")]
CircuitDepthTooSmall(usize, usize),
// Other
#[error("{0}")]
Custom(String),
}
#[derive(thiserror::Error)]
@ -31,8 +38,8 @@ pub enum TreeError {
inner: Box<TreeInnerError>,
backtrace: Box<Backtrace>,
},
#[error("anyhow::Error: {0}")]
Anyhow(#[from] anyhow::Error),
#[error("database error: {0}")]
Database(anyhow::Error),
}
impl Debug for TreeError {
@ -60,6 +67,9 @@ impl TreeError {
pub(crate) fn key_not_found() -> Self {
new!(KeyNotFound)
}
pub(crate) fn node_not_found(hash: Hash) -> Self {
new!(NodeNotFound(hash))
}
pub(crate) fn key_exists() -> Self {
new!(KeyExists)
}
@ -81,4 +91,7 @@ impl TreeError {
pub(crate) fn circuit_depth_too_small(circuit_depth: usize, proof_depth: usize) -> Self {
new!(CircuitDepthTooSmall(circuit_depth, proof_depth))
}
pub(crate) fn custom(s: impl Into<String>) -> Self {
new!(Custom(s.into()))
}
}

View file

@ -2,7 +2,7 @@
//! <https://0xparc.github.io/pod2/merkletree.html> .
use std::{collections::HashMap, fmt, iter::IntoIterator};
use anyhow::{anyhow, Result};
use anyhow::anyhow;
use itertools::zip_eq;
use plonky2::{
field::types::Field,
@ -16,10 +16,15 @@ use crate::middleware::{Hash, RawValue, EMPTY_HASH, EMPTY_VALUE, F};
pub mod circuit;
pub use circuit::*;
mod db;
use db::DB;
pub mod db;
pub use db::DB;
pub mod error;
pub use error::{TreeError, TreeResult};
use error::{TreeError as Error, TreeResult as Result};
// TODO: Replace all `&RawValue` for `RawValue`. This type is very small and `Copy` so there's
// no benefit in passing a reference instead of a copy. Moreover, most of the times the value is
// being copied in methods that receive the reference: see all `*key` and `*value` in the code.
/// Theoretical max depth of a merkle tree. This limits appears because we store keys of 256 bits.
const MAX_DEPTH: usize = 256;
@ -39,6 +44,20 @@ impl PartialEq for MerkleTree {
}
impl Eq for MerkleTree {}
pub(crate) fn load_node(db: &dyn DB, hash: Hash) -> Result<Node> {
match db.load_node(hash) {
Err(e) => Err(Error::Database(e)),
Ok(None) => Err(Error::node_not_found(hash)),
Ok(Some(node)) => Ok(node),
}
}
fn store_node(db: &mut dyn DB, node: Node) -> Result<()> {
match db.store_node(node) {
Ok(_) => Ok(()),
Err(e) => Err(Error::Database(e)),
}
}
impl MerkleTree {
/// builds a new `MerkleTree` where the leaves contain the given key-values
pub fn new(kvs: &HashMap<RawValue, RawValue>) -> Self {
@ -92,18 +111,18 @@ impl MerkleTree {
new_key: RawValue, // key to be added/found at the leaf
mut siblings: Option<&mut Vec<Hash>>,
op: MerkleTreeOp,
) -> TreeResult<Option<(RawValue, RawValue)>> {
) -> Result<Option<(RawValue, RawValue)>> {
let (path, lvl) = path_and_lvl;
if lvl > MAX_DEPTH {
return Err(TreeError::max_depth());
return Err(Error::max_depth());
}
if curr_node_hash == EMPTY_HASH {
return Ok(None);
}
let node = db.load_node(curr_node_hash.into())?;
let node = load_node(db, curr_node_hash)?;
match node {
Node::Intermediate(n) => {
if path[lvl] {
@ -126,7 +145,7 @@ impl MerkleTree {
if new_key == old_leaf.key {
if op == MerkleTreeOp::Insert {
// in Insert, key should not exist
return Err(TreeError::key_exists());
return Err(Error::key_exists());
}
// we're at the operation Update/Delete case
return Ok(Some((old_leaf.key, old_leaf.value)));
@ -137,7 +156,7 @@ impl MerkleTree {
curr_node_hash.into(),
old_leaf.path,
path,
siblings.ok_or(anyhow!("expected siblings, got None"))?,
siblings.ok_or(Error::custom("expected siblings, got None"))?,
)?;
Ok(Some((old_leaf.key, old_leaf.value)))
}
@ -154,9 +173,9 @@ impl MerkleTree {
old_path: Vec<bool>,
new_path: Vec<bool>,
siblings: &mut Vec<Hash>,
) -> TreeResult<()> {
) -> Result<()> {
if lvl > MAX_DEPTH {
return Err(TreeError::max_depth());
return Err(Error::max_depth());
}
if old_path[lvl] == new_path[lvl] {
siblings.push(EMPTY_HASH);
@ -181,7 +200,7 @@ impl MerkleTree {
first_zeroes: bool,
) -> Result<Hash> {
// recall, in the delete case, the `key` is the `remaining_key`
let key_node = db.load_node(key.into())?;
let key_node = load_node(db, key)?;
if op == MerkleTreeOp::Delete
&& first_zeroes
&& matches!(key_node, Node::Leaf(..))
@ -208,7 +227,7 @@ impl MerkleTree {
let node_hash = node.hash; // variable to avoid cloning `node` later
// store in db
db.store_node(Node::Intermediate(node))?;
store_node(db, Node::Intermediate(node))?;
if curr_lvl == 0 {
return Ok(node_hash);
@ -217,7 +236,7 @@ impl MerkleTree {
}
/// returns the value at the given key
pub fn get(&self, key: &RawValue) -> TreeResult<RawValue> {
pub fn get(&self, key: &RawValue) -> Result<Option<RawValue>> {
let path = keypath(*key);
let key_resolution = Self::down(
self.db.as_ref(),
@ -228,13 +247,13 @@ impl MerkleTree {
MerkleTreeOp::ReadOnly,
)?;
match key_resolution {
Some((k, v)) if &k == key => Ok(v),
_ => Err(TreeError::key_not_found()),
Some((k, v)) if &k == key => Ok(Some(v)),
_ => Ok(None),
}
}
/// returns a boolean indicating whether the key exists in the tree
pub fn contains(&self, key: &RawValue) -> TreeResult<bool> {
pub fn contains(&self, key: &RawValue) -> Result<bool> {
let path = keypath(*key);
match Self::down(
self.db.as_ref(),
@ -253,7 +272,7 @@ impl MerkleTree {
&mut self,
key: &RawValue,
value: &RawValue,
) -> TreeResult<MerkleTreeStateTransitionProof> {
) -> Result<MerkleTreeStateTransitionProof> {
let proof_non_existence = self.prove_nonexistence(key)?;
let old_root: Hash = self.root;
@ -287,7 +306,7 @@ impl MerkleTree {
&mut self,
key: &RawValue,
value: &RawValue,
) -> TreeResult<MerkleTreeStateTransitionProof> {
) -> Result<MerkleTreeStateTransitionProof> {
let (old_value, old_proof) = self.prove(key)?;
let old_root: Hash = self.root;
@ -316,7 +335,7 @@ impl MerkleTree {
})
}
pub fn delete(&mut self, key: &RawValue) -> TreeResult<MerkleTreeStateTransitionProof> {
pub fn delete(&mut self, key: &RawValue) -> Result<MerkleTreeStateTransitionProof> {
let (value, proof_existence) = self.prove(key)?;
let old_root: Hash = self.root;
@ -346,7 +365,7 @@ impl MerkleTree {
/// returns a proof of existence, which proves that the given key exists in
/// the tree. It returns the `value` of the leaf at the given `key`, and the
/// `MerkleProof`.
pub fn prove(&self, key: &RawValue) -> TreeResult<(RawValue, MerkleProof)> {
pub fn prove(&self, key: &RawValue) -> Result<(RawValue, MerkleProof)> {
let path = keypath(*key);
let mut siblings: Vec<Hash> = Vec::new();
@ -366,7 +385,7 @@ impl MerkleTree {
other_leaf: None,
},
)),
_ => Err(TreeError::key_not_found()),
_ => Err(Error::key_not_found()),
}
}
@ -374,7 +393,7 @@ impl MerkleTree {
/// `key` does not exist in the tree. The return value specifies
/// the key-value pair in the leaf reached as a result of
/// resolving `key` as well as a `MerkleProof`.
pub fn prove_nonexistence(&self, key: &RawValue) -> TreeResult<MerkleProof> {
pub fn prove_nonexistence(&self, key: &RawValue) -> Result<MerkleProof> {
let path = keypath(*key);
let mut siblings: Vec<Hash> = Vec::new();
@ -400,22 +419,17 @@ impl MerkleTree {
siblings,
other_leaf: Some((k, v)),
}),
_ => Err(TreeError::key_exists()),
_ => Err(Error::key_exists()),
}
// both cases prove that the given key don't exist in the tree.
}
/// verifies an inclusion proof for the given `key` and `value`
pub fn verify(
root: Hash,
proof: &MerkleProof,
key: &RawValue,
value: &RawValue,
) -> TreeResult<()> {
pub fn verify(root: Hash, proof: &MerkleProof, key: &RawValue, value: &RawValue) -> Result<()> {
let h = proof.compute_root_from_leaf(key, Some(*value))?;
if h != root {
Err(TreeError::proof_fail("inclusion".to_string()))
Err(Error::proof_fail("inclusion".to_string()))
} else {
Ok(())
}
@ -423,18 +437,16 @@ impl MerkleTree {
/// verifies a non-inclusion proof for the given `key`, that is, the given
/// `key` does not exist in the tree
pub fn verify_nonexistence(root: Hash, proof: &MerkleProof, key: &RawValue) -> TreeResult<()> {
pub fn verify_nonexistence(root: Hash, proof: &MerkleProof, key: &RawValue) -> Result<()> {
match proof.other_leaf {
Some((k, _v)) if &k == key => {
Err(TreeError::invalid_proof("non-existence".to_string()))
}
Some((k, _v)) if &k == key => Err(Error::invalid_proof("non-existence".to_string())),
_ => {
let k = proof.other_leaf.map(|(k, _)| k).unwrap_or(*key);
let v: Option<RawValue> = proof.other_leaf.map(|(_, v)| v);
let h = proof.compute_root_from_leaf(&k, v)?;
if h != root {
Err(TreeError::proof_fail("exclusion".to_string()))
Err(Error::proof_fail("exclusion".to_string()))
} else {
Ok(())
}
@ -442,7 +454,7 @@ impl MerkleTree {
}
}
pub fn verify_state_transition(proof: &MerkleTreeStateTransitionProof) -> TreeResult<()> {
pub fn verify_state_transition(proof: &MerkleTreeStateTransitionProof) -> Result<()> {
let mut old_siblings = proof.op_proof.siblings.clone();
let new_siblings = proof.siblings.clone();
@ -459,7 +471,7 @@ impl MerkleTree {
}
MerkleTreeOp::Update => {
if proof.value.is_none() {
return Err(TreeError::state_transition_fail(
return Err(Error::state_transition_fail(
"Invalid proof of update: proof.value should not be None".to_string(),
));
}
@ -485,7 +497,7 @@ impl MerkleTree {
// All siblings should agree
(proof.siblings == proof.op_proof.siblings)
.then_some(())
.ok_or(TreeError::state_transition_fail(format!(
.ok_or(Error::state_transition_fail(format!(
"Invalid proof of update for key {}: Siblings don't match.",
proof.op_key
)))
@ -514,11 +526,11 @@ impl MerkleTree {
let divergence_lvl: usize =
match zip_eq(old_path, new_path).position(|(x, y)| x != y) {
Some(d) => d,
None => return Err(TreeError::max_depth()),
None => return Err(Error::max_depth()),
};
if divergence_lvl != new_siblings.len() - 1 {
return Err(TreeError::state_transition_fail(
return Err(Error::state_transition_fail(
"paths divergence does not match".to_string(),
));
}
@ -534,7 +546,7 @@ impl MerkleTree {
if new_siblings.is_empty() {
return (old_siblings.is_empty() && proof.old_root == EMPTY_HASH)
.then_some(())
.ok_or(TreeError::state_transition_fail(
.ok_or(Error::state_transition_fail(
"new tree has no siblings yet old tree is not the empty tree"
.to_string(),
));
@ -544,14 +556,14 @@ impl MerkleTree {
old_siblings.resize(d + 1, EMPTY_HASH);
for i in 0..d {
if old_siblings[i] != new_siblings[i] {
return Err(TreeError::state_transition_fail(
return Err(Error::state_transition_fail(
"siblings don't match: old[i]!=new[i] ∀ i (except at i==d)".to_string(),
));
}
}
if old_siblings[d] != new_siblings[d] {
if old_siblings[d] != EMPTY_HASH {
return Err(TreeError::state_transition_fail(
return Err(Error::state_transition_fail(
"siblings don't match: old[d]!=empty".to_string(),
));
}
@ -559,20 +571,20 @@ impl MerkleTree {
.op_proof
.other_leaf
.map(|(k, _)| k)
.ok_or(TreeError::state_transition_fail(
.ok_or(Error::state_transition_fail(
"proof.proof_non_existence.other_leaf can not be empty for the case old_siblings[d]!=new_siblings[d]".to_string()
))?;
let v: Option<RawValue> = proof.op_proof.other_leaf.map(|(_, v)| v);
let old_leaf_hash = kv_hash(&k, v);
if new_siblings[d] != old_leaf_hash {
return Err(TreeError::state_transition_fail(
return Err(Error::state_transition_fail(
"siblings don't match: new[d]!=old_leaf_hash".to_string(),
));
}
}
Ok(())
}
_ => Err(TreeError::invalid_proof("proof.op".to_string())),
_ => Err(Error::invalid_proof("proof.op".to_string())),
}
}
}
@ -586,27 +598,25 @@ impl MerkleTree {
root: Hash,
k: RawValue,
maybe_value: Option<RawValue>,
) -> TreeResult<Hash> {
) -> Result<Hash> {
// Rule out invalid arguments
match (op, maybe_value) {
(MerkleTreeOp::Insert, None) | (MerkleTreeOp::Update, None) => {
Err(TreeError::invalid_state_transition_proof_arg(format!(
Err(Error::invalid_state_transition_proof_arg(format!(
"{:?} op requires a value argument.",
op
)))
}
(MerkleTreeOp::Delete, Some(_)) => {
Err(TreeError::invalid_state_transition_proof_arg(format!(
Err(Error::invalid_state_transition_proof_arg(format!(
"{:?} op requires no value argument, yet one was provided.",
op
)))
}
(MerkleTreeOp::ReadOnly, _) => {
Err(TreeError::invalid_state_transition_proof_arg(format!(
"{:?} 'read only' op should not reach the 'apply_op' method",
op
)))
}
(MerkleTreeOp::ReadOnly, _) => Err(Error::invalid_state_transition_proof_arg(format!(
"{:?} 'read only' op should not reach the 'apply_op' method",
op
))),
_ => Ok(()),
}?;
@ -627,8 +637,7 @@ impl MerkleTree {
Node::Leaf(Leaf::new(k, value))
}
(MerkleTreeOp::Delete, None) => {
// return an intermediate node whose hash is 'empty', to
// indicate that there is no leaf
// return a node whose hash is 'empty', to indicate that there is no leaf
Node::Intermediate(Intermediate {
hash: EMPTY_HASH,
left: EMPTY_HASH,
@ -636,16 +645,16 @@ impl MerkleTree {
})
}
_ => {
return Err(TreeError::invalid_state_transition_proof_arg(format!(
return Err(Error::invalid_state_transition_proof_arg(format!(
"{:?} op has invalid value type: {:?}",
op, maybe_value
)))
}
};
let node_hash = node.hash(); // variable to avoid cloning `node` later
db.store_node(node)?;
let node_hash = node.hash(); // variable to avoid cloning `leaf` later
store_node(db, node)?;
if siblings.is_empty() {
// return the node's hash as root
// return the leaf's hash as root
return Ok(node_hash);
}
@ -654,7 +663,7 @@ impl MerkleTree {
// we're at the root-1 level, there is only a sibling, and we're
// removing the current leaf.
// If the sibling is a Leaf, the sibling (leaf) is now the new root
let sibling_node = db.load_node(siblings[0].into())?;
let sibling_node = load_node(db, siblings[0])?;
if matches!(sibling_node, Node::Leaf(..)) {
return Ok(siblings[0]);
}
@ -669,7 +678,7 @@ impl MerkleTree {
let node_hash = node.hash; // variable to avoid cloning `node` later
// store in db
db.store_node(Node::Intermediate(node))?;
store_node(db, Node::Intermediate(node))?;
return Ok(node_hash);
}
// use the last sibling as the key that we will push up from
@ -743,48 +752,39 @@ fn hash_with_flag(flag: F, inputs: &[F]) -> Hash {
impl MerkleTree {
/// returns an iterator over the leaves of the tree
pub fn iter(&self) -> Iter<'_> {
pub fn iter(&self) -> Iter {
Iter {
state: if self.root == EMPTY_HASH {
vec![]
} else {
vec![self.root]
},
db: self.db.as_ref(),
db: self.db.clone(),
}
}
}
impl<'a> IntoIterator for &'a MerkleTree {
impl IntoIterator for &MerkleTree {
type Item = (RawValue, RawValue);
type IntoIter = Iter<'a>;
type IntoIter = Iter;
fn into_iter(self) -> Self::IntoIter {
self.iter()
}
}
pub struct Iter<'a> {
pub struct Iter {
state: Vec<Hash>,
db: &'a dyn DB,
db: Box<dyn DB>,
}
impl<'a> Iterator for Iter<'a> {
impl Iterator for Iter {
type Item = (RawValue, RawValue);
fn next(&mut self) -> Option<Self::Item> {
let node_hash = self.state.pop()?;
// Inspect node
let node = self.db.load_node(node_hash.into()).ok()?;
let node = load_node(self.db.as_ref(), node_hash).ok()?;
match node {
Node::Leaf(Leaf {
hash: _,
path: _,
key,
value,
}) => Some((key, value)),
Node::Intermediate(Intermediate {
hash: _,
left,
right,
}) => {
Node::Leaf(Leaf { key, value, .. }) => Some((key, value)),
Node::Intermediate(Intermediate { left, right, .. }) => {
[right, left].into_iter().for_each(|h| {
if h != EMPTY_HASH {
self.state.push(h)
@ -814,7 +814,7 @@ fn print_graph_viz(f: &mut fmt::Formatter<'_>, db: &dyn DB, hash: Hash) -> fmt::
return Ok(());
}
let node = db.load_node(hash.into()).map_err(|_| fmt::Error)?;
let node = load_node(db, hash).map_err(|_| fmt::Error)?;
match node {
Node::Intermediate(n) => {
let left_hash: String = if n.left == EMPTY_HASH {
@ -878,14 +878,14 @@ impl MerkleProof {
/// Computes the root of the Merkle tree suggested by a Merkle proof given a
/// key & value. If a value is not provided, the terminal node is assumed to
/// be empty.
fn compute_root_from_leaf(&self, key: &RawValue, value: Option<RawValue>) -> TreeResult<Hash> {
fn compute_root_from_leaf(&self, key: &RawValue, value: Option<RawValue>) -> Result<Hash> {
let path = keypath(*key);
let h = kv_hash(key, value);
self.compute_root_from_node(&h, path)
}
fn compute_root_from_node(&self, node_hash: &Hash, path: Vec<bool>) -> TreeResult<Hash> {
fn compute_root_from_node(&self, node_hash: &Hash, path: Vec<bool>) -> Result<Hash> {
if self.siblings.len() > MAX_DEPTH {
return Err(TreeError::max_depth());
return Err(Error::max_depth());
}
let mut h = *node_hash;
for (i, sibling) in self.siblings.iter().enumerate().rev() {
@ -1001,19 +1001,17 @@ pub enum Node {
impl Node {
pub fn hash(&self) -> Hash {
match self {
Node::Leaf(Leaf {
hash,
path: _,
key: _,
value: _,
}) => *hash,
Node::Intermediate(Intermediate {
hash,
left: _,
right: _,
}) => *hash,
Node::Leaf(Leaf { hash, .. }) => *hash,
Node::Intermediate(Intermediate { hash, .. }) => *hash,
}
}
// NOTE: this can be replaced by `.to_bytes` & `from_bytes` optimized methods at `Node`
pub fn encode(&self) -> Result<Vec<u8>, anyhow::Error> {
serde_json::to_vec(self).map_err(|e| anyhow!("failed to serialize node: {e}"))
}
pub fn decode(bytes: &[u8]) -> Result<Node, anyhow::Error> {
serde_json::from_slice(bytes).map_err(|e| anyhow!("failed to deserialize node: {e}"))
}
}
#[derive(Clone, Debug, Serialize, Deserialize)]
@ -1023,7 +1021,7 @@ pub struct Intermediate {
right: Hash,
}
impl Intermediate {
fn new(left: Hash, right: Hash) -> Self {
pub fn new(left: Hash, right: Hash) -> Self {
if left == EMPTY_HASH && right == EMPTY_HASH {
return Self {
hash: EMPTY_HASH,
@ -1045,7 +1043,7 @@ pub struct Leaf {
pub(crate) value: RawValue,
}
impl Leaf {
fn new(key: RawValue, value: RawValue) -> Self {
pub fn new(key: RawValue, value: RawValue) -> Self {
Self {
hash: kv_hash(&key, Some(value)),
path: keypath(key),
@ -1079,21 +1077,21 @@ pub mod tests {
use super::*;
#[test]
fn test_merkletree() -> TreeResult<()> {
fn test_merkletree() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_merkletree_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_merkletree_opt(db)?;
}
Ok(())
}
fn test_merkletree_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_merkletree_opt(db: Box<dyn DB>) -> Result<()> {
let mut kvs = HashMap::new();
for i in 0..8 {
if i == 1 {
@ -1168,21 +1166,40 @@ pub mod tests {
}
#[test]
fn test_delete_to_empty() -> TreeResult<()> {
fn test_key_not_found() -> Result<()> {
let db = Box::new(db::MemDB::new());
let mut tree = MerkleTree::empty_with_db(db.clone());
let value_option = tree.get(&RawValue::from(5)).unwrap();
assert_eq!(None, value_option);
tree.insert(&RawValue::from(1), &RawValue::from(42))?;
let value_option = tree.get(&RawValue::from(5)).unwrap();
assert_eq!(None, value_option);
// If the root doesn't exist there should be an error
let tree = MerkleTree::from_db(Hash::from(RawValue::from(42)), db);
let result = tree.get(&RawValue::from(5));
assert!(result.is_err());
Ok(())
}
#[test]
fn test_delete_to_empty() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_delete_to_empty_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_delete_to_empty_opt(db)?;
}
Ok(())
}
fn test_delete_to_empty_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_delete_to_empty_opt(db: Box<dyn DB>) -> Result<()> {
let mut tree = MerkleTree::new_with_db(db, &HashMap::new())?;
let (key, value) = (RawValue::from(2), RawValue::from(1002));
@ -1212,21 +1229,21 @@ pub mod tests {
}
#[test]
fn test_prove_verify() -> TreeResult<()> {
fn test_prove_verify() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_prove_verify_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_prove_verify_opt(db)?;
}
Ok(())
}
fn test_prove_verify_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_prove_verify_opt(db: Box<dyn DB>) -> Result<()> {
let kvs = [
(1.into(), 55.into()),
(2.into(), 88.into()),
@ -1255,21 +1272,21 @@ pub mod tests {
}
#[test]
fn test_update_leaf() -> TreeResult<()> {
fn test_update_leaf() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_update_leaf_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_update_leaf_opt(db)?;
}
Ok(())
}
fn test_update_leaf_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_update_leaf_opt(db: Box<dyn DB>) -> Result<()> {
let kvs = [
(1.into(), 1.into()),
(9.into(), 9.into()),
@ -1304,21 +1321,21 @@ pub mod tests {
}
#[test]
fn test_update_delete_leaf() -> TreeResult<()> {
fn test_update_delete_leaf() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_update_delete_leaf_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_update_delete_leaf_opt(db)?;
}
Ok(())
}
fn test_update_delete_leaf_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_update_delete_leaf_opt(db: Box<dyn DB>) -> Result<()> {
let kvs: HashMap<RawValue, RawValue> = (0..10)
.map(|i| (i.into(), i.into()))
.collect::<HashMap<_, _>>();
@ -1348,21 +1365,21 @@ pub mod tests {
}
#[test]
fn test_delete_leaf() -> TreeResult<()> {
fn test_delete_leaf() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_delete_leaf_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_delete_leaf_opt(db)?;
}
Ok(())
}
fn test_delete_leaf_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_delete_leaf_opt(db: Box<dyn DB>) -> Result<()> {
let kvs = [(1.into(), 1.into()), (9.into(), 9.into())]
.into_iter()
.collect();
@ -1403,21 +1420,21 @@ pub mod tests {
}
#[test]
fn test_delete_from_two_leaves() -> TreeResult<()> {
fn test_delete_from_two_leaves() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_delete_from_two_leaves_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_delete_from_two_leaves_opt(db)?;
}
Ok(())
}
fn test_delete_from_two_leaves_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_delete_from_two_leaves_opt(db: Box<dyn DB>) -> Result<()> {
// tree with two leaves whose keys diverge at the first bit, so that when
// deleting one key leads to a tree with a single Leaf as a root
let mut kvs = HashMap::new();
@ -1439,21 +1456,21 @@ pub mod tests {
}
#[test]
fn test_state_transition() -> TreeResult<()> {
fn test_state_transition() -> Result<()> {
let db = Box::new(db::MemDB::new());
test_state_transition_opt(db)?;
#[cfg(feature = "db_rocksdb")]
{
let db = Box::new(db::rocks::RocksDB::open(
tempfile::TempDir::new().unwrap().path(),
)?);
let db = Box::new(
db::rocks::RocksDB::open(tempfile::TempDir::new().unwrap().path()).unwrap(),
);
test_state_transition_opt(db)?;
}
Ok(())
}
fn test_state_transition_opt(db: Box<dyn DB>) -> TreeResult<()> {
fn test_state_transition_opt(db: Box<dyn DB>) -> Result<()> {
let mut kvs = HashMap::new();
for i in 0..8 {
kvs.insert(RawValue::from(i), RawValue::from(1000 + i));