Support persistent storage in Containers (#493)
Extend the work of https://github.com/0xPARC/pod2/pull/487 to the Containers (Dictionary, Set, Array). The merkle tree only stores `RawValue` for both the key and the value, so it is the responsibility of the Container to store the rich value. In order to handle containers with persistent storage efficiently (which means, cloning them or updating them should not cause an O(n) data copy) I figured we need to have a database of `Value`s indexed by their raw value; as this gives us deduplication and free cloning of containers. The issue with this approach is that in the current design we have collisions between Value's of different types: https://github.com/0xPARC/pod2/issues/426 and the current API relies on the single type of values. To resolve this issue I decided to change the API, instead of assuming that a Value has a fixed type, let the value be possibly multiple compatible types and let the user of the library try casting the Value to a particular type. For this I deprecated the public access of everything related to `TypedValue` and I propose for it to be considered an implementation detail and a blackbox from the external developer point of view. The `Value` type is now used like this: - To create a new Value use `Value::from(...)` where you can pass any compatible type (the same types as before) - To access the Value in typed form you cast it like `value.as_foo()` which returns `Option<Foo>`. Previously we had a collision between `true` and `1` (and `false` and `0`). Now it doesn't matter whether a value holds a `true` or a `1`, both should be seen as the same and both return `Some` when doing `as_int` and `as_bool`. Similarly we had collisions with containers. For example `set(0, 1, 2) == array[0, 1, 2]` and `set("a", "b") = dict("a": "a", "b": "b")`. Now any container can be casted to any of `set, array, dict`. There's a caveat here: each of these types expects a particular encoding of keys, so casting to the wrong type will return errors on some operations. With this design it no longer matters what is being stored and recovered because the API requires the user to express the expected type and any type with collisions for particular values can be casted to the right type. There's only one case where it's not desirable to swap one `TypedValue` for another: the `TypedValue::Raw`. If a non-`RawValue` in the DB is replaced by the corresponding `RawValue` we erase the required information to recover the rich value. For this reason the implementations of the database treat the `RawValue` as a special case: if an value is stored in non-`RawValue`, the corresponding `RawValue` can never overwrite it. If a value is stored in `RawValue`, a matching non-`RawValue` will overwrite it (promoting it to a rich value). This way we never lose data. A consequence of this is that the serialization, `Display` and `Debug` of a container is not stable. At any point any of the entries can be swapped for a "compatible" one if they share the storage with other containers that introduce collisions. I rewrote all containers as wrapper to a generic `Container` which holds a `Map` from `Value` to `Value`. The serialization of each container now uses the single implementation of the generic `Container`.
This commit is contained in:
parent
32f45872d7
commit
13cabdb511
22 changed files with 1187 additions and 621 deletions
30
src/middleware/db/mod.rs
Normal file
30
src/middleware/db/mod.rs
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
use std::{
|
||||
collections::HashMap,
|
||||
fmt::Debug,
|
||||
sync::{Arc, RwLock},
|
||||
};
|
||||
|
||||
use dyn_clone::DynClone;
|
||||
|
||||
#[cfg(feature = "backend_plonky2")]
|
||||
use crate::backends::plonky2::primitives::merkletree::{self};
|
||||
use crate::middleware::{Hash, RawValue, Value, EMPTY_HASH};
|
||||
|
||||
pub mod mem;
|
||||
#[cfg(feature = "db_rocksdb")]
|
||||
pub mod rocks;
|
||||
|
||||
// Trait for database that stores values. Must be cheap to clone.
|
||||
pub trait DB: Debug + DynClone + Sync + Send + merkletree::db::DB {
|
||||
fn load_value(&self, raw: RawValue) -> anyhow::Result<Option<Value>>;
|
||||
// If the DB is persistent, for containers only the root needs to be stored because the
|
||||
// Container type makes sure the underlying merkle tree is stored in the DB independently, so
|
||||
// that it can be recovered back just with the root and the DB.
|
||||
// If the value is RawValue and a previous non-RawValue exists, no store overwrite it.
|
||||
// should be done. If the value is non-RawValue and a previous RawValue exists, store
|
||||
// should overwrite it.
|
||||
fn store_value(&mut self, value: Value) -> anyhow::Result<()>;
|
||||
fn is_persistent(&self) -> bool;
|
||||
fn clone_box(&self) -> Box<dyn DB>;
|
||||
}
|
||||
dyn_clone::clone_trait_object!(DB);
|
||||
Loading…
Add table
Add a link
Reference in a new issue