Extend the work of https://github.com/0xPARC/pod2/pull/487 to the Containers (Dictionary, Set, Array).
The merkle tree only stores `RawValue` for both the key and the value, so it is the responsibility of the Container to store the rich value.
In order to handle containers with persistent storage efficiently (which means, cloning them or updating them should not cause an O(n) data copy) I figured we need to have a database of `Value`s indexed by their raw value; as this gives us deduplication and free cloning of containers.
The issue with this approach is that in the current design we have collisions between Value's of different types: https://github.com/0xPARC/pod2/issues/426 and the current API relies on the single type of values.
To resolve this issue I decided to change the API, instead of assuming that a Value has a fixed type, let the value be possibly multiple compatible types and let the user of the library try casting the Value to a particular type.
For this I deprecated the public access of everything related to `TypedValue` and I propose for it to be considered an implementation detail and a blackbox from the external developer point of view. The `Value` type is now used like this:
- To create a new Value use `Value::from(...)` where you can pass any compatible type (the same types as before)
- To access the Value in typed form you cast it like `value.as_foo()` which returns `Option<Foo>`.
Previously we had a collision between `true` and `1` (and `false` and `0`). Now it doesn't matter whether a value holds a `true` or a `1`, both should be seen as the same and both return `Some` when doing `as_int` and `as_bool`.
Similarly we had collisions with containers. For example `set(0, 1, 2) == array[0, 1, 2]` and `set("a", "b") = dict("a": "a", "b": "b")`. Now any container can be casted to any of `set, array, dict`. There's a caveat here: each of these types expects a particular encoding of keys, so casting to the wrong type will return errors on some operations.
With this design it no longer matters what is being stored and recovered because the API requires the user to express the expected type and any type with collisions for particular values can be casted to the right type.
There's only one case where it's not desirable to swap one `TypedValue` for another: the `TypedValue::Raw`. If a non-`RawValue` in the DB is replaced by the corresponding `RawValue` we erase the required information to recover the rich value. For this reason the implementations of the database treat the `RawValue` as a special case: if an value is stored in non-`RawValue`, the corresponding `RawValue` can never overwrite it. If a value is stored in `RawValue`, a matching non-`RawValue` will overwrite it (promoting it to a rich value). This way we never lose data.
A consequence of this is that the serialization, `Display` and `Debug` of a container is not stable. At any point any of the entries can be swapped for a "compatible" one if they share the storage with other containers that introduce collisions.
I rewrote all containers as wrapper to a generic `Container` which holds a `Map` from `Value` to `Value`. The serialization of each container now uses the single implementation of the generic `Container`.
* refactor merkletree to work with disk keyvalue database (wip)
* various fixes post reimplementation; pending delete leaf
* add delete operation case for the new in db tree approach
* polish tree update & delete; everything works (pending polishing)
* polish panics into errs, prints, etc
* Implement iterator
* Lint
* fix case no-siblings
* case delete with semi-empty branch
* polishing
* starting to add rocksdb & heeddb for the DB & Txn traits
* Satisfy the borrow checker
* abstract merkletree tests to use the various available DBs
* update store_node interface (rm hash input), rm heed.rs
* polishing
* typos
* Ditch transactions
* add feature for rocksdb, return errs at new_with_db, remove empty leaf case in Leaf::new
* intermediate instead of leaf in empty node when deleting leaf
---------
Co-authored-by: Ahmad <root@ahmadafuni.com>
There was a data race when multiple threads/processes found that the cache directory for a given params didn't exist as they both tried to:
- Create a tmp params file
- Rename the tmp params file
Both threads would create the tmp params file (one would overwrite the other). Then one would rename the tmp file successfully, and the other would try to rename it again but would find it missing.
The fix involves using a file lock on the tmp file so that only one thread goes through the writing and renaming while others wait. It's the same approach done for cached files.
* Add failing tests
* Model statements depending on public args as cheaper than those depending on private args
* Tidying
* Fix unnecessary propagation of unused public args
* More tidying
* Tidy test comments
* Fix incorrect Delete arities
* Dedupe statements during POD-building
* Fix failure to assume existence of Contains statement
* Remove possible source of non-determinism
* Faster ILP backend
* Formatting
Adds nicer errors for Podlang code, using the `annotate_snippets` crate, the same crate used by the Rust compiler to generate contextual errors. This prints a short snippet of the code containing the error within the error message, highlighting the part that needs to be fixed.
It also includes a change to the `load_module` function, changing a `Vec` function argument to a slice.
* First pass at removing batch splitting
* Refactor to separate module loading from request parsing
* Consolidate module functionality
* Tidy up comments
* Use array of modules instead of HashMap
* Formatting
* Use module hashes when importing modules
- Export MultiPodBuilder Error
- Redefine MultiPodBuilder error to wrap the frontend Error (this way we get better formatting instead of an embedded string)
- handle initial wildcard values in `MultiPodBuilder.op` just like the `MainPodBuilder` does
* merkletree: custom poseidon to use flag as initial state.
This allows to do the merkletree related hashing in 1 gate instead of 2,
reducing ~23% of gates per merkle proof.
| tree levels | 10 | 16 | 32 | 40 | 64 | 128 | 130 | 250 | 256 |
|---------------|----|----|-----|-----|-----|-----|-----|------|------|
| old num gates | 50 | 76 | 144 | 178 | 280 | 554 | 564 | 1076 | 1102 |
| new num gates | 39 | 59 | 111 | 137 | 215 | 425 | 433 | 825 | 845 |
* update docs with new tree hashing approach
* add inline comment stating clear how the flag is used in the state permutation
Resolve https://github.com/0xPARC/pod2/issues/466
Now batches are identified by the root of a merkle tree that contains all the predicates (using sequential indices as keys). This means that the format to identify a custom predicate reference is still a hash + index, but the calculation of the hash is different.
The MainPod circuit now isn't limited by number of batches but instead number of custom predicates; and for each one we verify a merkle proof to verify the batch id.
I've removed a bunch of tests from lang that were testing splitting into multiple batches because there's no longer any need for that. In a future PR we'll remove the code that handles batch splitting.
Each custom predicate needs 148.2 gates (which is very close to my estimate of 142.7 in https://github.com/0xPARC/pod2/issues/466#issuecomment-3823531286 where I actually made a mistake and considered 5 predicates per batch instead of 4 in the previous Params).
I thought it would be nice to have a Predicate for the typed value so that the developer can work with predicates as values comfortably. Then I noticed that hashing a predicate required `Params` which would have been annoying for converting a `TypedValue::Predicate` to `RawValue` and this led to a small refactor over how `Params` work.
We already had some fields in the `Params` struct that determine compatibility between encoded data. They can be seen as determining a kind of ABI compatibility. In general it's better if those parameters don't change so that different circuit configurations can still verify proofs from each other. So I decided to force those parameters to be constant in the code base and not allow the user of our library to change them. Many field element serialization/deserialization functions in our code depended on those parameters, and since now they are constant many functions get rid of the `Params` argument, which simplifies the code. This includes the serialization of a `Predicate` which was required to calculate its hash.
* Remove CopyStatement constraints
* Use a constant objective, since the iterative approach already finds the minimum number of PODs
* Make solving/proving consume the builder
* Remove use of cached builder
* Create multiple PODs where resource limits for a single POD are exceeded
* HashSet -> BTreeSet determinism fix
* Fixed incorrect assignment of input PODs and added test
* Ensure only a single output POD
* Return error when reveal() called with unknown statement
* Use unreachable! for presumed-impossible cases
* Use assert_eq! rather than debug_assert_eq
* Use FIFO for topological sort
* Simplify bounds calculation
* Some more simplifications/comments
* Enforce dep_idx < idx invariant
* Incrementally solve rather than estimating slack
* Fix tests to correctly test dependencies between private and public statements
* More tidying
* Note possible optimisation of MainPodBuilder cloning of input PODs
* Fix tracking of total input POD count
* Refactor tests
* Formatting
* Small optimisation: use Vec in place of BTreeSet
* Account for automatically-inserted Contains statements
* Formatting
* Fix possible issue with copied statements
* Simplify result type given only a single result MainPod
* Remove unnecessary POD count estimate functionality
* Simplify dependency ordering and tracking
* Remove notion of multiple output PODs from solver
* Minor simplifications
* Use add_constraint instead of with
* Remove unnecessary check following assertion
* Fix handling of anchored keys given that Contains statements are not auto-inserted if they already exist
* Fix confusing dependency graph test
* Remove prove_order
* Fix deduplication and possible double-counting of public but not copied statements
* Reorder so that the output POD is the final POD
* Add more detailed tests
* Remove redundant tests
* Simplify POD counting
* More docs
* Flag more branches as unreachable
* Formatting
* Fix for changed custom batch parsing
* Multi-batch splitting
* Invoke split predicates by name, passing in full argument list
* Reorder batches to prevent failure of forward references where possible
* Rename APIs for clarity
* Simplify example
* Add more docs
* Review updates
* Remove duplicate code
* Comment topological sort algorithm
Implement support for first order predicates in the backend.
Now a statement template can have a predicate hash or a wildcard.
## predicate <-> predicate hash constraints
To build the custom predicate table we need to calculate the custom predicate batch id, which uses the serialization of the statement templates before normalization. This serialization uses the predicate hash when the template uses a predicate (instead of a wildcard). Then in normalization we recalculate the predicate hash if it was a Batch Self.
This means that the relation between hash and predicate must be checked before and after normalization when the template is not using a wildcard. How this is achieved:
- Before normalization: the constructor of StatementTmplTarget forces that if we keep a predicate, it's hash must be equal to the pred_hash when the template has a predicate (and not a wildcard)
- After normalization: the predicate hash is calculated in the normalization and replaced in the case of the template using a predicate and it being a BatchSelf. If it was a predicate but not batch self, the old value was used which was constrained via the constructor.
See `CircuitBuilder::add_virtual_statement_tmpl` and `normalize_st_tmpl_circuit`
## Wildcard predicate resolution
It is done via `make_predicate_from_template_circuit` and is fairly simple as it's contains similar logic to `make_statement_arg_from_template_circuit` but simpler.
The MainPodBuilder automatically adds contains statements for operations
that are created from entries. But if the contains statement was
already added manually there will be duplicates. We now track manually
added contains statements so that we don't generate duplicates when
adding statements from entries that use them.
Resolve#448
Previously a predicate was 6 elements. Now it grows to 8 elements; and the hash is 4 elements.
Some parts of the circuit require only require equality checks with the predicate: that works with the predicate hash. Other parts require inspecting or working with particular elements in the predicate, those need the preimage of the predicate hash.
Both `StatementTarget` and `StatementTmplTarget` have been updated to include the predicate hash and optionally the predicate. When the predicate is included, constraints are automatically generated for `pred_hash = hash(pred)`. We only include the predicate when needed.
* Port the example that @ax0 told me months ago about a theorem-ish perspective on predicates, and updates the signature section adding a note (& references) on the current signature scheme.
* apply review corrections from @ed255
* update example with @artwyman IsComposite suggestion & transitive_eq example
This simplifies the MerkleTree (and container) API.
Defer the max depth check when assigning the witness (merkle proof siblings) to the merkle tree circuit.
In this implementation the native Merkle Tree branches grow as much as they needed. There are no checks of max depth in the merkle tree. All keys are 256 bits (I added a debug_assert for this); so in the worst case a path will have depth 256. It can't have a longer depth because the `insert` method calls `prove_nonexistence` which errors if the key already exists; another one may exist which must be different and thus require a path <= 256 depth.
Resolve#436
* Basic frontend AST and semantic validation
* Intro statement support
* Simplify validator lifetime
* Fix arity validation
* Lowering and splitting
* Remove legacy processor and use frontend AST by default
* Use builders instead of creating middleware types directly
* Typos/formatting
* Improve error messages when overflowing a batch due to splitting
* Add FromStr implementation for NativePredicate
* Remove 'raw' fields, and switch HashHex representation to byte vector rather than string
* Simpler wrapper types for batch and intro predicate hashes
* Parse secret and public keys to their respective data structures earlier
* More detail around string escape validity
* Simplify native predicate arity handling and move method to NativePredicate impl
* Store hashes using middleware::Hash, and simplify lowering by using pre-parsed values
* Simplify predicate building
* Formatting
* Better error messages/suggestions for cases where predicate splitting fails
* Formatting
* Clippy fix
* Return error if we get a too-large int
This is useful for a verifier that wants to verify a MainPod that
contains a custom predicate in a public statement without
knowledge of how the custom predicate is defined.
- Update formula in `estimate_verif_num_gates` after the update in the recursive verification from https://github.com/0xPARC/pod2/pull/397
- Update the parameters to get better utilization of 2^16 rows
- Update metrics report to be more compact
