* refactor merkletree to work with disk keyvalue database (wip)
* various fixes post reimplementation; pending delete leaf
* add delete operation case for the new in db tree approach
* polish tree update & delete; everything works (pending polishing)
* polish panics into errs, prints, etc
* Implement iterator
* Lint
* fix case no-siblings
* case delete with semi-empty branch
* polishing
* starting to add rocksdb & heeddb for the DB & Txn traits
* Satisfy the borrow checker
* abstract merkletree tests to use the various available DBs
* update store_node interface (rm hash input), rm heed.rs
* polishing
* typos
* Ditch transactions
* add feature for rocksdb, return errs at new_with_db, remove empty leaf case in Leaf::new
* intermediate instead of leaf in empty node when deleting leaf
---------
Co-authored-by: Ahmad <root@ahmadafuni.com>
* Dedupe statements during POD-building
* Fix failure to assume existence of Contains statement
* Remove possible source of non-determinism
* Faster ILP backend
* Formatting
Adds nicer errors for Podlang code, using the `annotate_snippets` crate, the same crate used by the Rust compiler to generate contextual errors. This prints a short snippet of the code containing the error within the error message, highlighting the part that needs to be fixed.
It also includes a change to the `load_module` function, changing a `Vec` function argument to a slice.
* Create multiple PODs where resource limits for a single POD are exceeded
* HashSet -> BTreeSet determinism fix
* Fixed incorrect assignment of input PODs and added test
* Ensure only a single output POD
* Return error when reveal() called with unknown statement
* Use unreachable! for presumed-impossible cases
* Use assert_eq! rather than debug_assert_eq
* Use FIFO for topological sort
* Simplify bounds calculation
* Some more simplifications/comments
* Enforce dep_idx < idx invariant
* Incrementally solve rather than estimating slack
* Fix tests to correctly test dependencies between private and public statements
* More tidying
* Note possible optimisation of MainPodBuilder cloning of input PODs
* Fix tracking of total input POD count
* Refactor tests
* Formatting
* Small optimisation: use Vec in place of BTreeSet
* Account for automatically-inserted Contains statements
* Formatting
* Fix possible issue with copied statements
* Simplify result type given only a single result MainPod
* Remove unnecessary POD count estimate functionality
* Simplify dependency ordering and tracking
* Remove notion of multiple output PODs from solver
* Minor simplifications
* Use add_constraint instead of with
* Remove unnecessary check following assertion
* Fix handling of anchored keys given that Contains statements are not auto-inserted if they already exist
* Fix confusing dependency graph test
* Remove prove_order
* Fix deduplication and possible double-counting of public but not copied statements
* Reorder so that the output POD is the final POD
* Add more detailed tests
* Remove redundant tests
* Simplify POD counting
* More docs
* Flag more branches as unreachable
* Formatting
* Fix for changed custom batch parsing
* Multi-batch splitting
* Invoke split predicates by name, passing in full argument list
* Reorder batches to prevent failure of forward references where possible
* Rename APIs for clarity
* Simplify example
* Add more docs
* Review updates
* Remove duplicate code
* Comment topological sort algorithm
- Add a function to calculate the hash of the `CommonCircuitData`. The hash uniquely identify the `CommonCircuitData` used for a circuit/proof. Serializing the struct is not enough because the polynomial identities of the custom gates are not serialized (only their parameters are); so I made a function to extract "fingerprints" of the custom gates by evaluating them over a predefined list of uniform values, and then doing a random linear combination over the results.
- Store the full verifier only circuit data of a proof in the MainPod so that we can verify pods from old circuits in new circuits and code
- Store the hash of the `CommonCircuitData` in the MainPod so that we can reject verifying old pods that use a different `CommonCircuitData` than the current one. This has two goals
- If the `CommonCircuitData` changes it's very likely that the verification will fail, but it will be hard to debug. Doing this early check helps identify the origin of the verification failure as early as possible
- There's a chance that the verification could succeed when the `CommonCircuitData` changes, and that could be dangerous because the verification will be doing different checks than the ones intended for the original proof, so we may be skipping some constraints that could lead to exploiting the system. For this reason, whenever the common circuit data hash changes, all previous verifying keys should be discarded (that is, not included in the VDSet)
- The fingerprint only has ~64 bits and the "random evaluation point" is fixed. The assumption is that the pod developers are not malicious and are not changing the gates such that different gates give the same fingerprint. With this assumption, I find it reasonable to assume that with high probability if a gate changes, its fingerprint changes as well.
- Add a github action that updates a wiki page with a table that contains: date, commit, params hash (with a link to the actual params), verifier data only circuit data hash and common circuit data hash. This will make it easy to track when the common circuit data changes as well as track the verifier data corresponding to various versions (identified by commit)
- The edited page is this one https://github.com/0xPARC/pod2/wiki/MainPod-circuit-info
Resolve https://github.com/0xPARC/pod2/issues/386
Summary of breaking changes:
- The `RecursivePod` trait has a new method `common_hash` that needs to return the result of `hash_common_data` on the `CommonCircuitData` that the circuit uses.
- Extend the `Flattenable` trait to include a `size` method that returns the number of `Target`s the type requires. This is used in the table to figure out the max length of an array that must fit all entry types.
- Move the circuit methods to precalculate hash states and do hashes started from a precomputed state to a new module
- Introduce `MuxTableTarget` which allows easy multiplexing of tables where each sub-table may have entries of different lengths. The table access is done via hashing + unhashing automatically (via use of a generator)
- Use the `MuxTableTarget` to access merkle tree claims and custom predicate verification, which where previously in different tables and accessed with independent random accesses each
- Move the public key derivation for the PublicKeyOf operation check to the same multiplexed table. Now we can choose how many of those operations a circuit supports.
Resolve https://github.com/0xPARC/pod2/issues/357
Resolve https://github.com/0xPARC/pod2/issues/361
Add the missing gates and generator in the serializer that were added
with the PublicKeyOf operation.
Add a test for CircuitData serialization+deserialization to avoid these
kind of bugs in the future.
* wrote some initial code
* added way to input private key into circuit
* TypedValue::SecretKey hashed as 10 32-bit limbs
* Check PublicKeyOf in Frontend and Middleware
* Diff review
* PR review
* Finish utest
* Fix bounds check
* added giving secret key witness to circuit
* Test & doc improvements
* added private key comparison to circuit and added test cases
* cargo fmt
* Add frontend tests for PublicKeyOf
* Add public_key_of and hash_of to op! macro
* Add ownership check to ticket example
* Group order checking in tests
* More negative test cases at circuit level
* Cleanups after self review
* clippy fixes
* Fixes after merge. Temporarily remove plonky2 commit hash
* Add a nullifier to the ticket test example
* Test PublicKeyOf with a real prover (not mock)
* plonky-u32 dependency
* feat: optimize operation checks
Skip the circuits that verify operation checks other than None, Copy or
NewEntry for the public statements. This works because public
statements are created by copying private statements, so we never use
the other operation checks in those slots.
---------
Co-authored-by: Andrew Twyman <artwyman@gmail.com>
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
- Bump rust version to `nightly-2025-07-02` because some of the nightly features we were using have been stabilized.
- Introduce feature `disk_cache` which enables caching to disk. Each time an artifact is retrieved from the cache it will be read and deserialized. On a cache miss the artifact will be created, serialized and stored to disk.
- Introduce feature `mem_cache` which enables caching to memory. All cached artifacts are kept in memory after they are created. The mem cache implementation avoids cloning of artifacts by extending their lifetime to `'static`. This is `unsafe` code, but I argue that this usage is safe.
- Add a `build.rs`
- When the feature `disk_cache` is enabled, the `build.rs` will inject env variables to the process with the git commit information, which is used to index the cached artifacts
- Replace all previous cached artifacts from `LazyStatic` methods that call the cache API
- Derive `Serialize, Deserialize` for all `*Target` types so that they can be serialized for caching to disk
- Add finer level of caching: now we cache the `CircuitData` and `VerifierData` independently. The reason for this is that `CircuitData` is a very big artifact which is not needed for verification. So by only accessing `VerifierData` in verification we don't pay a big overhead for reading from disk and deserializing
- Add missing artifacts to the cache: like the `CircuitData` for the `MainPod` indexed by `Params`
- Add helper types to serialize and deserialize `CircuitData`, `CommonData` and `VerifierData` with the set of gates and generators used in the recursive MainPod circuit
- Tweak the ids of our custom gates so that they remain unique when their generic parameters change
- Bugfix: several tests were using the standard `vd_set` but were using MainPod circuits with non-default parameters. This was working before because there was a bug: the MainPod circuit was reporting that the used verifier data was the standard one instead of picking the one corresponding to it's own Params.
Summary of breaking changes:
- One and only one of the features `mem_cache` or `disk_cache` need to be enabled. By default it's `mem_cache`
- To enable the `disk_cache` you need to disable the default features like this: `--no-default-features --features=backend_plonky2,zk,disk_cache`
- Removed `DEFAULT_PARAMS`, instead use `Params::default()`
- Removed `STANDARD_REC_MAIN_POD_CIRCUIT_DATA`, instead use `cache_get_standard_rec_main_pod_common_circuit_data`
- The library is now using `nightly-2025-07-02`. Some rust language features are unstable in previous versions.
* add zk config, enabled by a feature (on by default)
* Update src/backends/plonky2/recursion/circuit.rs
Co-authored-by: Ahmad Afuni <root@ahmadafuni.com>
---------
Co-authored-by: Ahmad Afuni <root@ahmadafuni.com>
* Compress EC subgroup points before serialising
* serialize and display point in base58
* Use Display for Points
---------
Co-authored-by: Ahmad <root@ahmadafuni.com>
* Initial commit for Podlog language
* Spell-checker thinks that 'lits' is a bad abbreviation for 'literals'
* Enable SetContains/SetNotContains
* Update language based on review feedback
* Typo/comment fix
* Make native predicates case-sensitive
* Enforce max batch size in CustomPredicateBatchBuilder
* Remove some unnecessary checks for things handled by the grammar
* Clean up more unnecessary error-checking
* Typo
* Simplify hex processing
* Replace various errors with unreachable!()
* Translate from big-endian hex string to little-endian RawValue
* Update hex en/decoding functions
* calculate MainPod id in a dynamic-friendly way
The MainPod id is now calculated with front padding and a fixed size
independent of max_public_statements so that introduction gadgets can be
verified by a MainPod while paying only for the number of statements
they use. This is because with front padding of none-statements we can
precompute the poseidon state corresponding to absorbing all the padding
statements and only pay constraints for the non-padding statements.
The id is calculated as follows:
`id = hash(serialize(reverse(statements || none-statements)))`
* add time feature and disable timing by default
* apply suggestions from @arnaucube
* link issues in todos
* Add RecursiveCircuit
The RecursiveCircuit verifies N proofs of itself (N=arity), together with
the logic defined at the InnerCircuit (in our case, used for the
MainPodCircuit logic).
The arity defines the maximum amount of proofs of itself that the
RecursiveCircuit verifies. When arity>1, using the RecursiveCircuit has the
shape of a tree of the same arity.
π_root
▲
┌───────┴────────┐
│RecursiveCircuit│
└─▲───▲───▲────▲─┘
┌───────┘ ┌┘ └┐ └──────┐
│π''_1 │ ... │ π''_N│
┌────────┴───────┐ ┌┴┐┌─┐┌┴┐ ┌───────┴────────┐
│RecursiveCircuit│ │.││.││.│ │RecursiveCircuit│
└──▲─────────▲───┘ └─┘└─┘└─┘ └──▲─────────▲───┘
│ │ │ │
π_1 ... π_N π'_1 ... π'_N
where
N: arity of the RecursiveCircuit
π_i: plonky2 proof of the RecursiveCircuit
* add different inner-circuits in the test, reusing the same verifier_data; polish recursion interfaces
* add InnerCircuit::Params
* rm non_base_node
* WIP refactor RecursiveCircuit
* wip. change approach on verifier_data of circuits used in recursivecircuit. arity=1 works
* recursion works fine without registering verifierdatas as publicinputs
* add hashing of verifier_data (out & in-circuit) methods with test
* connect previous and current verifier_datas hashes
* polish
* add InnerCircuit's public_inputs to the RecursiveCircuit
* extend recursive test
* polish & review suggestions
* migrate from anyhow to thiserror (#190). pending polish error msgs
* Add backtrace and compartmentalize errors
- Include backtraces in the errors we generate. To get this we can't
just return a literal enum, because the backtrace requires a call.
- Related to the previous point: add methods to create errors so
we can include the backtrace conveniently without changing too much
the syntax. So instead of `Err(Error::KeyNotFound(key))` (literal
enum) it will be `Err(Error::key_not_found(key))` (method call)
- Each error should be local to its scope, and each scope should
only return its own error.
- The merkle tree should return `TreeError` and not Error
- The middleware should return `MiddlewareError` and not Error
- With a global Error we can't easily include backend/frontend types in
the error fields, so declare a `BackendError` and a `FrontendError`
and follow the pattern from the previous point
- The Pod traits should be able to return backend errors and will be
used in the frontend; for that we change them to use trait object
Error: `dyn std::error::Error`
* fix error
* apply suggestions from @arnaucube
* rename XError and XResult to Error and Result
* reorg signature
* make frontend custom error more ergonomic
* remove unnecessary feature
---------
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
* Serialization tests now pass again
* Tidy up and test more edge-cases
* Use attributes rather than custom serializer for arrays
* Add JSON Schema support
* Tests for JSON Schema generation and validation
* Add comments
* Support custom predicates
* Clippy fixes
* Make deserialization/constructor functions pub(crate)
At the middleware we were defining some types that actually are dependant on the
backend no matter how we define them in the middleware.
For example, we were hardcoding the `Hash` and `Value` types and their related
behaviour (eg. `.to_fields()`) to be based on the length of 4 field elements,
but that's not a choice of the middleware, and in fact this is determined by the
backend itself. On the same time, those types and related methods do not belong
to the backend, since conceptually they are part of the middleware reasoning.
The intention of this PR is not to prematurely abstract the library, but to
avoid inconsistencies where a type or parameter is defined in the middleware to
have certain carachteristic and later in the backend it gets used differently.
The idea is that those types and parameters (eg. lengths) have a single source
of truth in the code; and in the case of the "base types" (hash, value, etc)
this is determined by the backend being used under the hood, not by a choice of
the middleware parameters.
The idea with this approach, is that the frontend & middleware should not need
to import the proving library used by the backend (eg. plonky2, plonky3, etc).
As mentioned earlier, the `Hash` and `Value` types are types belonging at the
middleware, and is the middleware who reasons about them, but depending on the
backend being used, the `Hash` and `Value` types will have different sizes. So
it's the backend being used who actually defines their nature under the hood.
For example with a plonky2 backend, these types will have a length of 4 field
elements, whereas with a plonky3 backend they will have a length of 8 field
eleements.
Note that his approach does not introduce new traits or abstract code, just
makes use of rust features to define 'base types' that are being used in the
middleware.
Add initial MerkleTree implementation, which is a wrapper on top of
Plonky2's MerkleTree, with the idea that the future iteration will
replace it by the MerkleTree specified at
https://0xparc.github.io/pod2/merkletree.html .
