I thought it would be nice to have a Predicate for the typed value so that the developer can work with predicates as values comfortably. Then I noticed that hashing a predicate required `Params` which would have been annoying for converting a `TypedValue::Predicate` to `RawValue` and this led to a small refactor over how `Params` work.
We already had some fields in the `Params` struct that determine compatibility between encoded data. They can be seen as determining a kind of ABI compatibility. In general it's better if those parameters don't change so that different circuit configurations can still verify proofs from each other. So I decided to force those parameters to be constant in the code base and not allow the user of our library to change them. Many field element serialization/deserialization functions in our code depended on those parameters, and since now they are constant many functions get rid of the `Params` argument, which simplifies the code. This includes the serialization of a `Predicate` which was required to calculate its hash.
Implement support for first order predicates in the backend.
Now a statement template can have a predicate hash or a wildcard.
## predicate <-> predicate hash constraints
To build the custom predicate table we need to calculate the custom predicate batch id, which uses the serialization of the statement templates before normalization. This serialization uses the predicate hash when the template uses a predicate (instead of a wildcard). Then in normalization we recalculate the predicate hash if it was a Batch Self.
This means that the relation between hash and predicate must be checked before and after normalization when the template is not using a wildcard. How this is achieved:
- Before normalization: the constructor of StatementTmplTarget forces that if we keep a predicate, it's hash must be equal to the pred_hash when the template has a predicate (and not a wildcard)
- After normalization: the predicate hash is calculated in the normalization and replaced in the case of the template using a predicate and it being a BatchSelf. If it was a predicate but not batch self, the old value was used which was constrained via the constructor.
See `CircuitBuilder::add_virtual_statement_tmpl` and `normalize_st_tmpl_circuit`
## Wildcard predicate resolution
It is done via `make_predicate_from_template_circuit` and is fairly simple as it's contains similar logic to `make_statement_arg_from_template_circuit` but simpler.
Resolve#448
Previously a predicate was 6 elements. Now it grows to 8 elements; and the hash is 4 elements.
Some parts of the circuit require only require equality checks with the predicate: that works with the predicate hash. Other parts require inspecting or working with particular elements in the predicate, those need the preimage of the predicate hash.
Both `StatementTarget` and `StatementTmplTarget` have been updated to include the predicate hash and optionally the predicate. When the predicate is included, constraints are automatically generated for `pred_hash = hash(pred)`. We only include the predicate when needed.
- Update formula in `estimate_verif_num_gates` after the update in the recursive verification from https://github.com/0xPARC/pod2/pull/397
- Update the parameters to get better utilization of 2^16 rows
- Update metrics report to be more compact
- middleware:
- Add `Statement::Intro`
- Add `SignedBy` native predicate and operation. The signature is auxiliary data to the operation
- Rename `PodSigner` to `Signer` with a new API (just for signing `RawValue`)
- Removed `NewEntry` operation. Use `ContainsFromEntries` instead
- Remove `KEY_SIGNER` and `KEY_TYPE` which are no longer used
- Merge `RecursivePod` and `Pod` traits
- Change the `Pod::deserialize_data` method to use `Self` instead of `Box<dyn Pod>`
- Extend `Pod` trait with these methods:
- `is_main`: when the pod is Main, in a (recursive) verification its vk will be checked to exist in the vd_set but not if it's intro pod
- `is_mock`: skip some verifications in the recursive mock MainPod verification
- `verifier_data_hash`
- `pod_id` renamed to `statements_hash`
- AnchoredKeys are now a pair of dictionary root and key
- Entry statements are now defined as Contains with literal arguments
- Operations that take Entries now use Contains statements with literal arguments
- frontend:
- Rename `SignedPod` to `SignedDict` (which now contains the dict, public key and signature, and can still `verify(self)`ed)
- The `SignedDict` keeps the method `get_statement` for convenience but now it returns a `Contains` statement that proves the existence of the key in the dict
- The `MainPodBuilder` automatically inserts a `Contains` statement when an operation is added that uses an entry as argument that was not yet "opened".
- Removed the `literal` methods from the `MainPodBuilder` that were loading literals to anchored keys: that was no longer needed after we introduced literal arguments
- backend
- Only verify inclusion of the verifying key into the vd_set if the pod is MainPod. A pod is not MainPod if the first statement is Intro.
- Reject intro pods that have non-intro statements
- Empty pod now returns an intro statement
- Don't insert a type statement automatically in MainPod and MockMainPod. We get rid of the type entry.
- Implement `SignedBy` operation, which uses the muxed table to store signature verifications
- Rename `PodId` to `statements_hash` or `sts_hash` for short. Now this is only used as a hash of the statements for the circuits public inputs.
- Refactor normalization of `self` statements:
- Before: replace values that contain `SELF` by the given pod_id
- After: place the verifying key hash into the Intro predicates
Changed the middleware to only allow comparison of integers and to
use the implementation of Ord for i64. This matches the backend
behavior.
Also fixed a separate bug where LtEqFromEntries was producing a
NotEquals statement.
* Add container update ops
* Update src/middleware/operation.rs
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
* Update src/backends/plonky2/mainpod/mod.rs
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
* Code review
---------
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
- Add a function to calculate the hash of the `CommonCircuitData`. The hash uniquely identify the `CommonCircuitData` used for a circuit/proof. Serializing the struct is not enough because the polynomial identities of the custom gates are not serialized (only their parameters are); so I made a function to extract "fingerprints" of the custom gates by evaluating them over a predefined list of uniform values, and then doing a random linear combination over the results.
- Store the full verifier only circuit data of a proof in the MainPod so that we can verify pods from old circuits in new circuits and code
- Store the hash of the `CommonCircuitData` in the MainPod so that we can reject verifying old pods that use a different `CommonCircuitData` than the current one. This has two goals
- If the `CommonCircuitData` changes it's very likely that the verification will fail, but it will be hard to debug. Doing this early check helps identify the origin of the verification failure as early as possible
- There's a chance that the verification could succeed when the `CommonCircuitData` changes, and that could be dangerous because the verification will be doing different checks than the ones intended for the original proof, so we may be skipping some constraints that could lead to exploiting the system. For this reason, whenever the common circuit data hash changes, all previous verifying keys should be discarded (that is, not included in the VDSet)
- The fingerprint only has ~64 bits and the "random evaluation point" is fixed. The assumption is that the pod developers are not malicious and are not changing the gates such that different gates give the same fingerprint. With this assumption, I find it reasonable to assume that with high probability if a gate changes, its fingerprint changes as well.
- Add a github action that updates a wiki page with a table that contains: date, commit, params hash (with a link to the actual params), verifier data only circuit data hash and common circuit data hash. This will make it easy to track when the common circuit data changes as well as track the verifier data corresponding to various versions (identified by commit)
- The edited page is this one https://github.com/0xPARC/pod2/wiki/MainPod-circuit-info
Resolve https://github.com/0xPARC/pod2/issues/386
Summary of breaking changes:
- The `RecursivePod` trait has a new method `common_hash` that needs to return the result of `hash_common_data` on the `CommonCircuitData` that the circuit uses.
- Extend the `Flattenable` trait to include a `size` method that returns the number of `Target`s the type requires. This is used in the table to figure out the max length of an array that must fit all entry types.
- Move the circuit methods to precalculate hash states and do hashes started from a precomputed state to a new module
- Introduce `MuxTableTarget` which allows easy multiplexing of tables where each sub-table may have entries of different lengths. The table access is done via hashing + unhashing automatically (via use of a generator)
- Use the `MuxTableTarget` to access merkle tree claims and custom predicate verification, which where previously in different tables and accessed with independent random accesses each
- Move the public key derivation for the PublicKeyOf operation check to the same multiplexed table. Now we can choose how many of those operations a circuit supports.
Resolve https://github.com/0xPARC/pod2/issues/357
Resolve https://github.com/0xPARC/pod2/issues/361
Support arrays up to 256 elements (hardcoded maximum just to avoid abuse) by combining multiple random_accesses.
The index is now split into low and high parts. It's a bit more inconvenient than using a single Target but this allows avoiding bit decomposition.
* wrote some initial code
* added way to input private key into circuit
* TypedValue::SecretKey hashed as 10 32-bit limbs
* Check PublicKeyOf in Frontend and Middleware
* Diff review
* PR review
* Finish utest
* Fix bounds check
* added giving secret key witness to circuit
* Test & doc improvements
* added private key comparison to circuit and added test cases
* cargo fmt
* Add frontend tests for PublicKeyOf
* Add public_key_of and hash_of to op! macro
* Add ownership check to ticket example
* Group order checking in tests
* More negative test cases at circuit level
* Cleanups after self review
* clippy fixes
* Fixes after merge. Temporarily remove plonky2 commit hash
* Add a nullifier to the ticket test example
* Test PublicKeyOf with a real prover (not mock)
* plonky-u32 dependency
* feat: optimize operation checks
Skip the circuits that verify operation checks other than None, Copy or
NewEntry for the public statements. This works because public
statements are created by copying private statements, so we never use
the other operation checks in those slots.
---------
Co-authored-by: Andrew Twyman <artwyman@gmail.com>
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
- Bump rust version to `nightly-2025-07-02` because some of the nightly features we were using have been stabilized.
- Introduce feature `disk_cache` which enables caching to disk. Each time an artifact is retrieved from the cache it will be read and deserialized. On a cache miss the artifact will be created, serialized and stored to disk.
- Introduce feature `mem_cache` which enables caching to memory. All cached artifacts are kept in memory after they are created. The mem cache implementation avoids cloning of artifacts by extending their lifetime to `'static`. This is `unsafe` code, but I argue that this usage is safe.
- Add a `build.rs`
- When the feature `disk_cache` is enabled, the `build.rs` will inject env variables to the process with the git commit information, which is used to index the cached artifacts
- Replace all previous cached artifacts from `LazyStatic` methods that call the cache API
- Derive `Serialize, Deserialize` for all `*Target` types so that they can be serialized for caching to disk
- Add finer level of caching: now we cache the `CircuitData` and `VerifierData` independently. The reason for this is that `CircuitData` is a very big artifact which is not needed for verification. So by only accessing `VerifierData` in verification we don't pay a big overhead for reading from disk and deserializing
- Add missing artifacts to the cache: like the `CircuitData` for the `MainPod` indexed by `Params`
- Add helper types to serialize and deserialize `CircuitData`, `CommonData` and `VerifierData` with the set of gates and generators used in the recursive MainPod circuit
- Tweak the ids of our custom gates so that they remain unique when their generic parameters change
- Bugfix: several tests were using the standard `vd_set` but were using MainPod circuits with non-default parameters. This was working before because there was a bug: the MainPod circuit was reporting that the used verifier data was the standard one instead of picking the one corresponding to it's own Params.
Summary of breaking changes:
- One and only one of the features `mem_cache` or `disk_cache` need to be enabled. By default it's `mem_cache`
- To enable the `disk_cache` you need to disable the default features like this: `--no-default-features --features=backend_plonky2,zk,disk_cache`
- Removed `DEFAULT_PARAMS`, instead use `Params::default()`
- Removed `STANDARD_REC_MAIN_POD_CIRCUIT_DATA`, instead use `cache_get_standard_rec_main_pod_common_circuit_data`
- The library is now using `nightly-2025-07-02`. Some rust language features are unstable in previous versions.
The PodSigner trait was taking `&mut self` in the `sign` method, but the
signer doesn't need mutation in the Shcnorr implementation. Remove the
`mut`.
Previously the PodProver trait was also taking `&mut self` in the
`prove` method, and we had many tests creating a `mut Prover/mut
MockProver`. Remove all those `mut`.
Breaking change: `PodSigner` trait method `sign` replaces `&mut self` by
`&self`
* Compress EC subgroup points before serialising
* serialize and display point in base58
* Use Display for Points
---------
Co-authored-by: Ahmad <root@ahmadafuni.com>
- Update the `RecursivePod` trait to return `vd_set` instead of `vds_root`
- A native verifier requires the entire set to reason about the circuits that have been used in the recursive tree
- Implement serialization/deserialization for `VDSet`
- Remove `DynError` and use `BackendError` instead for middleware functions that wrap or define trait functions implemented in the backend. This is based on the fact that we will only have a single backend enabled at a time, so there's no need for a `dyn Error`
- Move the implementations of `_verify` functions to `verify` and similarly for `_prove`
- Complete the verification of a MockMainPod: the verification of input pods was missing. The inclusion of these input pods in the serialization was also missing. With this change a `MockMainPod` will grow after each recursion. This was expected from the design but was not the case because of the missing recursive native verification implementation.
* apply feedback from @arnaucube
This PR is a continuation of the work done in #276
- Fix PodType in MainPod (we were using `MockMain` instead of `Main`)
- Update anchored keys in statement template arguments to only support wildcards in the origin and literal keys as the key.
- Update the pest grammar accordingly
- Update the parser accordingly
- Rewrite the eth_dos example in a recursive manner so that we use one recursive pod for every distance increment of 1.
- I've also used the podlang to define the eth_dos custom predicates. Currently all predicates are in a single batch (previously `eth_friend` was in a different batch). With #286 we could define `eth_friend` in a different batch again.
- I was feeling a bit creative and used a format macro to pass `Value`s from rust to the podlang code.
- The eth_dos is now written using literals. This resolves https://github.com/0xPARC/pod2/issues/255
- Remove `StatementArg::WildcardValue` in favor of `StatementArg::Literal`. The `WildcardValue` was just a way to have some kind of typing for values that would be used as arguments in custom predicates. Now that we can have literals in any statement this value can be anything, so I just removed the `WildcardValue` and use `Literal` instead. On the backend it was already the case that both cases were treated the same way (after all, `WildcardValue` and `Literal` were 4 fields in the backend).
- Added a new type for Value: `PodId` so that we can use it for custom predicates that take a pod id to be used in a wildcard
- Add a mock vd_set that is empty for tests that don't use plonky2; this allows running those tests individually without paying for the expensive work of calculating the vd for various circuits.
- rename StatementTmplArg::WildcardValue to StatementTmplArg::Wildcard
* containers: add method to create new {Dict,Set,Array} with custom max_depth
* add vds_tree computation, update tree circuit interface
* add VDTree struct, add DEFAULT_VD_TREE, integrate it with MainPod,EmptyPod,frontend,etc.
* adapt frontend/serialization tests to new containers field (max_depth)
* adapt interfaces to allow using custom vd_tree in frontend & backend constructors
* rename VDTree to VDSet (and derivate namings too)
* containers 'new' always with param 'max_depth', use params.max_depth_mt_containers instead of the global constant MAX_DEPTH
* adapt after rebasing the branch to main latest changes
* apply review suggestions from @ed255
* use emptypod vd_mt_proofs (using vd_set as circuit input), merge the two existing set_targets methods of MainPodVerifyTarget
* document VDSet & vds_root
* calculate MainPod id in a dynamic-friendly way
The MainPod id is now calculated with front padding and a fixed size
independent of max_public_statements so that introduction gadgets can be
verified by a MainPod while paying only for the number of statements
they use. This is because with front padding of none-statements we can
precompute the poseidon state corresponding to absorbing all the padding
statements and only pay constraints for the non-padding statements.
The id is calculated as follows:
`id = hash(serialize(reverse(statements || none-statements)))`
* add time feature and disable timing by default
* apply suggestions from @arnaucube
* link issues in todos
* calculate MainPod id in a dynamic-friendly way
The MainPod id is now calculated with front padding and a fixed size
independent of max_public_statements so that introduction gadgets can be
verified by a MainPod while paying only for the number of statements
they use. This is because with front padding of none-statements we can
precompute the poseidon state corresponding to absorbing all the padding
statements and only pay constraints for the non-padding statements.
The id is calculated as follows:
`id = hash(serialize(reverse(statements || none-statements)))`
* fix test
* WIP
* WIP
* Working serialization for both Mock and Plonky2 versions of Signed and Main Pods
* Restore useful comment about serialized_proof()
* Use plonky2 serialization for signatures and proofs
* Add schema renames for Serialized SignedPod/MainPod types
* Break out utility function for generating common circuit data
* Review feedback fixes
* migrate from anyhow to thiserror (#190). pending polish error msgs
* Add backtrace and compartmentalize errors
- Include backtraces in the errors we generate. To get this we can't
just return a literal enum, because the backtrace requires a call.
- Related to the previous point: add methods to create errors so
we can include the backtrace conveniently without changing too much
the syntax. So instead of `Err(Error::KeyNotFound(key))` (literal
enum) it will be `Err(Error::key_not_found(key))` (method call)
- Each error should be local to its scope, and each scope should
only return its own error.
- The merkle tree should return `TreeError` and not Error
- The middleware should return `MiddlewareError` and not Error
- With a global Error we can't easily include backend/frontend types in
the error fields, so declare a `BackendError` and a `FrontendError`
and follow the pattern from the previous point
- The Pod traits should be able to return backend errors and will be
used in the frontend; for that we change them to use trait object
Error: `dyn std::error::Error`
* fix error
* apply suggestions from @arnaucube
* rename XError and XResult to Error and Result
* reorg signature
* make frontend custom error more ergonomic
* remove unnecessary feature
---------
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
* Serialization tests now pass again
* Tidy up and test more edge-cases
* Use attributes rather than custom serializer for arrays
* Add JSON Schema support
* Tests for JSON Schema generation and validation
* Add comments
* Support custom predicates
* Clippy fixes
* Make deserialization/constructor functions pub(crate)
* unify fe/be NativeOp and NativePred
* remove Origin in favour of PodId
* Combine string and hash in Key
* use middleware::AnchoredKey in frontend
* merge frontend/middleware types
* refactor custom predicates
* clean up a bit
* fix middleware custom tests
* clean up
* clean up 2
* add acronyms in typos list
* implement circuit to verify signature (proof-based signature), ie. a 1-level recursion verification
* as agreed in the call, rename Gate -> Gadget when it's not a 'gate'
* make SignatureVerifyGadget conditional on the selector input
* small naming polish
* sigverifygadget: add s computation in-circuit, connect pk,msg,s to internalproof's public_inputs
* optimize signature verify
---------
Co-authored-by: Eduard S. <eduardsanou@posteo.net>
