Adds nicer errors for Podlang code, using the `annotate_snippets` crate, the same crate used by the Rust compiler to generate contextual errors. This prints a short snippet of the code containing the error within the error message, highlighting the part that needs to be fixed.
It also includes a change to the `load_module` function, changing a `Vec` function argument to a slice.
* First pass at removing batch splitting
* Refactor to separate module loading from request parsing
* Consolidate module functionality
* Tidy up comments
* Use array of modules instead of HashMap
* Formatting
* Use module hashes when importing modules
Resolve https://github.com/0xPARC/pod2/issues/466
Now batches are identified by the root of a merkle tree that contains all the predicates (using sequential indices as keys). This means that the format to identify a custom predicate reference is still a hash + index, but the calculation of the hash is different.
The MainPod circuit now isn't limited by number of batches but instead number of custom predicates; and for each one we verify a merkle proof to verify the batch id.
I've removed a bunch of tests from lang that were testing splitting into multiple batches because there's no longer any need for that. In a future PR we'll remove the code that handles batch splitting.
Each custom predicate needs 148.2 gates (which is very close to my estimate of 142.7 in https://github.com/0xPARC/pod2/issues/466#issuecomment-3823531286 where I actually made a mistake and considered 5 predicates per batch instead of 4 in the previous Params).
I thought it would be nice to have a Predicate for the typed value so that the developer can work with predicates as values comfortably. Then I noticed that hashing a predicate required `Params` which would have been annoying for converting a `TypedValue::Predicate` to `RawValue` and this led to a small refactor over how `Params` work.
We already had some fields in the `Params` struct that determine compatibility between encoded data. They can be seen as determining a kind of ABI compatibility. In general it's better if those parameters don't change so that different circuit configurations can still verify proofs from each other. So I decided to force those parameters to be constant in the code base and not allow the user of our library to change them. Many field element serialization/deserialization functions in our code depended on those parameters, and since now they are constant many functions get rid of the `Params` argument, which simplifies the code. This includes the serialization of a `Predicate` which was required to calculate its hash.
* Multi-batch splitting
* Invoke split predicates by name, passing in full argument list
* Reorder batches to prevent failure of forward references where possible
* Rename APIs for clarity
* Simplify example
* Add more docs
* Review updates
* Remove duplicate code
* Comment topological sort algorithm
Implement support for first order predicates in the backend.
Now a statement template can have a predicate hash or a wildcard.
## predicate <-> predicate hash constraints
To build the custom predicate table we need to calculate the custom predicate batch id, which uses the serialization of the statement templates before normalization. This serialization uses the predicate hash when the template uses a predicate (instead of a wildcard). Then in normalization we recalculate the predicate hash if it was a Batch Self.
This means that the relation between hash and predicate must be checked before and after normalization when the template is not using a wildcard. How this is achieved:
- Before normalization: the constructor of StatementTmplTarget forces that if we keep a predicate, it's hash must be equal to the pred_hash when the template has a predicate (and not a wildcard)
- After normalization: the predicate hash is calculated in the normalization and replaced in the case of the template using a predicate and it being a BatchSelf. If it was a predicate but not batch self, the old value was used which was constrained via the constructor.
See `CircuitBuilder::add_virtual_statement_tmpl` and `normalize_st_tmpl_circuit`
## Wildcard predicate resolution
It is done via `make_predicate_from_template_circuit` and is fairly simple as it's contains similar logic to `make_statement_arg_from_template_circuit` but simpler.
* Basic frontend AST and semantic validation
* Intro statement support
* Simplify validator lifetime
* Fix arity validation
* Lowering and splitting
* Remove legacy processor and use frontend AST by default
* Use builders instead of creating middleware types directly
* Typos/formatting
* Improve error messages when overflowing a batch due to splitting
* Add FromStr implementation for NativePredicate
* Remove 'raw' fields, and switch HashHex representation to byte vector rather than string
* Simpler wrapper types for batch and intro predicate hashes
* Parse secret and public keys to their respective data structures earlier
* More detail around string escape validity
* Simplify native predicate arity handling and move method to NativePredicate impl
* Store hashes using middleware::Hash, and simplify lowering by using pre-parsed values
* Simplify predicate building
* Formatting
* Better error messages/suggestions for cases where predicate splitting fails
* Formatting
* Clippy fix
* Return error if we get a too-large int
- middleware:
- Add `Statement::Intro`
- Add `SignedBy` native predicate and operation. The signature is auxiliary data to the operation
- Rename `PodSigner` to `Signer` with a new API (just for signing `RawValue`)
- Removed `NewEntry` operation. Use `ContainsFromEntries` instead
- Remove `KEY_SIGNER` and `KEY_TYPE` which are no longer used
- Merge `RecursivePod` and `Pod` traits
- Change the `Pod::deserialize_data` method to use `Self` instead of `Box<dyn Pod>`
- Extend `Pod` trait with these methods:
- `is_main`: when the pod is Main, in a (recursive) verification its vk will be checked to exist in the vd_set but not if it's intro pod
- `is_mock`: skip some verifications in the recursive mock MainPod verification
- `verifier_data_hash`
- `pod_id` renamed to `statements_hash`
- AnchoredKeys are now a pair of dictionary root and key
- Entry statements are now defined as Contains with literal arguments
- Operations that take Entries now use Contains statements with literal arguments
- frontend:
- Rename `SignedPod` to `SignedDict` (which now contains the dict, public key and signature, and can still `verify(self)`ed)
- The `SignedDict` keeps the method `get_statement` for convenience but now it returns a `Contains` statement that proves the existence of the key in the dict
- The `MainPodBuilder` automatically inserts a `Contains` statement when an operation is added that uses an entry as argument that was not yet "opened".
- Removed the `literal` methods from the `MainPodBuilder` that were loading literals to anchored keys: that was no longer needed after we introduced literal arguments
- backend
- Only verify inclusion of the verifying key into the vd_set if the pod is MainPod. A pod is not MainPod if the first statement is Intro.
- Reject intro pods that have non-intro statements
- Empty pod now returns an intro statement
- Don't insert a type statement automatically in MainPod and MockMainPod. We get rid of the type entry.
- Implement `SignedBy` operation, which uses the muxed table to store signature verifications
- Rename `PodId` to `statements_hash` or `sts_hash` for short. Now this is only used as a hash of the statements for the circuits public inputs.
- Refactor normalization of `self` statements:
- Before: replace values that contain `SELF` by the given pod_id
- After: place the verifying key hash into the Intro predicates
This PR is a continuation of the work done in #276
- Fix PodType in MainPod (we were using `MockMain` instead of `Main`)
- Update anchored keys in statement template arguments to only support wildcards in the origin and literal keys as the key.
- Update the pest grammar accordingly
- Update the parser accordingly
- Rewrite the eth_dos example in a recursive manner so that we use one recursive pod for every distance increment of 1.
- I've also used the podlang to define the eth_dos custom predicates. Currently all predicates are in a single batch (previously `eth_friend` was in a different batch). With #286 we could define `eth_friend` in a different batch again.
- I was feeling a bit creative and used a format macro to pass `Value`s from rust to the podlang code.
- The eth_dos is now written using literals. This resolves https://github.com/0xPARC/pod2/issues/255
- Remove `StatementArg::WildcardValue` in favor of `StatementArg::Literal`. The `WildcardValue` was just a way to have some kind of typing for values that would be used as arguments in custom predicates. Now that we can have literals in any statement this value can be anything, so I just removed the `WildcardValue` and use `Literal` instead. On the backend it was already the case that both cases were treated the same way (after all, `WildcardValue` and `Literal` were 4 fields in the backend).
- Added a new type for Value: `PodId` so that we can use it for custom predicates that take a pod id to be used in a wildcard
- Add a mock vd_set that is empty for tests that don't use plonky2; this allows running those tests individually without paying for the expensive work of calculating the vd for various circuits.
- rename StatementTmplArg::WildcardValue to StatementTmplArg::Wildcard
* Basic 'use' syntax for importing custom predicates
* Add extra test for unknown batches
* Fix unused import
* Enforce that imports must match number of predicates in a batch
* Initial commit for Podlog language
* Spell-checker thinks that 'lits' is a bad abbreviation for 'literals'
* Enable SetContains/SetNotContains
* Update language based on review feedback
* Typo/comment fix
* Make native predicates case-sensitive
* Enforce max batch size in CustomPredicateBatchBuilder
* Remove some unnecessary checks for things handled by the grammar
* Clean up more unnecessary error-checking
* Typo
* Simplify hex processing
* Replace various errors with unreachable!()
* Translate from big-endian hex string to little-endian RawValue
* Update hex en/decoding functions
