Update description of consistency metric#639
Conversation
Replace the order-count consistency metric description with the new metric combining bid quality and settlement success rate, effective June 30, 2026, and link the announcement forum post. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR updates the Consistency rewards documentation under CIP-85, replacing the order-count metric with a June 30, 2026 metric based on bid quality and settlement success rate, and revises the Solver’s strategy text accordingly. ChangesConsistency Rewards Documentation
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/cow-protocol/reference/core/auctions/rewards.md`:
- Around line 80-84: The bid-quality formula in rewards.md leaves
`orderShare_i(o)` undefined when all bidders have zero surplus, so add the
intended zero-surplus fallback directly in this section. Update the
`orderShare_i(o)` definition to specify what happens when the denominator is
zero, and keep the explanation consistent with the surrounding `Bid quality`
description so readers can see how solver shares are assigned in that edge case.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bc7af1c7-e036-4358-8824-d47312de33b0
📒 Files selected for processing (1)
docs/cow-protocol/reference/core/auctions/rewards.md
| **1. Bid quality.** A solver bids on an order if it submits a solution containing that order. For each executed order, the solvers that bid on the order in the auction in which it was executed share a total weight of one, in proportion to the surplus they proposed. The share of solver $$i$$ for order $$o$$ is | ||
|
|
||
| $$ | ||
| \textrm{orderShare}_i(o) = \frac{\textrm{surplus}_i(o)}{\sum_{j} \textrm{surplus}_j(o)}, | ||
| $$ |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Define the zero-surplus fallback.
orderShare_i(o) divides by the sum of surplus values. If every fair bid on an order has zero surplus, the denominator is zero and the metric is undefined. Please document the intended fallback here.
🧰 Tools
🪛 LanguageTool
[style] ~80-~80: ‘in proportion to’ might be wordy. Consider a shorter alternative.
Context: ...s executed share a total weight of one, in proportion to the surplus they proposed. The share of...
(EN_WORDINESS_PREMIUM_IN_PROPORTION_TO)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/cow-protocol/reference/core/auctions/rewards.md` around lines 80 - 84,
The bid-quality formula in rewards.md leaves `orderShare_i(o)` undefined when
all bidders have zero surplus, so add the intended zero-surplus fallback
directly in this section. Update the `orderShare_i(o)` definition to specify
what happens when the denominator is zero, and keep the explanation consistent
with the surrounding `Bid quality` description so readers can see how solver
shares are assigned in that edge case.
There was a problem hiding this comment.
Agree with the comment; would be good to clarify this
There was a problem hiding this comment.
@harisang, glad this is helpful. Here's a suggested addition to handle the zero-surplus edge case, which you could place right after the orderShare_i(o) formula:
where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. Only solutions that pass the fairness filtering are considered. If every fair bid on the order has zero surplus, the weight of one is instead split equally among all bidding solvers. Since each order distributes a total weight of one, orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation.Feel free to adjust the exact fallback rule (equal split vs. some other convention) to match the actual behavior described in the forum post — I just wanted to make sure the edge case is explicitly called out so readers aren't left wondering what happens when the denominator is zero.
🧠 Learnings used
Learnt from: MarcusWentz
Repo: cowprotocol/docs PR: 586
File: docs/cow-protocol/tutorials/solvers/local_test.md:26-31
Timestamp: 2026-01-21T21:57:26.174Z
Learning: Document that the --native-price-estimators argument for the CoW Protocol autopilot CLI must use the Driver|name|url format (e.g., 'Driver|baseline|http://driver/baseline'), not just 'name|url'. Apply this guidance broadly in docs under the docs directory and include explicit examples to avoid ambiguity.
Learnt from: alfetopito
Repo: cowprotocol/docs PR: 606
File: docs/cow-protocol/tutorials/solvers/onboard.md:124-124
Timestamp: 2026-03-26T09:21:38.746Z
Learning: In the cowprotocol/docs documentation, treat the hostnames `barn.cow.fi` and `dev.cow.fi` as intentionally different environments. When reviewing docs, do NOT flag references where `barn.cow.fi` is replaced with `dev.cow.fi` (including `dev.swap.cow.fi`) as inconsistent—this swap is intentional because `barn.cow.fi` is being decommissioned and replaced by `dev.cow.fi`. Note that `barn.api.cow.fi` remains the backend staging API and should continue to be referred to as `barn`.
There was a problem hiding this comment.
I do not agree with this comment. The current rules prohibit zero surplus. Adding a comment here sounds just confusing to me.
There was a problem hiding this comment.
Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!
| \textrm{orderShare}_i(o) = \frac{\textrm{surplus}_i(o)}{\sum_{j} \textrm{surplus}_j(o)}, | ||
| $$ | ||
|
|
||
| where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. Only solutions that pass the fairness filtering are considered. Since each order distributes a total weight of one, orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation. |
There was a problem hiding this comment.
| where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. Only solutions that pass the fairness filtering are considered. Since each order distributes a total weight of one, orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation. | |
| where $$\textrm{surplus}_i(o)$$ is the largest surplus that any of solver $$i$$'s solutions proposed for order $$o$$ in that auction (the difference between the proposed execution amounts and the order's limit amounts), and the sum runs over all solvers that bid on the order. We stress that only solutions that pass the fairness filtering are considered in the above computations. Since each executed order distributes a total weight of one, orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation. |
There was a problem hiding this comment.
orders with few bidders yield a larger share for each of them, an incentive for solvers to join competitions with little participation.
Not sure i agree with this one; in the sense that if we have 100 bids close to limit price and 2 good bids far from limit price, then the full share will be split among the 2 good bids while everyone else will get essentially zero
There was a problem hiding this comment.
One approach would be to say few competitive bids instead of few bidders. Alternatively, the reasoning can also be removed from the docs.
There was a problem hiding this comment.
I added your suggestion and changed the formulation towards competitive bids (from bidders).
|
|
||
| where the sum runs over all order executions in the period (an order that is executed in several auctions, e.g., a partially fillable order, counts once per execution). | ||
|
|
||
| For example, suppose four solvers bid on an order with proposed surplus of 0.1, 0.08, 0.019, and 0.001 ETH. The total proposed surplus is 0.2 ETH, so the order shares are 0.5, 0.4, 0.095, and 0.005. If the solvers have success rates of 0.8, 0.9, 1.0, and 0.5 in the accounting period, this order contributes 0.4, 0.36, 0.095, and 0.0025, respectively, to their consistency metrics. |
There was a problem hiding this comment.
| For example, suppose four solvers bid on an order with proposed surplus of 0.1, 0.08, 0.019, and 0.001 ETH. The total proposed surplus is 0.2 ETH, so the order shares are 0.5, 0.4, 0.095, and 0.005. If the solvers have success rates of 0.8, 0.9, 1.0, and 0.5 in the accounting period, this order contributes 0.4, 0.36, 0.095, and 0.0025, respectively, to their consistency metrics. | |
| For example, suppose four solvers bid on an order with proposed surplus of 0.1, 0.08, 0.019, and 0.001 ETH. The total proposed surplus is 0.2 ETH, so the order shares are 0.5, 0.4, 0.095, and 0.005, respectively. If the solvers have success rates of 0.8, 0.9, 1.0, and 0.5 in the accounting period, this order contributes 0.4, 0.36, 0.095, and 0.0025, respectively, to their consistency metrics. |
There was a problem hiding this comment.
I added this suggestion.
| @@ -73,7 +73,41 @@ where $$\textrm{protocolFee}_i$$ is the protocol fee (excluding partner fees) ea | |||
|
|
|||
| The consistency budget is distributed at the end of each accounting period according to a consistency metric. The core team has a mandate to adapt this metric when they see fit; every change will be announced in advance on the [CoW Protocol forum](https://forum.cow.fi). | |||
There was a problem hiding this comment.
@fhenneke should we also clarify here that once you have consistency metric, you distribute the budget proportionally to that metric, i.e., each solver i gets
consistency_budget * consistency_metric(i) / sum_{all solvers s} consistency_metric(s)
?
or do you think this is clear enough already?
There was a problem hiding this comment.
I added the formula.
- stress that only fair solutions count towards bid quality - attribute the larger share on uncontested orders to competitive bids rather than the number of bidders - add explicit formula for the proportional distribution of the consistency budget - add "respectively" to the example order shares Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
The consistency metric has changed, as announced in this forum post. This PR updates the documentation accordingly.
Summary by CodeRabbit