-
Notifications
You must be signed in to change notification settings - Fork 2
feat(api/runway): add external merge queue contract #260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,082
−302
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| load("@rules_go//go:def.bzl", "go_library", "go_test") | ||
|
|
||
| go_library( | ||
| name = "messagequeue", | ||
| srcs = [ | ||
| "merge.go", | ||
| "topics.go", | ||
| ], | ||
| importpath = "github.com/uber/submitqueue/api/runway/messagequeue", | ||
| visibility = ["//visibility:public"], | ||
| deps = [ | ||
| "//api/base/messagequeue/protopb", # keep | ||
| "//api/runway/messagequeue/protopb", # keep | ||
| "//platform/consumer", | ||
| "@org_golang_google_protobuf//encoding/protojson", | ||
| "@org_golang_google_protobuf//proto", | ||
| ], | ||
| ) | ||
|
|
||
| go_test( | ||
| name = "messagequeue_test", | ||
| srcs = ["merge_test.go"], | ||
| embed = [":messagequeue"], | ||
| deps = [ | ||
| "//api/base/change/protopb", | ||
| "//api/base/mergestrategy/protopb", | ||
| "@com_github_stretchr_testify//assert", | ||
| "@com_github_stretchr_testify//require", | ||
| "@org_golang_google_protobuf//proto", | ||
| ], | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Runway message queue contract | ||
|
|
||
| The published, language-neutral contract for the merge queues Runway owns. A client — in any language — publishes a merge request and consumes the result without access to Runway's Go types or storage. See [the message queue contract RFC](../../../doc/rfc/messagequeue-contract.md) for the design. | ||
|
|
||
| Payloads are defined as proto3 messages in [`proto/merge.proto`](proto/merge.proto) and generated into [`protopb/`](protopb); the proto is the authority and a non-Go client compiles against it directly. On the wire, payloads are serialized as protobuf JSON (`protojson`), so the queue keeps storing self-describing JSON. The message types are generated, so the Go helpers in this package are just generic `protojson` glue — `Marshal(m)` and `Unmarshal[T](b, m)` — for Go callers; field names stay snake_case (`UseProtoNames`) and enums serialize as their UPPER_SNAKE value name. | ||
|
|
||
| The shared field types `Change` and `MergeStrategy` come from `api/base/change` and `api/base/mergestrategy`, imported by the contract. | ||
|
|
||
| ## Topic keys | ||
|
|
||
| The binding between a topic key and its payload lives in each message's `topic_keys` option (defined in `api/base/messagequeue`); `TopicKeys` reads it back by reflection. A topic key is a stable logical name, not a concrete wire topic — each implementer maps the key to whatever topic name its broker/queue requires. Our Go wiring maps it via `consumer.TopicRegistry`. | ||
|
|
||
| | Message | Direction | Topic keys | | ||
| |---|---|---| | ||
| | `MergeRequest` | client → Runway | `merge-conflict-check`, `merge` | | ||
| | `MergeResult` | Runway → client | `merge-conflict-check-signal`, `merge-signal` | | ||
|
|
||
| One message serves a queue pair because a merge-conflict check is a dry run of a merge: Runway applies the same ordered steps onto the same target branch, and the topic key the request arrives on decides whether it commits the result and reports the produced revisions. A request on `merge-conflict-check` is a dry run; a request on `merge` commits. | ||
|
|
||
| ## Result shape | ||
|
|
||
| `MergeResult.outcome` is an `Outcome` enum (`OUTCOME_UNSPECIFIED`/`SUCCEEDED`/`FAILED`): `SUCCEEDED` means mergeable (check) or merged (commit), `FAILED` a conflict or a failed apply; `reason` carries the explanation when `FAILED`. Per-step detail is in `steps` (request order): each `StepResult.outputs` is the list of `StepOutput`s the step produced on the target branch, **in application order** (the order they were created). A committing merge populates `outputs`; a dry-run check, an already-present change, or a failed step leaves them empty. `StepOutput.id` is the VCS-neutral revision identifier (git SHA, Mercurial hash, Subversion revision, Perforce changelist, …), with room to grow (author, timestamp, …). | ||
|
|
||
| ## Evolution | ||
|
|
||
| Contract changes are additive-only: add new fields; never remove, rename, repurpose, or retype an existing field, and never reuse a field number. protojson ignores unknown fields on read and omits zero-valued fields on write, so a new optional field is backward-compatible in both directions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| // Copyright (c) 2025 Uber Technologies, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| // Package messagequeue holds Runway's external message-queue contract: the wire | ||
| // payloads for the merge queues Runway owns, defined by the proto files in | ||
| // proto/ and generated into protopb/. The proto is the language-neutral | ||
| // authority; the generated Go types in protopb are the binding for Go callers. | ||
| // | ||
| // The message types are generated into protopb; this package adds only generic | ||
| // protojson glue (Marshal/Unmarshal) and the topic-key reflection lookup | ||
| // (TopicKeys), so there is no per-message serialization code. Payloads are | ||
| // serialized as protobuf JSON, not binary, so the MySQL-backed queue keeps | ||
| // storing self-describing JSON. The topic key that carries each payload is | ||
| // declared on the message itself via the topic_keys proto option (see | ||
| // api/base/messagequeue). | ||
| // | ||
| // One contract serves two queue pairs because a merge-conflict check is a dry | ||
| // run of a merge: Runway applies the same ordered steps onto the same target | ||
| // branch, and the only difference is whether it commits the result and reports | ||
| // the revisions it produced. The topic key a request arrives on encodes that choice | ||
| // — the merge-conflict-check pair for a dry run, the merge pair for a | ||
| // committing merge — so MergeRequest and MergeResult are identical on both. | ||
| package messagequeue | ||
|
|
||
| import ( | ||
| "google.golang.org/protobuf/encoding/protojson" | ||
| "google.golang.org/protobuf/proto" | ||
|
|
||
| basemqpb "github.com/uber/submitqueue/api/base/messagequeue/protopb" | ||
| "github.com/uber/submitqueue/api/runway/messagequeue/protopb" | ||
| ) | ||
|
|
||
| // Wire payload types. These alias the generated protobuf bindings so callers | ||
| // reference the contract through this curated package rather than protopb. | ||
| type ( | ||
| // MergeRequest is the payload a client publishes to one of Runway's merge | ||
| // queues: the merge-conflict-check topic for a dry-run check, the merge | ||
| // topic for a committing merge. | ||
| MergeRequest = protopb.MergeRequest | ||
| // MergeStep is one step of an ordered merge: a single set of change(s) | ||
| // applied with a strategy. | ||
| MergeStep = protopb.MergeStep | ||
| // MergeResult is the payload Runway publishes to the corresponding signal | ||
| // queue once a request completes. | ||
| MergeResult = protopb.MergeResult | ||
| // StepResult reports what happened to a single MergeStep. | ||
| StepResult = protopb.StepResult | ||
| // StepOutput is a single revision a merge step produced on the target branch. | ||
| StepOutput = protopb.StepOutput | ||
| ) | ||
|
|
||
| // marshalOpts keeps the JSON field names identical to the proto field names | ||
| // (snake_case), so the wire shape matches the declared contract rather than | ||
| // protojson's default lowerCamelCase. Zero-valued fields are omitted. | ||
| var marshalOpts = protojson.MarshalOptions{UseProtoNames: true} | ||
|
|
||
| // unmarshalOpts tolerates unknown fields so an additive contract change (a new | ||
| // field a producer sends but this consumer does not yet know) is ignored rather | ||
| // than rejected. | ||
| var unmarshalOpts = protojson.UnmarshalOptions{DiscardUnknown: true} | ||
|
|
||
| // Marshal serializes any contract message to protojson bytes for the queue | ||
| // payload, keeping the proto field names (snake_case) on the wire. | ||
| func Marshal(m proto.Message) ([]byte, error) { | ||
| return marshalOpts.Marshal(m) | ||
| } | ||
|
|
||
| // Unmarshal deserializes protojson bytes into the contract message m, tolerating | ||
| // unknown fields so an additive contract change is ignored rather than rejected. | ||
| func Unmarshal[T proto.Message](b []byte, m T) error { | ||
| return unmarshalOpts.Unmarshal(b, m) | ||
| } | ||
|
|
||
| // TopicKeys returns the stable logical topic keys bound to a message via the | ||
| // topic_keys proto option — not concrete wire names; a caller maps each key to | ||
| // its backend's topic name. Returns nil for a message that declares no keys. | ||
| func TopicKeys(m proto.Message) []string { | ||
| opts := m.ProtoReflect().Descriptor().Options() | ||
| if opts == nil { | ||
| return nil | ||
| } | ||
| keys, ok := proto.GetExtension(opts, basemqpb.E_TopicKeys).([]string) | ||
| if !ok { | ||
| return nil | ||
| } | ||
| return keys | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can this be autogenerated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
porbably but it's marshaling using protojson right now