feat(compiler): Add grpc support for Swift - #3776
Conversation
Schemas with services now emit a <Service>Grpc.swift companion beside the Swift model. Each service gets Fory-backed async and NIO providers plus an async client; request and response bytes ride a private GRPCPayload wrapper that serializes through the schema module's Fory instance.
Before writing Swift output, check that no two schemas or services claim the same file path or top-level symbol. A service named after a generated type, or a duplicate service, now fails fast with a clear message instead of emitting Swift that will not compile.
Exercise the Swift companion across the four streaming shapes, keyword-escaped methods, imported request and response types, the default package, both IDL frontends, and the collision preflight, so the emitter and its symbol names stay pinned.
Generate a two-package schema, then swift build and run a SwiftPM package on grpc-swift and local Fory that hosts the generated provider and round-trips all four streaming shapes across the import boundary. Skipped when swift is absent.
Wire Swift into the shared gRPC generation step so the interop schemas emit Swift companions alongside the other targets.
Add a Swift gRPC guide covering dependencies, server and client usage, streaming, and troubleshooting, link it from the Swift guide index, and note the Swift companion in the compiler guide and agent rules.
Break the streaming handler closures across lines so generated companions stay under the swiftlint line-length limit even with long package-qualified names.
Put handler braces on the declaration line, give each async parameter its own aligned line, name the unwrapped stream value, and scope a type_name disable around the package-prefixed symbols so swiftlint reports no violations.
Schemas that share a top-level package component make the model generator emit a duplicate root enum, which the Swift compiler rejects in one module. Pin it with a strict xfail fixture and a docs note pointing at disjoint packages.
The Swift Fory instance is single-threaded, but gRPC drives the marshaller from many threads at once, so sharing one instance races. Build one Fory per thread from the module config and registrations, and fire 200 parallel calls in the fixture to exercise it.
Record that the generated client is async only and that interceptors are not emitted, both because grpc-swift types them on the internal Fory wrapper, and describe the per-thread marshalling.
Name the wire wrapper per service so it is reachable, then drive it from 2000 parallel threads under ThreadSanitizer, asserting no data race and that the per-thread Fory stays wire-compatible with the module's shared instance. Against a shared instance TSan flags a race in the type resolver.
The ThreadSanitizer build adds about three minutes and is environment sensitive, so keep it opt-in for a sanitizer or nightly job while the functional fixtures still run on every swift-capable run.
An rpc whose Swift name is handle, serviceName, channel, or defaultCallOptions would clash with a member the generated provider or client inherits, so fail codegen with a clear message. Cover the reserved names and nested plus imported request and response payloads.
# Conflicts: # compiler/fory_compiler/tests/test_service_codegen.py # docs/compiler/compiler-guide.md # integration_tests/grpc_tests/generate_grpc.py
|
@yash-agarwa-l Please git merge apache/main first to address the conflicts |
|
I already did that, so I'm not sure why it's still showing up. Let me mark it as ready for review and double-check. |
The SwiftPM build-and-run round-trip and the ThreadSanitizer marshaller test need the Swift toolchain, so move them out of the compiler pytest suite (where they only skipped) into a SwiftPM package under integration_tests/grpc_tests/swift. The common-root package limitation stays pinned as a build-free generation check.
yash-agarwa-l
changed the title
Mark the generated message wrapper @unchecked Sendable, a transient single-owner carrier that is serialized synchronously and never shared, so grpc-swift's async APIs accept it under Swift 6 strict concurrency, and make its value immutable. Wrap the service descriptor method list to stay within the line limit. Document that companions compile in Swift 5 language mode until generated models are Sendable.
Point the shared generator at the Swift package's generated sources and run the relocated marshaller round-trip and concurrency tests with `swift test` when the toolchain is present, so they execute outside the JVM-driven suite
|
Hi @chaokunyang, small update and one thing I'd like your call on. The Swift gRPC codegen is done and Swift↔Swift round-trips fine, but Java↔Swift interop fails. After digging in, it looks like a gap in the Fory Swift core serializer, not the gRPC code. When refTracking is on, Swift writes If that sounds right to you, I'd like to do the same for Swift. One thing to flag: it changes tracking-mode output for all Swift structs, not just gRPC. Tracking is opt-in and off by default, so the impact should be small, but it's a core wire-format change, so I wanted to ask first. No gRPC module changes; the contract stays refTracking:true. Two questions:
Since it's a wire-format change, I'm happy to do whatever works best for you here, a separate issue or PR, or just keep it on this PR. Let me know. |
|
When ref tracking is enabled, java/python will still push an id -1 to ref reader, and when inner struct invoke /** Stores {@code object} under an already reserved read ref id. */
@Override
public void setReadRef(int id, Object object) {
if (id >= 0) {
readObjects.set(id, object);
}
}There are some code invoking if (refMode != RefMode.NULL_ONLY || buffer.readByte() != Fory.NULL_FLAG) {
refReader.preserveRefId(-1);
return readContext.readNonRef(fieldInfo.typeInfo);
} |
Apply ruff 0.16 format and check fixes flagged by the Code Style Check job: combine nested validation ifs, parenthesize implicit string concatenations in generated-line lists, annotate class-level tables with ClassVar, narrow Path.resolve fallbacks to OSError/ValueError, and mark generate_grpc.py executable to match its shebang. Generated output is unchanged: compiler tests pass and regenerating the Swift gRPC interop sources produces a zero diff.
A peer with ref tracking disabled globally or per type sends NOT_NULL_VALUE_FLAG where a tracking reader preserves no read ref id, so reads that bind instances through ReadContext#reference popped an outer frame's id or underflowed the ref id stack (Index -1). Reserve a no-op -1 placeholder once the value's serializer is resolved and participates in ref tracking; setReadRef ignores negative ids. Applied per call site instead of centrally in tryPreserveRefId because string, primitive, and compatible-array reads never pop.
374c84b to
de06611
Compare
# Conflicts: # compiler/fory_compiler/generators/swift.py
|
Thanks, I have traced the failure, you were right that this is a reader-side issue and that NOT_NULL_VALUE is legal. Under tracking, The natural place to fix it is centrally, inside Checked on current main, including the string path (round-trips with no stray id) and a tracked compatible struct with polymorphic leaves plus a back-reference. It's proven at the byte level, Swift bytes into Java; the live process-to-process harness is still parked. PTAL if this is fine? Also shall I create a separate PR for the java fix or is it fine here? |
|
@yash-agarwa-l You should only change swift code for this error, you should not change java code. Swift is similiar to rust/c++, both of them can work without changing java code. You could take how fory rust/c++ work as a reference. |
4c1739a to
f7ea499
Compare
…zers Fory.deserialize requires Target == Self, so the generated marshaller and streaming carriers no longer compiled against a bare Serializer bound. Repeat that constraint on the six generic service declarations.
The Swift generated-code page had no gRPC section, and the compiler guide, schema IDL, and FlatBuffers IDL pages omitted Swift from the list of languages that emit gRPC companions.
Add a Swift interop peer (server/client) for all three schemas, both directions, all four streaming modes plus unions, and a Java SwiftGrpcTest that drives it.
d26563b to
044dddc
Compare
|
Thanks @chaokunyang, I have made the revision, One thing to note: this does change Swift's tracked wire output 0xff to 0x00, so an older Swift peer won't match a new one under tracking. |
| class SwiftServiceMixin: | ||
| """Generates Swift gRPC service companions backed by Fory serialization.""" | ||
|
|
||
| def generate_services(self) -> list[GeneratedFile]: |
There was a problem hiding this comment.
Keep the compiler importable on Python 3.8 and 3.9
fory-compiler still declares Python 3.8+ support, but this module has no from __future__ import annotations; Python 3.8 evaluates list[GeneratedFile] during the eager SwiftGenerator import and raises TypeError. cli.py has the same problem, and its list[Path] | None annotations also break Python 3.9. Please retain compatible annotation syntax or postpone annotations in every affected module, and add Python 3.8/3.9 CLI-import coverage.
| # The Swift Fory instance is single-threaded, so keep one per thread. | ||
| return [ | ||
| "private enum ForyRuntime {", | ||
| f' private static let key = "org.apache.fory.grpc.{module}"', |
There was a problem hiding this comment.
Make the thread-local key unique across Swift modules
This key contains only the generated textual type path, so every default-package schema uses org.apache.fory.grpc.ForyModule. If two generated schemas live in separate Swift modules, the second wrapper retrieves the first module's Fory, skips its own registrations, and serializes with the wrong resolver. Please include the runtime Swift module/type identity in the key and cover two generated targets round-tripping sequentially on the same thread.
| # enum, so there are no collidable top-level names. Flatten and default | ||
| # packages put each type and the module helper at file scope. | ||
| components = self._package_components_for_schema(self.schema) | ||
| if self.get_namespace_style() == "enum" and components: |
There was a problem hiding this comment.
Include the package namespace in collision preflight
Enum-style packages still emit a top-level namespace enum. For example, demo.shared and demo.greeter each emit public enum Demo, but this branch records no symbol, so preflight succeeds and Swift compilation later fails with invalid redeclaration. Please record the actual top-level namespace owner, or change generation so duplicate declarations are avoided.
| components = self._package_components_for_schema(self.schema) | ||
| if self.get_namespace_style() == "enum" and components: | ||
| return set() | ||
| symbols: set[str] = set() |
There was a problem hiding this comment.
Preserve duplicate symbol owners during preflight
Returning a set loses duplicate declarations produced within one schema. For a default-package schema declaring message ForyModule {}, the model and generated helper both become top-level ForyModule, but preflight sees only one symbol. Flattened helper collisions and normalized-name collisions have the same problem. Please preserve each declaration and its Swift scope instead of deduplicating before ownership validation.
| base = self._service_symbol(service) | ||
| modes = {streaming_mode(m) for m in service.methods} | ||
| symbols = [ | ||
| f"{base}Message", |
There was a problem hiding this comment.
Do not reserve a marshaller symbol for empty services
<Base>Message is registered unconditionally, but the marshaller is emitted only when the service has methods. Consequently message GreeterMessage {}; service Greeter {} is rejected even though the service file contains no GreeterMessage wrapper. Please register this symbol only for non-empty services and add an empty-service preflight case.
| return symbols | ||
|
|
||
| def _swift_grpc_method_name(self, method: RpcMethod) -> str: | ||
| return self.safe_member_name(method.name) |
There was a problem hiding this comment.
Reject or rename an underscore-only RPC method
FDL accepts an RPC named _, but the generated member remains _, producing declarations and references such as static let _, func _, and Methods._ that Swift cannot use as identifiers. Please generate a referencable name or reject this RPC name during preflight.
| "// loop, so the payload must be Sendable. The carrier itself only stores that", | ||
| "// payload, so @unchecked covers the wrapper while Value carries the guarantee.", | ||
| ( | ||
| f"struct {base}Message<Value: Serializer & Sendable>: GRPCPayload," |
There was a problem hiding this comment.
Align the Sendable constraint with generated model types
This constraint applies to every request and response wrapper, but generated public structs, enums, unions, and classes do not conform to Sendable. Swift 6 therefore rejects even unary providers and clients when Message<Request/Response> is instantiated, not only client-streaming and bidi as the guide states. Please add a safe Sendable strategy plus Swift 6 build coverage, or state that the entire companion currently requires Swift 5 mode.
| ```swift | ||
| // Package.swift | ||
| dependencies: [ | ||
| .package(url: "https://github.com/apache/fory.git", from: "1.2.0"), |
There was a problem hiding this comment.
Require a Fory version that contains this runtime support
from: "1.2.0" allows versions whose Swift Serializer lacks Target, and even 1.6.1 lacks this PR's tracked-value root ref-slot fix required for Java/Swift interoperability. Please use the first release containing both the compiler and runtime changes, or follow the existing $version convention.
| final class RoundTripTests: XCTestCase { | ||
| func testInProcessAllStreamingModes() async throws { | ||
| let group = MultiThreadedEventLoopGroup(numberOfThreads: 1) | ||
| defer { try? group.syncShutdownGracefully() } |
There was a problem hiding this comment.
Use async cleanup in the async test
The current Java/Swift CI job reports async-context warnings for this blocking shutdown, lines 118/124, and main.swift:456; Swift 6 upgrades at least one to an error. Please use awaited get() or structured async lifecycle cleanup while preserving failure-path cleanup. Swift changes are required to compile without warnings.
| - name: Run Swift gRPC package tests | ||
| run: | | ||
| cd integration_tests/grpc_tests/swift/interop | ||
| swift test |
There was a problem hiding this comment.
Run the concurrency regression under ThreadSanitizer
This runs only plain swift test. The sole --sanitize=thread command is behind FORY_SWIFT_TSAN=1 in another script that this workflow never invokes, so the test and PR description's TSan safety claim is not exercised by CI. Please add a sanitized invocation, preferably filtered to the marshaller concurrency test.
| fi | ||
| # Swift toolchain tests (generated marshaller round-trip and concurrency). These | ||
| # need the Swift toolchain rather than the JVM, so they run in their own package. | ||
| if command -v swift >/dev/null 2>&1 && [ -d "${SCRIPT_DIR}/swift/interop" ]; then |
There was a problem hiding this comment.
Honor the selected test classes for Swift work
This block ignores TEST_CLASSES, so ./run_tests.sh GoGrpcTest still resolves and runs the Swift package whenever Swift is installed. Conversely, the default includes SwiftGrpcTest and later invokes swift build without checking that Swift exists. Please gate all Swift work on both the selected test class and tool availability, or make it an explicit opt-in.
2 participants
Why?
Swift users can generate Fory model types today, but schemas that define services
do not produce Swift gRPC companions. This leaves Swift out of the existing
--grpcworkflow used by the other supported service-generation targets.
What does this PR do?
metadata descriptors, an
EventLoopFutureprovider, an async/await provider, anasync client, and Fory-backed request/response stream adapters, targeting grpc-swift 1.x.
internal
GRPCPayloadwrapper. Because the SwiftForyruntime is single-threaded,the wrapper builds one
Foryper thread from the schema module's own configurationand registrations, so concurrent RPCs are race-free (verified under ThreadSanitizer).
EventLoopFutureclient and interceptor hooks areomitted because their generated types would expose the internal wrapper.
collisions, and reserves inherited provider/client member names (
handle,serviceName,channel,defaultCallOptions) so a clashing rpc fails codegenwith a clear message.
identifier escaping, imported and nested service types, the default package, the
protobuf and FlatBuffers frontends, collision handling, a SwiftPM build-and-run
fixture, and a marshaller concurrency test under ThreadSanitizer (gated behind
FORY_SWIFT_TSAN).shared-top-level-package limitation, troubleshooting, and compiler guide updates.
Draft: Java<->Swift cross-language interop tests (all four modes, all three IDL
frontends) are in progress and will be added before this PR is marked ready for review.
Related issues
#3266
#3370
AI Contribution Checklist
yesyes, I included a completed AI Contribution Checklist in this PR description and the requiredAI Usage Disclosure.yes, my PR description includes the requiredai_reviewsummary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.Does this PR introduce any user-facing change?
foryc --swift_out=... --grpcis used.change the Fory binary protocol.
Benchmark
Not applicable.