[WiP] Huawei API integration - #204
Draft
yatharthranjan wants to merge 22 commits into
Draft
Conversation
…erns Captures the two connector generations (generic kafka-connect-rest-source framework used by Fitbit vs. the newer oura-library + thin Connect glue split used by Oura), the runtime polling/auth/conversion flow, config and Docker/CI setup, and a checklist for adding a new vendor integration such as Huawei following the Oura pattern.
Registers the two new Gradle modules, pins radar-schemas-commons 0.9.0-SNAPSHOT (huawei_schemas branch not yet released) as a separate version-catalog entry so existing modules keep the stable release, and adds the user/UserRepository domain types mirroring oura-library's pattern.
Adds the domain logic for the Huawei Health Kit connector, mirroring oura-library's split of pure-Kotlin logic from Kafka Connect glue: - request/: RestRequest, RequestGenerator, HuaweiRequestGenerator (offset tracking, per-route backoff on 401/403/429/etc), HuaweiResult/HuaweiError. - route/: HuaweiRoute base (OAuth2-authorized GET/POST + time-range chunking), three concrete route kinds covering the Health Kit Data API's endpoints - HuaweiSampleSetRoute (POST sampleSet:polymerize, raw or groupByTime-aggregated), HuaweiHealthRecordRoute (GET healthRecords), HuaweiActivityRecordRoute (GET activityRecords) - and HuaweiRouteFactory, a single registry mapping all ~54 Huawei data types from the radar-huawei-connector schema spec to their endpoint, dataTypeName, and Avro record builder. - converter/: FieldValues (typed accessor for Huawei's field-value sample point format) and generic converters that turn API responses into TopicData for the registered Avro records. Field-value key names are best-effort (Huawei HiHealth Field naming convention); this was verified to compile against a locally-published radar-schemas-commons 0.9.0-SNAPSHOT (huawei_schemas branch) since none of the real snapshot hosts are reachable from this sandbox - flagged in comments for verification against a live API response.
Mirrors kafka-connect-oura-source: HuaweiSourceConnector (periodic user refresh + task reconfiguration), HuaweiSourceTask (round-robin polling across routes, JSON->Avro->SourceRecord conversion via AvroData), KafkaOffsetManager, and HuaweiServiceUserRepository (Ktor-based rest-source-authorizer client with OAuth2 client-credentials auth and cached user/token lookups). HuaweiRestSourceConnectorConfig is written in Kotlin and loop-generates its ~110 per-data-type `huawei.<key>.enabled`/`huawei.<key>.topic` ConfigDef entries from huawei-library's HuaweiRouteFactory.definitions registry, rather than hand-duplicating a static field per topic as Fitbit/Oura do - the same registry also drives which routes HuaweiSourceTask actually builds, so the connector config and the set of polled routes can never drift out of sync. Could not compile this module in this sandbox: packages.confluent.io (needed for kafka-connect-api/kafka-connect-avro-converter) is blocked by the sandbox's egress policy - confirmed this is pre-existing and applies equally to kafka-connect-oura-source, not something introduced here. huawei-library (the part of this change with real logic to verify) does compile cleanly against a locally-published radar-schemas-commons 0.9.0-SNAPSHOT.
Adds docker/source-huawei.properties.template, a radar-huawei-connector docker-compose service, a README section, and registers the kafka-connect-huawei-source image in both CI workflow matrices. Updates ARCHITECTURE.md to document the Huawei module and the route-registry config-generation pattern it introduces for connectors with a large number of data types.
huawei-library tests (verified passing in this sandbox against a locally published radar-schemas-commons 0.9.0-SNAPSHOT): - FieldValuesTest: typed-value array and flattened-object parsing. - HuaweiRouteFactoryTest: builds every one of the ~54 registered route definitions, feeds each a fixture payload shaped for its endpoint kind, and asserts the converter produces exactly one record on the expected topic without throwing - this caught two real bugs during development (a bad endTime field on HuaweiCgmBloodGlucose, and a private extension function shadowing HuaweiDataConverter's). kafka-connect-huawei-source gets a config test mirroring the existing *RestSourceConnectorConfigTest convention, plus checks that enabledTopics() reflects the shared HuaweiRouteFactory.definitions registry and honors per-type huawei.<key>.enabled overrides. Could not run this one in-sandbox (see prior commit: packages.confluent.io is blocked here for every kafka-connect-*-source module, pre-existing and unrelated to this change). Also switches huawei-library's JUnit integration from kotlin-test-junit (JUnit4) to kotlin-test-junit5, since oura-library's copy-pasted dependency block conflicts with the JUnit Platform the radar-kotlin Gradle plugin configures the test task with - previously unnoticed only because no *-library module had tests yet.
`./gradlew check` runs ktlint over every module; auto-formatted the mechanical issues (semicolons, wrapping) and manually split a handful of lines ktlint couldn't auto-correct. huawei-library:check is now fully green (ktlint + all 5 tests) in this sandbox.
ktlint is static analysis and doesn't need dependency resolution to run, so this module's Kotlin sources could be linted even though they can't be compiled in this sandbox. Wraps long ConfigDef.define() calls and doc-string constants, and simplifies the getUserRepository()/ initialize() flow slightly in the process.
Adds yatharthranjan@onsentia.com alongside the existing KCL address in the Huawei Docker image label and CI workflow author fields.
yatharthranjan
marked this pull request as draft
July 14, 2026 13:16
yatharthranjan
changed the title
- Wrap the okhttp3.HttpUrl from getHuaweiUserRepositoryUrl() in URLBuilder(...).build() before passing it as createClient's Ktor Url parameter, matching how the token URL was already handled (and how OuraServiceUserRepository does it). - Throwable.message is nullable; fall back to a default string before passing it to UserNotAuthorizedException's non-null constructor. These only surfaced in real CI since packages.confluent.io is blocked in this sandbox, so kafka-connect-huawei-source couldn't be compiled here - verified via ktlint (which needs no dependency resolution) that the fix introduces no new style issues.
ktlintKotlinScriptCheck also lints .gradle.kts files; wraps the two GitHub Packages credential lines that exceeded 100 chars.
Applies a standard Apache-2.0 copyright header (Copyright 2026 Onsentia) to every .kt/.java file in huawei-library and kafka-connect-huawei-source - the two modules added for the Huawei Health Kit integration. Files that had copied The Hyve's 2018 header from the Oura pattern get it replaced; files with no header get one prepended. Pre-existing Fitbit/Oura files are left untouched since their copyright belongs to their original authors.
Was still copied from Oura's Dockerfile (Copyright 2018 The Hyve); updates to match the Onsentia header applied to the rest of this module's files.
Adds an @author yatharthranjan KDoc/Javadoc tag to the primary class/interface/object of every .kt/.java file in huawei-library and kafka-connect-huawei-source: appended to existing class-level doc comments where present, added as a new minimal doc comment otherwise.
Lets the Huawei connector run against per-user YAML credential files under huawei.user.dir, mirroring Fitbit's YamlUserRepository, so it can be tested locally without a rest-source-authorizer webservice. Adds a docker/huawei-user.yml.template and README instructions for the flow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
isAuthorizedOverride was mapped to the same JSON key ("isAuthorized")
as the isAuthorized computed property, causing Jackson to fail with
"Conflicting getter definitions" when reading user YAML files. Rename
the manual-override field's JSON key to "authorized".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
The 400/401/403 branches previously logged only a hardcoded generic message, discarding the real error body Huawei's API returned. This made it impossible to diagnose the actual cause of a failed request (e.g. wrong client/app, invalid scope) from the logs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
HuaweiHealthRecordRoute sent the health record type identifier under a "subDataTypeName" query parameter, but Huawei's healthRecords API expects it under "dataTypeName" - the API was silently rejecting every health_record_* request with "DataTypeName is null" since it never received the parameter it actually looks for. Renamed the parameter (and the route/factory field) to dataTypeName throughout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
Per Huawei's official "Querying Health Records of a Data Type" spec: - The endpoint is on API version v2, not v1. - The data type query parameter is named "dataType", not "dataTypeName" (and not "subDataTypeName" as it was before that). - startTime/endTime, both in the request and in each returned record, are in nanoseconds since the epoch, not milliseconds. This was causing every health_record_* request to fail with "DataTypeName is null", and would have produced wildly wrong timestamps for any record that did come back. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
The generic error branch (405/409/500/502/503/590, and anything else not explicitly handled) hardcoded its HuaweiGenericError's code to "500" regardless of the actual response status, which was misleading for diagnostics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
Per the official Health Kit REST API reference (Postman "HMS Core" collection, "Querying Sampling Data Statistics of Multiple Days"), the groupByTime-aggregated variant of a data type is obtained by polymerizing the underlying *raw* dataTypeName with groupByTime, not by sending a literal "*.statistics"-suffixed dataTypeName - that suffix is only RADAR-Schemas'/this connector's own label for "the daily-aggregated route", not a real Huawei data type identifier. Sending it verbatim is exactly why Huawei's API had no dataCollector for e.g. "com.huawei.continuous.heart_rate.statistics", "com.huawei.vo2max.statistics", "com.huawei.resting_calories.statistics", etc. - across every single statistics route built via sampleSetDefinition(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
…erize Huawei's live API confirmed the root cause behind most "no default dataCollector found"/"Invalid dataTypeName" errors on statistics routes: sampleSet:polymerize does not support a groupByTime-aggregated query for every data type (confirmed live: "com.huawei.resting_calories does not support the query mode, please use dailyPolymerize API"). Per the official REST API reference for "Querying Sampling Data Statistics of Multiple Days", the day-aggregated variant of a data type must instead go through the dedicated POST /healthkit/v2/sampleSet:dailyPolymerize endpoint, which takes a startDay/endDay (yyyyMMdd) + timeZone request body and returns a differently-shaped, doubly-nested group[].sampleSet[].samplePoints[] response (with group-level times in milliseconds but sample-point times in nanoseconds). Adds HuaweiDailyPolymerizeRoute/HuaweiDailyPolymerizeConverter and routes every "*.statistics" definition through it instead of HuaweiSampleSetRoute, which now only handles raw (non-statistics) data types and has had its now-dead groupByTime support removed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w6G3vGzKQ4ogcLgtWwzun
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial implementation of Huawei HealthKit, powered by claude