feat: AJDA-2445 add command to migrate data-apps orchestrator/flow tasks to data-app-control#101
Open
sykora-ji wants to merge 2 commits into
Open
feat: AJDA-2445 add command to migrate data-apps orchestrator/flow tasks to data-app-control#101sykora-ji wants to merge 2 commits into
sykora-ji wants to merge 2 commits into
Conversation
…sks to data-app-control Adds manage:migrate-data-apps-orchestrator-tasks, which rewrites orchestration and conditional-flow tasks pointing at the legacy keboola.data-apps component so they use keboola.data-app-control instead. Supports a single project or an entire stack via a Manage token, defaults to dry-run, and is idempotent.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an operational CLI command to bulk-migrate orchestrator/flow tasks from the legacy keboola.data-apps component to keboola.data-app-control, with a dry-run default and an opt-in force mode to persist changes via Storage API.
Changes:
- Introduces
manage:migrate-data-apps-orchestrator-taskscommand to iterate selected or all projects, mint temporary Storage tokens, and run the migration (dry-run by default;--forceupdates configs). - Adds
DataAppOrchestratorTaskMigratorto scankeboola.orchestratorandkeboola.flowconfigurations, rewrite eligible tasks, and summarize migrated vs skipped tasks. - Registers the command in
cli.phpand documents usage/behavior inREADME.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/Keboola/Console/Command/MigrateDataAppsOrchestratorTasks.php | New Symfony Console command wiring Manage API + Storage API access, project selection, dry-run/force execution, and summary output. |
| src/Keboola/Console/Command/DataAppOrchestratorTaskMigrator.php | Implements the config/task scanning and rewrite logic for orchestrator/flow configurations. |
| README.md | Documents the new command under “Project manipulation”, including args/options and behavior. |
| cli.php | Registers the new command in the CLI application bootstrap. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Validate the projects argument strictly instead of silently dropping invalid IDs, catch Storage API errors when resolving a configId reference so one bad reference doesn't abort the whole project, cache resolved configIds to avoid repeated lookups, and fix the "Checked N projects" summary to include disabled/errored projects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue
Description
Adds a new
manage:migrate-data-apps-orchestrator-taskscommand to cli-utils that rewrites orchestration/flow tasks referencing the legacykeboola.data-appscomponent so they usekeboola.data-app-controlinstead.DataAppOrchestratorTaskMigrator(new): given a Storage APIComponentsclient, lists all configurations of bothkeboola.orchestrator(legacy orchestrations) andkeboola.flow(conditional flows — verified on a live stack to be a distinct component, not a variant ofkeboola.orchestrator), scans each configuration'stasks[]for tasks pointing atkeboola.data-apps, resolves the app ID from either inlineconfigDataor a referencedconfigId, and rewrites the task'scomponentIdtokeboola.data-app-controlwithparameters.appIdset — while preserving every other field on the task object (type,mode,delay,retry,variableOverrides) unchanged, since thekeboola.flowschema requirestypealongsidecomponentId/mode. Tasks whosekeboola.data-appsparameters carry an unsupportedtaskvalue (anything other than the implicit/app-start/startvariants used to launch an app) are left untouched and reported separately, so unrelated task types (create/delete/terminate/restore/...) are never silently mistransformed. Already-migrated tasks are skipped, making re-runs idempotent.MigrateDataAppsOrchestratorTasks(new command): accepts a Manage API token, stack URL, and either a comma-separated list of project IDs orall. For each target project it mints a short-lived, scoped Storage API token viacreateProjectStorageToken()and runs the migrator against it; failures on one project are caught and logged without aborting the run. Defaults to dry-run (only reports what would change);--force/-fperforms the actualupdateConfigurationcalls with achangeDescription. Prints a summary of projects/configurations/tasks scanned, migrated, and skipped.cli.phpand documented inREADME.mdunder "Project manipulation".Manually verified end-to-end against a test flow on the canary-orion stack (dry-run, force run, idempotent re-run, and schema validity after the rewrite) before opening this PR.
Release Notes
keboola.data-appscomponent to the newerkeboola.data-app-controlcomponent. Orchestrations and conditional flows created before this change still reference the legacy component, so a one-off bulk-migration tool is needed to move them over before the legacy component/UI path can be retired.keboola.data-apps; everything else is left untouched. Tasks with an unrecognizedkeboola.data-appsparameter shape are skipped rather than guessed at, to avoid silently breaking a flow.keboola.data-appstask.cli-utilsDocker image via the existing CI pipeline (tag push). The command itself is then run manually by an engineer, per project or per stack, as needed; it is not invoked automatically.changeDescriptionset by this command), so any specific config can be rolled back to a prior version via Storage API if needed.