-
Notifications
You must be signed in to change notification settings - Fork 56
feat: support bulk-export command #261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ohadedry
merged 21 commits into
microsoft:main
from
ohadedry:support-bulk-export-command
Jul 23, 2026
Merged
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
1aec777
feat: support bulk-export command
ohadedry 1c2c809
Merge branch 'main' into support-bulk-export-command
ohadedry ebfd5f6
Fix lint issue, return dict
ohadedry d760057
Fix changie file
ohadedry 50cd1b3
Review fixes
ohadedry 0a15d1d
fix tests, remove test which is no longer required
ohadedry a107136
Add more tests and review fixes
ohadedry 87d103b
fix doc and remove redundant check
ohadedry 4ca5a7a
review fixes
ohadedry 2292d22
fix bulk-export output
ohadedry 76046f2
review fix and supported items update
ohadedry 193cb89
remove unreachable code and add tests
ohadedry 6c9c36c
fix test typo
ohadedry cbf5fe5
improve logger processing and remove unused error
ohadedry bcfb021
improve logger text
ohadedry cf5bdad
review fixes
ohadedry b11d6ae
fix tests review
ohadedry 2aae3f9
Refactor logger response processing and update doc
ohadedry 3e96065
Merge branch 'main' into support-bulk-export-command
ohadedry 4f975e4
minor doc fix
ohadedry 3e9988a
fix test review
ohadedry File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| kind: added | ||
| body: Add new 'bulk-export' command for exporting workspace or folder items | ||
| time: 2026-07-06T09:11:26.916814199Z | ||
| custom: | ||
| Author: ohadedry | ||
| AuthorLink: https://github.com/ohadedry |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # `bulk-export` Command | ||
|
|
||
| Export a workspace or folder in bulk while preserving folder structure and item bindings. | ||
|
|
||
| The `bulk-export` command exports all supported items from a workspace or folder in a single bulk API request, preserving the folder hierarchy and item bindings (logical IDs) in the exported definitions. | ||
|
|
||
| !!! warning "When exporting items, the item definitions are exported without their sensitivity labels" | ||
|
|
||
| !!! note "Permissions" | ||
| - **Workspace export**: Only items that the caller has both read and write permissions for are exported. | ||
| - **Folder export**: The caller must have read and write permissions for all folder items. | ||
|
ohadedry marked this conversation as resolved.
|
||
|
|
||
| !!! note "Beta API" | ||
| This command relies on the Fabric [bulk export item definitions API](https://learn.microsoft.com/en-us/rest/api/fabric/core/items/bulk-export-item-definitions(beta)), which is currently in beta. | ||
|
|
||
| !!! note "Differences from `export`" | ||
| - **Folder structure**: `bulk-export` preserves the workspace folder hierarchy in the output. `export` exports items flat. | ||
| - **Item bindings**: `bulk-export` preserves item logical IDs, enabling round-trip import/update. `export` uses nil UUIDs. | ||
| - **Format**: `bulk-export` does not support the `--format` flag. Items are exported in their default format only. | ||
| - **Target scope**: `bulk-export` operates on workspaces and folders. `export` operates on individual items or workspaces. | ||
|
|
||
| **Supported Targets:** | ||
|
|
||
| - `.Workspace` — exports all supported items in the workspace | ||
| - `.Folder` — exports all supported items in the folder and its sub-folders | ||
|
|
||
| **Supported Item Types:** | ||
|
|
||
| `CopyJob`, `CosmosDBDatabase`, `Dataflow`, `DataPipeline`, `DigitalTwinBuilder`, `DigitalTwinBuilderFlow`, `Environment`, `Eventhouse`, `Eventstream`, `GraphQLApi`, `GraphQuerySet`, `KQLDashboard`, `KQLDatabase`, `KQLQueryset`, `Lakehouse`, `Map`, `MirroredDatabase`, `MLExperiment`, `MLModel`, `MountedDataFactory`, `Notebook`, `Reflex`, `Report`, `SemanticModel`, `SparkJobDefinition`, `SQLDatabase`, `UserDataFunction`, `VariableLibrary` | ||
|
|
||
| Unsupported item types are automatically skipped and reported in the output summary. | ||
|
|
||
| **Usage:** | ||
|
|
||
| ``` | ||
| fab bulk-export <path> -o <output_path> --recursive [-f] | ||
|
ohadedry marked this conversation as resolved.
|
||
| ``` | ||
|
|
||
| **Parameters:** | ||
|
|
||
| | Parameter | Description | | ||
| |-----------|-------------| | ||
| | `<path>` | Path to the workspace or folder to export. | | ||
| | `-o, --output <output_path>` | Output directory path. Required. | | ||
| | `--recursive` | Recursively export folder contents. Required for workspace and folder targets. | | ||
| | `-f, --force` | Skip confirmation prompts. Exports without sensitivity label confirmation and proceeds when the output folder is not empty. Optional. | | ||
|
|
||
| **Examples:** | ||
|
|
||
| ```bash | ||
| # Bulk-export an entire workspace to a local directory | ||
| fab bulk-export ws1.Workspace -o /tmp --recursive --force | ||
|
|
||
| # Bulk-export a specific folder | ||
| fab bulk-export ws1.Workspace/folder1.Folder -o /tmp --recursive --force | ||
| ``` | ||
|
|
||
| **Output:** | ||
|
|
||
| On success, the command prints a summary with the number of exported and skipped items: | ||
|
|
||
| ``` | ||
| Exported 15 items to '/tmp'. Skipped 1 items due to unsupported item types: Dashboard (1) | ||
| ``` | ||
|
|
||
| When using `--output_format json`, the output includes structured data: | ||
|
|
||
| ```json | ||
| { | ||
| "exported": 15, | ||
| "exported_types": { | ||
| "SemanticModel": 5, | ||
| "Report": 4, | ||
| "Notebook": 3, | ||
| "DataPipeline": 3 | ||
| }, | ||
| "skipped": 1, | ||
| "skipped_types": { | ||
| "Dashboard": 1 | ||
| }, | ||
| "output": "/tmp" | ||
| } | ||
| ``` | ||
|
|
||
| **Output Directory Structure:** | ||
|
|
||
| The exported output mirrors the workspace folder structure: | ||
|
|
||
| ``` | ||
| <output_path>/ | ||
| ├── notebook1.Notebook/ | ||
| │ ├── .platform | ||
| │ └── notebook-content.ipynb | ||
| ├── Folder1/ | ||
| │ ├── notebook2.Notebook/ | ||
| │ │ ├── .platform | ||
| │ │ └── notebook-content.ipynb | ||
| │ └── report1.Report/ | ||
| │ ├── .platform | ||
| │ ├── definition.pbir | ||
| │ └── StaticResources/ | ||
| │ └── ... | ||
| └── Folder2/ | ||
| └── model1.SemanticModel/ | ||
| ├── .platform | ||
| └── definition/ | ||
| ├── model.tmdl | ||
| └── ... | ||
| ``` | ||
|
ohadedry marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. |
70 changes: 70 additions & 0 deletions
70
src/fabric_cli/commands/fs/bulk_export/fab_fs_bulk_export_folder.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # Copyright (c) Microsoft Corporation. | ||
| # Licensed under the MIT License. | ||
|
|
||
| from argparse import Namespace | ||
| from copy import deepcopy | ||
| import json | ||
|
|
||
| from fabric_cli.core.hiearchy.fab_folder import Folder | ||
| from fabric_cli.core.fab_exceptions import FabricCLIError | ||
| from fabric_cli.core import fab_constant | ||
| from fabric_cli.client import fab_api_item as item_api | ||
| from fabric_cli.errors.bulk_export import BulkExportErrors | ||
| from fabric_cli.core.hiearchy.fab_hiearchy import Item | ||
| from fabric_cli.utils import fab_cmd_fs_utils as utils_fs | ||
| from fabric_cli.utils.fab_cmd_bulk_export_utils import ContextItemsSupportMap | ||
| from fabric_cli.utils import fab_cmd_bulk_export_utils as bulk_export_utils | ||
|
|
||
|
|
||
| def bulk_export_folder(context: Folder, args: Namespace) -> None: | ||
| """Bulk-export a folder with its contents, including sub-folders recursively.""" | ||
| args = deepcopy(args) | ||
|
|
||
| workspace_id = context.workspace.id | ||
| args.ws_id = workspace_id | ||
| args.from_path = context.path.strip("/") | ||
|
|
||
| items_support = _collect_context_items(context) | ||
| if not items_support["supported_items"]: | ||
| error_message = ( | ||
| BulkExportErrors.empty_target(context.full_name) | ||
| if not items_support["unsupported_items"] | ||
| else BulkExportErrors.no_exportable_items() | ||
| ) | ||
| raise FabricCLIError( | ||
| error_message, | ||
| fab_constant.ERROR_INVALID_OPERATION, | ||
| ) | ||
|
|
||
| exported_item_ids = [item.id for item in items_support["supported_items"]] | ||
| payload = bulk_export_utils.create_bulk_export_payload(exported_item_ids) | ||
| response = item_api.bulk_export_definitions(args, payload) | ||
| exported_definitions = json.loads(response.text) | ||
| bulk_export_utils.export_definition_parts_to_storage( | ||
| args, context.full_name, exported_definitions | ||
| ) | ||
| bulk_export_utils.print_bulk_export_summary(args, items_support) | ||
|
|
||
|
|
||
| def _collect_context_items( | ||
| context: Folder, | ||
| ) -> ContextItemsSupportMap: | ||
| """Recursively collect all item IDs under a folder and its sub-folders.""" | ||
| supported_items: list[Item] = [] | ||
| unsupported_items: list[Item] = [] | ||
| elements = utils_fs.get_ws_elements(context) | ||
|
|
||
| for element in elements: | ||
| if isinstance(element, Item): | ||
| try: | ||
| if bulk_export_utils.is_command_supported(element): | ||
| supported_items.append(element) | ||
| except FabricCLIError: | ||
| unsupported_items.append(element) | ||
|
ohadedry marked this conversation as resolved.
|
||
| pass | ||
| elif isinstance(element, Folder): | ||
| folder_items = _collect_context_items(element) | ||
| supported_items.extend(folder_items["supported_items"]) | ||
| unsupported_items.extend(folder_items["unsupported_items"]) | ||
|
|
||
| return {"supported_items": supported_items, "unsupported_items": unsupported_items} | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.