diff --git a/_freeze/docs/force/guide/level2/execute-results/html.json b/_freeze/docs/force/guide/level2/execute-results/html.json new file mode 100644 index 0000000..8438206 --- /dev/null +++ b/_freeze/docs/force/guide/level2/execute-results/html.json @@ -0,0 +1,16 @@ +{ + "hash": "d7fc8d38e1e1a8c9b093993fa7f2fd53", + "result": { + "engine": "jupyter", + "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\n---\n\n# FORCE level 2\n\n## Recap\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n::: {#748a2267 .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n#w,s,e,n = 10.386, 44.437, 11.423, 44.973\nw, s, e, n = 11.0, 44.5, 11.1, 44.6\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method.\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n## openEO\n\n::: {#9026a05b .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent, }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#a6cae6cc .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n## Level 2 processing\n\n\nThe [openeo Process](processes.qmd) is straightforwardly named `force_level2`. It takes most of the parameters that can be specified in the FORCE level 2 parameter file. See the article on [parametrization](parametrization.qmd) for more details which parameters are (not) exposed by the process.\n\n:::{.callout-tip}\nThe FORCE parameter file format uses [`SCREAMING_SNAKE_CASE`](https://en.wikipedia.org/wiki/screaming_snake_case) to name its parameters.\nThe integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) instead, to be consistent with openEO conventions. Otherwise, the parameter names are unchanged, so it is straightforward to determine the openEO name if you are familiar with the FORCE parameters.\n:::\n\nThe parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process).\n\n\n\n::: {#d3c0a21b .cell execution_count=4}\n``` {.python .cell-code}\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#5b560fce .cell execution_count=5}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n:::{.callout-note collapse=\"true\" title=\"FORCE level 2 parameters\"}\n\n::: {#6479554f .cell execution_count=6}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=24}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n### Building the Process Graph\n\nThe FORCE process graph is based on the `force_level2` process, as described in [openEO processes](processes.qmd). Because FORCE does not use the standard openEO version of the \"data cube\" concept, but\nuses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a StacResource and export the processing results to a workspace.\n\n\nConcept | Qualified name (Python client) | Purpose\n--------|--------------------------------|-------------\n`StacResource` | [`openeo.rest.stac_resource.StacResource`](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) | A handle to the process graph results, makes it possible to inspect the results as usual for openEO datacubes.\n`export_workspace` | [`openeo.rest.result.SaveResult.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.result.SaveResult.export_workspace) | A process to publish the processing results (e.g., a FORCE ARD data cube) to a shared cloud storage location. There is a special workspace configured for FORCE (`apex-force-force-results`) accessible to all users of the CDSE openEO backend. The workspace serves as a storage location to keep data cubes for later processing (e.g., with the [TSA](tsa.qmd) module).\n`PGNode` | [`openeo.internal.graph_building.PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode) | A single node in a process graph. `PGNode` is an internal class but may be used for advanced graph building purposes [as described here](https://open-eo.github.io/openeo-python-client/processes.html#callback-as-pgnode). We use it for the `graph` parameter of `StacResource`.\n\n\n#### Workspaces\n\nOpenEO uses the concept of [Workspaces](https://docs.openeo.cloud/workspaces/) to interface with object storage. For processing with FORCE, we can use a workspace dedicated to FORCE to store and exchange processing results between level 2 and higher level processing, and with colleagues.\n\nWe can use either the dedicated workspace `apex-force-results-workspace`, which is accessible to all CDSE openeo users, or another registered workspace. See the [openEO platform documentation on workspaces](https://docs.openeo.cloud/workspaces/#registering-a-workspace) on how workspaces can be accessed and registered.\n\nWhen running FORCE level2 on its own, we do not need to use a workspace. Simply download the results when you are done with processing. However, to reference the level 2 processing results in [further processing](tsa.qmd), we need to use a workspace.\n\nWe can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process.\n\nWhen using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system. \nIt is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident.\n\n\n:::{.callout-tip}\n# FORCE workspace\n\nThe dedicated workspace for FORCE results is accessible to all users:\n`apex-force-results-workspace` ()\n:::\n\n\nWe generate the `force_level2` process graph using a Process Graph Node ([`PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode)). In order to interpret the backend response, we wrap the resulting process graph as a [`StacResource`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource).\nThis makes it possible to render the backend response nicely in Jupyter environments and gives us convenient methods to access our results when the process is done.\n\nPass [parameters](parametrization.qmd) to the level 2 processing system in the `arguments` dictionary.\nYou may give the processing a name which will determine the name of the STAC item generated for the results using the parameter `name`.\n\n\n#### Area of extent\n\nAnother parameter which is handled in a special way (i.e., not passed directly to FORCE) is `aoi`. In FORCE level 2, you can specify the `FILE_AOI` parameter to clip the output to a certain extent.\nInstead of a file, `aoi` takes a geojson object to specify area of interest.\n\n- The `aoi` parameter controls the **area of the output produced by FORCE**. The `spatial_extent` parameter passed to the query limits the **input files**\n- `aoi` is applied after radiometric correction (so it will not reduce the computational effort) and before tiling [as per the FORCE documentation](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html) (description of parameter `FILE_AOI`)\n- By default, the output will cover all of the inputs, without clipping.\n\n::: {#be232338 .cell execution_count=7}\n``` {.python .cell-code}\nw, s, e, n = spatial_extent[\"west\"], spatial_extent[\"south\"], spatial_extent[\"east\"], spatial_extent[\"north\"]\naoi = f'{{ \"type\": \"Feature\", \"geometry\": {{ \"type\": \"Polygon\", \"coordinates\": [[[{w},{s}],[{w},{n}],[{e},{n}],[{e},{s}],[{w},{s}]]] }}, \"properties\": {{ \"name\": \"FORCE test\" }} }}'\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\nWhen using the `query_stac` process, we pass the **contents** of a STAC document to the process (not a reference). To do this, pass the `process_graph` directly as `stac_document`\n\n:::{.callout-warning}\nDue to backend limitations, the `stac_document` parameter can only handle a document of limited length. The `query_stac` process generates an item collection which quickly grows in size (the entire content of each STAC item is contained in it). Therefore, use `query_stac` only for small processing tasks with only a handful of items. Otherwise use a different query method to pass an URL or a catalog instead.\n\n**It is recommended to rely on Pystac client for catalog queries for now.**\n:::\n\n::: {#5eed5b0f .cell execution_count=8}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nprocessing_name = \"Modena\"\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"aoi\": aoi,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n#### Pystac client {.active}\n\n:::{.callout-warning}\n\nDue to backend limitations the `stac_document` parameter can only handle a document of limited length. This means it can only handle collections of very few items.\nIt is therefore advisable to either pass an URL (if possible) or to convert the item_collection returned by [`pystac_client.Client.search`](https://pystac-client.readthedocs.io/en/latest/api.html#pystac_client.Client.search) to a catalog. The item collection contains the entire content of each STAC item, whereas the catalog only contains a reference to the items, making a much shorter document.\n\n:::\n\nIn case the backend limitation will be fixed, the transformation step will no longer be necessary.\nIt is already possible to pass `item_collection.to_dict()` directly as `stac_document`, if the item collection is small.\n\n##### Convert Item Collection to Catalog\n\n::: {#25094442 .cell execution_count=10}\n``` {.python .cell-code}\nimport pystac\n\ndef transform_item_collection_to_catalog_with_links(item_collection):\n catalog = pystac.Catalog(\n id=\"item-collection-catalog\",\n description=\"Catalog from item collection\",\n )\n for item in item_collection.items:\n item_link = pystac.Link(\n rel=pystac.RelType.ITEM,\n target=item.self_href,\n media_type=pystac.MediaType.GEOJSON,\n title=f\"Item {item.id}\",\n )\n catalog.add_link(item_link)\n return catalog\n\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n```\n:::\n\n\n##### Process Graph\n\n::: {#26d74c23 .cell execution_count=11}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\n# Temporary fix\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"run_cwl_to_stac\",\n arguments=dict(\n cwl=\"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\",\n context={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n ),\n connection=connection,\n)\n# Optional when only processing level 2\n# If you intend to continue processing, or share the results with others,\n# add the `export_workspace` call:\nimport getpass\n\n# Be careful: The merge determines the storage location of your results.\n# You may overwrite your results or those of others, if the `merge` parameter is not\n# Set to a unique value.\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(\"merge: \", merge)\nforce_l2_stac_resource = force_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nmerge: hannes/experiments/Modena\n```\n:::\n:::\n\n\n\n\n:::\n\n\nYou can (optionally) inspect the process graph to visualize it.\n\n:::{.callout-note title=\"Process Graph JSON\" collapse=\"true\"}\n\n:::{.panel-tabset group=\"query_engine\"}\n\n### openEO\n\n\n\n::: {#1da7be58 .cell execution_count=14}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"querystac1\": {\n \"process_id\": \"query_stac\",\n \"arguments\": {\n \"spatial_extent\": {\n \"west\": 11.0,\n \"south\": 44.5,\n \"east\": 11.1,\n \"north\": 44.6\n },\n \"temporal_extent\": [\n \"2026-04-17\",\n \"2026-04-27\"\n ],\n \"url\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\n }\n },\n \"forcelevel21\": {\n \"process_id\": \"force_level2\",\n \"arguments\": {\n \"aoi\": \"{ \\\"type\\\": \\\"Feature\\\", \\\"geometry\\\": { \\\"type\\\": \\\"Polygon\\\", \\\"coordinates\\\": [[[11.0,44.5],[11.0,44.6],[11.1,44.6],[11.1,44.5],[11.0,44.5]]] }, \\\"properties\\\": { \\\"name\\\": \\\"FORCE test\\\" } }\",\n \"do_brdf\": true,\n \"name\": \"Modena\",\n \"stac_document\": {\n \"from_node\": \"querystac1\"\n }\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"forcelevel21\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n### Pystac client {.active}\n\n\n\n::: {#257249b8 .cell execution_count=16}\n``` {.python .cell-code}\nprint(force_l2_stac_resource.to_json())\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n{\n \"process_graph\": {\n \"runcwltostac1\": {\n \"process_id\": \"run_cwl_to_stac\",\n \"arguments\": {\n \"context\": {\n \"stac_document\": {\n \"type\": \"Catalog\",\n \"id\": \"item-collection-catalog\",\n \"stac_version\": \"1.1.0\",\n \"description\": \"Catalog from item collection\",\n \"links\": [\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\"\n },\n {\n \"rel\": \"item\",\n \"href\": \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\",\n \"type\": \"application/geo+json\",\n \"title\": \"Item S2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\"\n }\n ]\n },\n \"name\": \"Modena\",\n \"do_brdf\": true\n },\n \"cwl\": \"https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl\"\n }\n },\n \"exportworkspace1\": {\n \"process_id\": \"export_workspace\",\n \"arguments\": {\n \"data\": {\n \"from_node\": \"runcwltostac1\"\n },\n \"merge\": \"hannes/experiments/Modena\",\n \"workspace\": \"apex-force-results-workspace\"\n },\n \"result\": true\n }\n }\n}\n```\n:::\n:::\n\n\n\n:::\n\n:::\n\n\n### Executing the Process Graph\n\nWith the process graph in place, we can submit it and wait for it to complete.\nYou can [monitor the progress](processes.qmd#monitoring-progress) through the client library or the web interface.\n\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job(title=f\"FORCE level 2 {processing_name}\")\nl2_job.start() # use start_and_wait instead to block until the job is done\n```\n\n## Accessing results\n\nWhen the job has completed, we can inspect the generated STAC metadata of the FORCE data cube and download it.\n\n**Downloading is optional**, if you wish to proceed with [higher level processing](tsa.qmd). If your processing chain is complete after the level 2 module, you may download the results to continue your analysis.\n\nThe data cube is persistent on cloud storage, so you may download it at a later time or run multiple higher level processes with the same level 2 cube.\nThe data cube is referenced by the URL to its STAC catalog (or item) when passing it to other processes.\n\n\n\n:::{.callout-warning}\nWhile the cube will stay in place for multiple days, eventually the storage will be cleared. So please **make sure to download your processing results for long term storage**.\n:::\n\n\n\n\n\n#### Inspecting STAC metadata {#sec-inspecting-stac}\n\n:::{.callout-tip}\nThis step is optional\n:::\n\nThe basic URL of the FORCE workspace `apex-force-results-workspace` is \n. Together with the `merge` path defined for our processing, we can determine the path to the `catalog.json` for our FORCE level results:\n\n```{.python}\nfrom pystac.catalog import Catalog\n\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nl2_catalog = Catalog.from_file(l2_catalog_url)\n\nl2_catalog.set_self_href(l2_catalog_url)\nl2_catalog\n```\n\n\n\n\n\n#### Downloading files\n\n:::{.callout-tip}\nThis step is optional if you plan to continue processing with FORCE on CDSE\n:::\n\nAssets can be downloaded using the standard openEO mechanism to access result assets.\nThe STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files). \n\nTo inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead.\n\n```{.python}\nl2_results = l2_job.get_results()\nl2_results\n```\n\n```{.python}\nl2_results.download_files(\"force-level2-results\")\n```\n\n\n\n", + "supporting": [ + "level2_files" + ], + "filters": [], + "includes": { + "include-in-header": [ + "\n\n\n" + ] + } + } +} \ No newline at end of file diff --git a/_freeze/docs/force/guide/processes/execute-results/html.json b/_freeze/docs/force/guide/processes/execute-results/html.json new file mode 100644 index 0000000..8acd317 --- /dev/null +++ b/_freeze/docs/force/guide/processes/execute-results/html.json @@ -0,0 +1,16 @@ +{ + "hash": "684b200a45fcc2c82940d1005b0ab3c3", + "result": { + "engine": "jupyter", + "markdown": "---\nformat:\n html:\n code-fold: false\n page-navigation: true\nexecute:\n # Cache is necessary to avoid having to authenticate to build the documentation\n freeze: true\n---\n\n# openEO processes\n\nUsing an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental.\n\nThe processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file ([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). See the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults.\n\n\nTo discover the processes, we must first connect to the CDSE openEO backend:\n\n::: {#6c51ce37 .cell execution_count=1}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"https://openeo.dataspace.copernicus.eu\")\nconnection.authenticate_oidc()\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=1}\n```\n\n```\n:::\n:::\n\n\n\n\n::: {#e7080f59 .cell execution_count=2}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\nIf you don't have your [credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables), you will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using Open ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html#device-flow), [Python client](https://openeo.org/documentation/1.0/authentication.html))\n\n\n## Query\n\n\nWe can use either the openEO process `query_stac` or another method to select input products for processing with FORCE. See the [next section](query.qmd) for an in depth discussion on the different methods how inputs may be specified.\n\n\n:::{.callout-note collapse=\"true\" title=\"query_stac process definition\"}\n\n::: {#fc6319b2 .cell execution_count=3}\n``` {.python .cell-code}\nconnection.describe_process(\"query_stac\")\n```\n\n::: {.cell-output .cell-output-display execution_count=3}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n\n\nWe can use openEO's `describe_process` API call to discover the parameters of an openEO process.\n\n\n## Level 2\n\n\nThe openEO integration of FORCE provides a `force_level2` process wrapping the [FORCE level 2 processing system](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/).\n\n:::{.callout-note collapse=\"true\" title=\"force_level2 process definition\"}\n\n::: {#8395daba .cell execution_count=4}\n``` {.python .cell-code}\nconnection.describe_process(\"force_level2\")\n```\n\n::: {.cell-output .cell-output-display execution_count=4}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\nThe openEO processes exposes most of the options that can be set in the FORCE parameter file.\n\n\n\n## Time Series Analysis\n\nThe `force_tsa` process wraps the [Time Series Analysis](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) module.\n\n:::{.callout-note collapse=\"true\" title=\"force_tsa process definition\"}\n\n::: {#a5306e02 .cell execution_count=5}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=5}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n# openEO job management\n\nBesides the openEO processes which make up the process graph, the openEO client provides functionality to submit, schedule and control jobs.\nPlease refer to the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html) (or the documentation of your preferred openEO client) for a detailed explanation of job management using openEO.\n\n\nName | Qualified name (Python client) | Purpose\n-----|----------------|--------\n`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend \n`start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion\n`start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution\n`get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files)\n`download_files` | [`openeo.rest.job.JobResults.download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) | Download all assets (files) produced by a job\n`download_file` | [`openeo.rest.job.JobResults.download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) | Download a singular asset (file) produced by a job\n\n\n## Monitoring progress\n\nWhen using `start_and_wait` to start a job, status updates will automatically be printed to the command line or jupyter output cell.\n\nAlternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained e.g., in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job).\n\n", + "supporting": [ + "processes_files" + ], + "filters": [], + "includes": { + "include-in-header": [ + "\n\n\n" + ] + } + } +} \ No newline at end of file diff --git a/_freeze/docs/force/guide/processes_alt/execute-results/html.json b/_freeze/docs/force/guide/processes_alt/execute-results/html.json new file mode 100644 index 0000000..f616297 --- /dev/null +++ b/_freeze/docs/force/guide/processes_alt/execute-results/html.json @@ -0,0 +1,12 @@ +{ + "hash": "022f19f714a3830b15ad0bdb6c418d37", + "result": { + "engine": "jupyter", + "markdown": "---\nformat:\n html:\n code-fold: false\n panel: fill\n page-navigation: true\nexecute:\n # Cache is necessary to avoid having to authenticate \n freeze: true\n---\n\n# openEO processes\n\nUsing an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental.\n\nTo discover the processes, we must first connect to the CDSE openEO backend\n\n::: {#e2240125 .cell execution_count=1}\n``` {.python .cell-code}\nimport openeo\n#connection = openeo.connect(\"https://openeo-staging.dataspace.copernicus.eu\")\n#connection.authenticate_oidc()\n```\n:::\n\n\nCannot end with a code block...\n\n## Another title\n\n\nWe can use either the openEO process `query_stac` or another method to select input products for processing with FORCE. See the [next section](query.qmd) for an in depth discussion on how to access inputs.\n\n\n## Level 2\n\n\n## Time Series Analysis\n\n\n:::{.callout-warning}\nThe `force_tsa` process is not yet deployed\n:::\n\n\n\n# openEO job management\n\n", + "supporting": [ + "processes_alt_files" + ], + "filters": [], + "includes": {} + } +} \ No newline at end of file diff --git a/_freeze/docs/force/guide/query/execute-results/html.json b/_freeze/docs/force/guide/query/execute-results/html.json new file mode 100644 index 0000000..c0aba8b --- /dev/null +++ b/_freeze/docs/force/guide/query/execute-results/html.json @@ -0,0 +1,16 @@ +{ + "hash": "90ad4503d338cebe5ed8c4be181c03c8", + "result": { + "engine": "jupyter", + "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\n---\n\n# Selecting input data\n\nWhen running FORCE level 2 locally, we are responsible to manually provide the necessary input data and reference their full path in the `FILE_QUEUE` parameter of the [FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html).\n\nWhen running on CDSE, we do not have access to the excuting environment and its filesystem directly.\nDownloads, file management and creation of the file queue are managed by the integration. \nTo specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process. \n\n:::{.callout-warning}\nMake sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to the entire Sentinel-2 L1C collection on CDSE for example\n:::\n\n## STAC\n\nThe integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata. \n\nIf you are unfamiliar with STAC, we recommend that you familiarize yourself with the basic components of the specification (What are catalogs, items and feature collections?) [on the official STAC specification website](https://stacspec.org/en) before proceeding.\n\n\n**The CDSE Sentinel-2 L1C STAC collection which we use as input for FORCE level 2 processing, is hosted at [https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c](https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c)**\n\nIn the following, we show how this collection may be queried to to obtain inputs for FORCE level 2 processing.\n\n\n## Querying STAC catalogs/collections\n\nFORCE level 2 can work with any STAC URL or document referencing input products of the correct type (Sentinel-2 L1C). The following describes how to make a query to the CDSE STAC catalog.\nBe advised to query the CDSE STAC collection of L1C, as data access will be faster than external archives and the integration makes some assumptions about the metadata provided in the STAC items describing\nthe products.\n\nIt is recommended to make the query using `pystac_client` for now, as using the `query_stac` process limits us to very small data sets See below for more details.\n\n### Area of interest\n\nAs an example, we selected an area in northern Italy, around the city of Modena.\n\n::: {#cc4cc097 .cell execution_count=1}\n``` {.python .cell-code}\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.callout-warning}\nWhen defining the temporal extent as shown above, be aware that different methods may interpret time ranges differently. OpenEO's `query_stac` behaves just like [`load_collection`](https://openeo.org/documentation/1.0/processes.html#load_collection): The second instant is **excluded** from the search.\n\nContrarily, `pystac_client`'s [`search`](https://pystac-client.readthedocs.io/en/stable/api.html#pystac_client.Client.search) method **includes** the second date when given as a list. Note that `pystac_client`'s search has many more options for specifying time ranges.\n:::\n\n::: {#efd5e51c .cell execution_count=2}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"AOI plot\"}\nimport matplotlib.pyplot as plt\nimport contextily as ctx\nimport geopandas as gpd\nfrom shapely import Polygon\n\ndef plot_area_of_interest(\n w, s, e, n, *, \n context=5,\n color=\"red\",\n linewidth=2,\n title=\"Spatial Extent\",\n figsize=(12, 8),\n show_tick_labels=True,\n ):\n polygon = Polygon([(w, s), (e, s), (e, n), (w, n)])\n gdf = gpd.GeoDataFrame(geometry=[polygon], crs=\"EPSG:4326\")\n fig, ax = plt.subplots(1, 1, figsize=figsize)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n gdf.plot(edgecolor=color, facecolor=\"none\", linewidth=linewidth, ax=ax)\n ax.set_xlim(w - context, e + context)\n ax.set_ylim(s - context, n + context)\n if not show_tick_labels:\n ax.set_xticks([])\n ax.set_yticks([])\n ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik, crs=gdf.crs)\n fig.suptitle(title)\n fig.tight_layout()\n\nplot_area_of_interest(\n w=w,\n s=s,\n e=e,\n n=n,\n context=3,\n figsize=(10, 6),\n title=f\"Spatial Extent: Modena\",\n show_tick_labels=False\n)\n```\n\n::: {.cell-output .cell-output-display}\n![](query_files/figure-html/cell-3-output-1.png){width=414 height=570}\n:::\n:::\n\n\n### Querying a catalog\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#0aa7078d .cell execution_count=3}\n``` {.python .cell-code}\nimport openeo\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nconnection = openeo.connect(\"openeo.dataspace.copernicus.eu\").authenticate_oidc()\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n\n\n::: {#06d59308 .cell execution_count=4}\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\nWe can directly use `query_pg` as a parameter for FORCE level 2, using openEO's process chaining.\nTo inspect the query results ahead of time, we may want to execute it ahead of time.\nIf we do not do this, execution of the query will be delayed until we submit the entire process graph (including processing with FORCE).\n\n::: {#3a0c3777 .cell execution_count=5}\n``` {.python .cell-code}\nquery_res = connection.execute(query_pg)\n\nprint(f\"Found products ({len(query_res['features'])}):\")\nprint(\"\\n\".join([f[\"id\"] for f in query_res[\"features\"]]), \"\\n\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound products (18):\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842 \n\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#971ae466 .cell execution_count=6}\n``` {.python .cell-code}\nimport pystac_client\n\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nclient = pystac_client.Client.open(\"https://stac.dataspace.copernicus.eu/v1\")\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n\nprint(\"Found items:\")\nfor item in search.items():\n print(item.id)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFound items:\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954\nS2C_MSIL1C_20260430T102021_N0512_R065_T32TNQ_20260430T135954\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TPQ_20260429T152547\nS2A_MSIL1C_20260429T101041_N0512_R022_T32TNQ_20260429T152547\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TPQ_20260427T142022\nS2C_MSIL1C_20260427T101021_N0512_R022_T32TNQ_20260427T142022\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TPQ_20260425T142712\nS2B_MSIL1C_20260425T101559_N0512_R065_T32TNQ_20260425T142712\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TPQ_20260422T153540\nS2A_MSIL1C_20260422T102041_N0512_R065_T32TNQ_20260422T153540\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TPQ_20260422T134442\nS2B_MSIL1C_20260422T101019_N0512_R022_T32TNQ_20260422T134442\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TPQ_20260420T135843\nS2C_MSIL1C_20260420T102021_N0512_R065_T32TNQ_20260420T135843\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521\nS2A_MSIL1C_20260419T100711_N0512_R022_T32TNQ_20260419T152521\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TPQ_20260417T134842\nS2C_MSIL1C_20260417T101021_N0512_R022_T32TNQ_20260417T134842\n```\n:::\n:::\n\n\n:::\n\n:::{.content-visible when-format=\"html\"}\n\n### Inspecting query results\n\nWe may use pystac to inspect or process the feature collection further. In this example, we inspect the first of the returned items:\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#d3ca7a76 .cell execution_count=7}\n``` {.python .cell-code}\nimport pystac\n\nitem = pystac.Item.from_dict(query_res[\"features\"][0])\nitem\n```\n\n::: {.cell-output .cell-output-display execution_count=15}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n#### Pystac client {.active}\n\n::: {#7c6ee016 .cell execution_count=8}\n``` {.python .cell-code}\nnext(iter(search.items()))\n```\n\n::: {.cell-output .cell-output-display execution_count=16}\n```{=html}\n\n\n\n
\n
\n <Item id=S2C_MSIL1C_20260430T102021_N0512_R065_T32TPQ_20260430T135954>\n
\n\n
\n```\n:::\n:::\n\n\n:::\n:::\n\n\n## Other methods\n\nCDSE provides a [graphical STAC browser](https://browser.stac.dataspace.copernicus.eu/collections/sentinel-2-l1c) that can be used to search for and select singular items.\n\n", + "supporting": [ + "query_files" + ], + "filters": [], + "includes": { + "include-in-header": [ + "\n\n\n" + ] + } + } +} \ No newline at end of file diff --git a/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png b/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png new file mode 100644 index 0000000..4caba72 Binary files /dev/null and b/_freeze/docs/force/guide/query/figure-html/cell-3-output-1.png differ diff --git a/_freeze/docs/force/guide/tsa/execute-results/html.json b/_freeze/docs/force/guide/tsa/execute-results/html.json new file mode 100644 index 0000000..2e374ba --- /dev/null +++ b/_freeze/docs/force/guide/tsa/execute-results/html.json @@ -0,0 +1,16 @@ +{ + "hash": "fd24067de41fe2ca504c8761cc78a40d", + "result": { + "engine": "jupyter", + "markdown": "---\nformat:\n html:\n page-navigation: true\nexecute:\n freeze: true\ntitle: FORCE Time Series Analysis\n---\n\n\n\n\n\n\n\n## Recap\n\n:::{.callout-warning}\nYou should use `pystac_client` based querying if your inputs encompass more than just a handful of items\n:::\n\n:::{.callout-tip collapse=\"true\" title=\"Selecting input products\"}\n\n\n::: {#37b34abb .cell execution_count=1}\n``` {.python .cell-code}\nL1C_COLLECTION_URL = \"https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c\"\nSTAC_ROOT_URL = \"https://stac.dataspace.copernicus.eu/v1\"\n\nw,s,e,n = 10.386, 44.437, 11.423, 44.973\nspatial_extent = { \"west\": w, \"south\": s, \"east\": e, \"north\": n}\n# ATTENTION: Inclusion of the second date in the search depends on the query method (see below).\ntemporal_extent=[\"2026-04-17\", \"2026-05-01\"]\n```\n:::\n\n\n:::{.panel-tabset group=\"query_engine\"}\n\n\n## openEO\n\n::: {#b9f2ed7d .cell execution_count=2}\n``` {.python .cell-code}\nimport openeo\nconnection = openeo.connect(\"openeo-staging.dataspace.copernicus.eu\").authenticate_oidc()\n\nquery_pg = openeo.processes.process(\n \"query_stac\",\n arguments={\n \"url\": L1C_COLLECTION_URL,\n \"temporal_extent\": temporal_extent,\n \"spatial_extent\": spatial_extent,\n }\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nAuthenticated using refresh token.\n```\n:::\n:::\n\n\n## Pystac client {.active}\n\n::: {#3f754c40 .cell execution_count=3}\n``` {.python .cell-code}\nimport pystac_client\n\nclient = pystac_client.Client.open(STAC_ROOT_URL)\nsearch = client.search(\n datetime=temporal_extent,\n collections=[\"sentinel-2-l1c\"],\n bbox=[w, s, e, n]\n)\nitem_collection = search.item_collection()\n```\n:::\n\n\n \n:::\n \n:::\n\n:::{.callout-tip collapse=\"true\" title=\"level2 processing\"}\n:::{.panel-tabset group=\"query_engine\"}\n\n#### openEO\n\n::: {#f0ea342d .cell execution_count=4}\n``` {.python .cell-code}\nimport getpass\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\n# Make sure to choose a unique name \nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_pg,\n \"name\": processing_name,\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n);\n```\n:::\n\n\n\n\n#### Pystac client {.active}\n\n\n\n::: {#b963273a .cell execution_count=7}\n``` {.python .cell-code}\nfrom openeo.internal.graph_building import PGNode\nquery_catalog = transform_item_collection_to_catalog_with_links(item_collection)\n\nprocessing_name = \"Modena\"\n\nforce_l2_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_level2\",\n arguments={\n \"stac_document\": query_catalog.to_dict(),\n \"name\": \"Modena\",\n \"do_brdf\": True,\n }\n ),\n connection=connection,\n)\n\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\nforce_l2_stac_resource.export_workspace(\n workspace=\"apex-force-results-workspace\",\n merge=merge,\n)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=7}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n \n:::\n\n```{.python}\nl2_job = force_l2_stac_resource.create_job()\nl2_job.start()\n```\n\n\n\n \n:::\n\n## Time Series Analysis\n\nThe openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process, it takes most of the parameters usually passed to the [TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the [parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). Parameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files).\n\nDetails of parametrization are discussed in the [dedicated article](parametrization.qmd).\n\n:::{.callout-note collapse=\"true\" title=\"FORCE TSA parameters\"}\n\n::: {#39173dcf .cell execution_count=9}\n``` {.python .cell-code}\nconnection.describe_process(\"force_tsa\")\n```\n\n::: {.cell-output .cell-output-display execution_count=9}\n```{=html}\n\n \n \n \n \n \n```\n:::\n:::\n\n\n:::\n\n### Using FORCE level 2 results as an input\n\nThe process graph for FORCE TSA built in the same way as the level 2 process graph is built.\nThe main input for FORCE TSA is the STAC catalog (or item) from the level 2 processing.\n\nWe can access the STAC catalog produced by our level 2 job in the workspace,\ndescribed in [level 2 processing article](level2.qmd#Accessing results).\n\nIt is important to remember the `merge` paramter passed to `export_workspace`, this is the path under which we can find the catalog.\n\n::: {#0a39712c .cell execution_count=10}\n``` {.python .cell-code}\nprocessing_name = \"Modena\"\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nprint(merge)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhannes/experiments/Modena\n```\n:::\n:::\n\n\nRemember, the dedicated workspace for FORCE has the URL \n\n\n\n::: {#336d5bc8 .cell execution_count=11}\n``` {.python .cell-code}\nWORKSPACE_URL = \"https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me\"\n```\n:::\n\n\n### TSA Parameters\n\n\nSome parameters can be set a-priori or from prior knowledge of our level 2 processing parameters\n\n::: {#e0106455 .cell execution_count=12}\n``` {.python .cell-code}\ntemporal_extent=[\"2026-04-17\", \"2026-04-27\"]\n```\n:::\n\n\nFor other parameters, we may want to inspect the metadata of the level 2 processing.\nFor example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube. \n\n::: {#54dbfd3c .cell execution_count=13}\n``` {.python .cell-code}\nmerge = f\"{getpass.getuser()}/experiments/{processing_name}\"\nl2_catalog_url = f\"{WORKSPACE_URL}/{merge}/catalog.json\"\nprint(l2_catalog_url)\nl2_catalog = pystac.Catalog.from_file(l2_catalog_url)\nl2_item = next(l2_catalog.get_items()) # there is just one item\nl2_item\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nhttps://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me/hannes/experiments/Modena/catalog.json\n```\n:::\n\n::: {.cell-output .cell-output-display execution_count=13}\n```{=html}\n\n\n\n
\n
\n <Item id=Modena-level2>\n
\n\n
\n```\n:::\n:::\n\n\nWe can extract the tiles manually or programmatically\n\n::: {#24b6c076 .cell execution_count=14}\n``` {.python .cell-code}\ntiles = set([k.split(\".\")[0] for k in l2_item.assets.keys() if k.endswith(\"BOA\")])\ntile_x = set(int(tile[1:5]) for tile in tiles)\ntile_y = set(int(tile[7:]) for tile in tiles)\ntile_x, tile_y\n```\n\n::: {.cell-output .cell-output-display execution_count=14}\n```\n({31, 32}, {29, 30})\n```\n:::\n:::\n\n\n### Process Graph\n\nThe main process is `force_tsa`.\n\n::: {#5851e5f5 .cell execution_count=15}\n``` {.python .cell-code}\nfrom openeo.rest.stac_resource import StacResource\nfrom openeo.internal.graph_building import PGNode\n\n\nforce_tsa_stac_resource = StacResource(\n graph=PGNode(\n process_id=\"force_tsa\",\n arguments={\n \"stac_url\": l2_catalog_url,\n \"name\": \"Modena_TSA\",\n \"date_range\": temporal_extent,\n \"x_tile_range\": [min(tile_x), min(tile_x)],\n \"y_tile_range\": [min(tile_y), max(tile_y)],\n \"stm\": [\"AVG\"],\n \"output_stm\": True,\n # ... more TSA parameters (optional)\n }\n ),\n connection=connection,\n)\n```\n:::\n\n\nWe can then create and start the job as before:\n\n```{.python}\ntsa_job = force_tsa_stac_resource.create_job(title=f\"FORCE TSA {processing_name}\")\ntsa_job.start() # use start_and_wait to block until finished\n```\n\nInspecting STAC metadata and downloading results works in the same way as for level 2 processing:\n\n\n\n```{.python}\ntsa_catalog_url = extract_catalog_url_from_job_logs(tsa_job.logs())\ntsa_catalog = pystac.Catalog.from_file(tsa_catalog_url)\ntsa_item = next(tsa_catalog.get_items())\ntsa_item\n```\n\nDownloading assets as well:\n\n```{.python}\ntsa_results = l2_job.get_results()\ntsa_results\n```\n\n```{.python}\ntsa_results.download_files(\"force-tsa-results\")\n```\n\n\n\n### Further processing\n\nBecause level 2 results are provided by a STAC URL to the TSA process, it is possible run multiple TSA jobs with the same level 2 input, without expensively reprocessing the level 2 data.\nWe just create a new TSA process graph and submit it, passing the same level 2 data cube URL.\n\nRemember that the job assets will be stored for some time on the backend which allows you to do processing over a span of a few days, but will not be stored permanently. Please download and archive results you wish to keep on your own infrastructure.\n\n", + "supporting": [ + "tsa_files" + ], + "filters": [], + "includes": { + "include-in-header": [ + "\n\n\n" + ] + } + } +} \ No newline at end of file diff --git a/_freeze/docs/force/site_libs/clipboard/clipboard.min.js b/_freeze/docs/force/site_libs/clipboard/clipboard.min.js new file mode 100644 index 0000000..1103f81 --- /dev/null +++ b/_freeze/docs/force/site_libs/clipboard/clipboard.min.js @@ -0,0 +1,7 @@ +/*! + * clipboard.js v2.0.11 + * https://clipboardjs.com/ + * + * Licensed MIT © Zeno Rocha + */ +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return b}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),r=n.n(e);function c(t){try{return document.execCommand(t)}catch(t){return}}var a=function(t){t=r()(t);return c("cut"),t};function o(t,e){var n,o,t=(n=t,o="rtl"===document.documentElement.getAttribute("dir"),(t=document.createElement("textarea")).style.fontSize="12pt",t.style.border="0",t.style.padding="0",t.style.margin="0",t.style.position="absolute",t.style[o?"right":"left"]="-9999px",o=window.pageYOffset||document.documentElement.scrollTop,t.style.top="".concat(o,"px"),t.setAttribute("readonly",""),t.value=n,t);return e.container.appendChild(t),e=r()(t),c("copy"),t.remove(),e}var f=function(t){var e=1 Not suited for large scale processing in a single job +- The executors have limited memory available. Large jobs will not run successfully. You can parallelize time series analysis computations to avoid running into memory issues (see Tips & Tricks below). +- Many parameters relying on auxiliary files are not supported. See the [guide on parametrization](guide/parametrization.qmd) for details. + +### TSA + +- Python and R user defined functions (UDFs) are not supported +- Spectral Mixture Analyis is not supported + +## Tips & Tricks + +### Parallelizing large Time Series Analysis jobs + +Server-side parallelization of the TSA module is not currently enabled. It is possible to parallelize TSA by creating one openEO job per tile by setting the `x_tile_range` and `y_tile_range` arguments to a single tile per job. +Furthermore, consider computing spatio-temporal metrics (STM) in separate jobs to reduce memory usage. + +Finally, you may reduce the `chunk_size` which will reduce memory usage at the cost of a longer processing time. diff --git a/docs/force/guide/intro.qmd b/docs/force/guide/intro.qmd new file mode 100644 index 0000000..9fb2647 --- /dev/null +++ b/docs/force/guide/intro.qmd @@ -0,0 +1,43 @@ +# Introduction + +The FORCE integration on CDSE makes it possible to use selected functionality of the FORCE processing engine on the CDSE infrastructure through the openEO API. + +Instead of installing FORCE locally, manually downloading data and annotating your results with metadata, the FORCE integration runs on cloud hardware, takes care of the input data staging and automatically generates a STAC catalog and item from the resulting FORCE data cube. + + +## Prerequisites + +This guide assumes basic familiarity with the FORCE processing engine. +You do not need to be an expert FORCE user to make use of the CDSE deployment. + +::: {.callout-tip title="What to read if you are unfamiliar with FORCE"} +If you are an openEO user who wants to get familiar with FORCE, we recommend to have a look at the FORCE documentation pages for + +- **The [data cube](https://force-eo.readthedocs.io/en/latest/howto/datacube.html) concept and organization**: This is the file structure you will receive when running FORCE processing modules +- **The [level 2 processing system](https://force-eo.readthedocs.io/en/latest/howto/l2-ard.html#level-2-ard)**: the basic FORCE module to generate analysis ready data cubes from raw Satellite input products +- (Optional) **The [Time Series Analysis (TSA)](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/index.html#time-series-analysis) module**: This is the higher level processing module made available through openEO +::: + +::: {.callout-tip title="What to read if you are unfamiliar with openEO"} +If you are a FORCE user curious how to process in the cloud and the openEO interface, it will be helpful to be familiar with + +- The [openEO vocabulary](https://openeo.org/documentation/1.0/glossary.html#processes) +- **The [openEO Python client](https://open-eo.github.io/openeo-python-client/api.html)**: The main interface used to interact with the openEO backend. You may want to use the [R client](https://open-eo.github.io/openeo-r-client/) or [Javascript client](https://open-eo.github.io/openeo-js-client/latest/) instead, if you prefer. However, in this guide, examples will be given with the Python client. +::: + +::: {.callout-important title="Data cube representation"} +Please note that the FORCE integration does not make use of [openEO's data cube concept](https://openeo.org/documentation/1.0/datacubes.html), processing is done on FORCE's native data cubes. +::: + +## Scope + +In this guide, you will learn how to create a full processing pipeline using FORCE and openEO: + +1. Discover inputs: Query a STAC catalog to determine input products +2. Generate an Analysis Ready (ARD) data cube with the FORCE level 2 processing system +3. Analyze time series with FORCE's higher level processing system + +On the way, we will cover how to + +- Download data cubes for local processing / visualization / permanent storage +- Apply higher level processing without having to download intermediate results diff --git a/docs/force/guide/level2.qmd b/docs/force/guide/level2.qmd new file mode 100644 index 0000000..48c7def --- /dev/null +++ b/docs/force/guide/level2.qmd @@ -0,0 +1,446 @@ +--- +format: + html: + page-navigation: true +execute: + freeze: true +--- + +# FORCE level 2 + +## Recap + +:::{.callout-tip collapse="true" title="Selecting input products"} + +```{python} + +L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c" +STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1" +#w,s,e,n = 10.386, 44.437, 11.423, 44.973 +w, s, e, n = 11.0, 44.5, 11.1, 44.6 +spatial_extent = { "west": w, "south": s, "east": e, "north": n} +# ATTENTION: Inclusion of the second date in the search depends on the query method. +temporal_extent=["2026-04-17", "2026-04-27"] + +``` + + +:::{.panel-tabset group="query_engine"} + +## openEO + +```{python} +import openeo +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() + +query_pg = openeo.processes.process( + "query_stac", + arguments={ + "url": L1C_COLLECTION_URL, + "temporal_extent": temporal_extent, + "spatial_extent": spatial_extent, } +) +``` + +## Pystac client {.active} + +```{python} +import pystac_client + +client = pystac_client.Client.open(STAC_ROOT_URL) +search = client.search( + datetime=temporal_extent, + collections=["sentinel-2-l1c"], + bbox=[w, s, e, n] +) +item_collection = search.item_collection() +``` + +::: + +::: + +## Level 2 processing + + +The [openeo Process](processes.qmd) is straightforwardly named `force_level2`. It takes most of the parameters that can be specified in the FORCE level 2 parameter file. See the article on [parametrization](parametrization.qmd) for more details which parameters are (not) exposed by the process. + +:::{.callout-tip} +The FORCE parameter file format uses [`SCREAMING_SNAKE_CASE`](https://en.wikipedia.org/wiki/screaming_snake_case) to name its parameters. +The integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) instead, to be consistent with openEO conventions. Otherwise, the parameter names are unchanged, so it is straightforward to determine the openEO name if you are familiar with the FORCE parameters. +::: + +The parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process). + + + +```{python} +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +``` + + + +```{python} +#| echo: false +connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() +``` + +:::{.callout-note collapse="true" title="FORCE level 2 parameters"} +```{python} +connection.describe_process("force_level2") + +``` +::: + +### Building the Process Graph + +The FORCE process graph is based on the `force_level2` process, as described in [openEO processes](processes.qmd). Because FORCE does not use the standard openEO version of the "data cube" concept, but +uses its own data cube representation, we need to wrap the process so that we can interact with the results in a convenient way as a StacResource and export the processing results to a workspace. + + +Concept | Qualified name (Python client) | Purpose +--------|--------------------------------|------------- +`StacResource` | [`openeo.rest.stac_resource.StacResource`](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) | A handle to the process graph results, makes it possible to inspect the results as usual for openEO datacubes. +`export_workspace` | [`openeo.rest.result.SaveResult.export_workspace`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.result.SaveResult.export_workspace) | A process to publish the processing results (e.g., a FORCE ARD data cube) to a shared cloud storage location. There is a special workspace configured for FORCE (`apex-force-force-results`) accessible to all users of the CDSE openEO backend. The workspace serves as a storage location to keep data cubes for later processing (e.g., with the [TSA](tsa.qmd) module). +`PGNode` | [`openeo.internal.graph_building.PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode) | A single node in a process graph. `PGNode` is an internal class but may be used for advanced graph building purposes [as described here](https://open-eo.github.io/openeo-python-client/processes.html#callback-as-pgnode). We use it for the `graph` parameter of `StacResource`. + + +#### Workspaces + +OpenEO uses the concept of [Workspaces](https://docs.openeo.cloud/workspaces/) to interface with object storage. For processing with FORCE, we can use a workspace dedicated to FORCE to store and exchange processing results between level 2 and higher level processing, and with colleagues. + +We can use either the dedicated workspace `apex-force-results-workspace`, which is accessible to all CDSE openeo users, or another registered workspace. See the [openEO platform documentation on workspaces](https://docs.openeo.cloud/workspaces/#registering-a-workspace) on how workspaces can be accessed and registered. + +When running FORCE level2 on its own, we do not need to use a workspace. Simply download the results when you are done with processing. However, to reference the level 2 processing results in [further processing](tsa.qmd), we need to use a workspace. + +We can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process. + +When using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system. +It is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident. + + +:::{.callout-tip} +# FORCE workspace + +The dedicated workspace for FORCE results is accessible to all users: +`apex-force-results-workspace` () +::: + + +We generate the `force_level2` process graph using a Process Graph Node ([`PGNode`](https://open-eo.github.io/openeo-python-client/api.html#openeo.internal.graph_building.PGNode)). In order to interpret the backend response, we wrap the resulting process graph as a [`StacResource`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.stac_resource.StacResource). +This makes it possible to render the backend response nicely in Jupyter environments and gives us convenient methods to access our results when the process is done. + +Pass [parameters](parametrization.qmd) to the level 2 processing system in the `arguments` dictionary. +You may give the processing a name which will determine the name of the STAC item generated for the results using the parameter `name`. + + +#### Area of extent + +Another parameter which is handled in a special way (i.e., not passed directly to FORCE) is `aoi`. In FORCE level 2, you can specify the `FILE_AOI` parameter to clip the output to a certain extent. +Instead of a file, `aoi` takes a geojson object to specify area of interest. + +- The `aoi` parameter controls the **area of the output produced by FORCE**. The `spatial_extent` parameter passed to the query limits the **input files** +- `aoi` is applied after radiometric correction (so it will not reduce the computational effort) and before tiling [as per the FORCE documentation](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html) (description of parameter `FILE_AOI`) +- By default, the output will cover all of the inputs, without clipping. + +```{python} +w, s, e, n = spatial_extent["west"], spatial_extent["south"], spatial_extent["east"], spatial_extent["north"] +aoi = f'{{ "type": "Feature", "geometry": {{ "type": "Polygon", "coordinates": [[[{w},{s}],[{w},{n}],[{e},{n}],[{e},{s}],[{w},{s}]]] }}, "properties": {{ "name": "FORCE test" }} }}' +``` + + + +:::{.panel-tabset group="query_engine"} + +#### openEO + +When using the `query_stac` process, we pass the **contents** of a STAC document to the process (not a reference). To do this, pass the `process_graph` directly as `stac_document` + +:::{.callout-warning} +Due to backend limitations, the `stac_document` parameter can only handle a document of limited length. The `query_stac` process generates an item collection which quickly grows in size (the entire content of each STAC item is contained in it). Therefore, use `query_stac` only for small processing tasks with only a handful of items. Otherwise use a different query method to pass an URL or a catalog instead. + +**It is recommended to rely on Pystac client for catalog queries for now.** +::: + + + +```{python} +from openeo.rest.stac_resource import StacResource +from openeo.internal.graph_building import PGNode + +processing_name = "Modena" +force_l2_stac_resource = StacResource( + graph=PGNode( + process_id="force_level2", + arguments={ + "stac_document": query_pg, + "name": processing_name, + "aoi": aoi, + "do_brdf": True, + } + ), + connection=connection, +) + +# Optional when only processing level 2 +# If you intend to continue processing, or share the results with others, +# add the `export_workspace` call: +import getpass + +# Be careful: The merge determines the storage location of your results. +# You may overwrite your results or those of others, if the `merge` parameter is not +# Set to a unique value. +merge = f"{getpass.getuser()}/experiments/{processing_name}" +print("merge: ", merge) +force_l2_stac_resource = force_l2_stac_resource.export_workspace( + workspace="apex-force-results-workspace", + merge=merge, +) +``` + +```{python} +#| echo: false +force_l2_stac_resource_openeo = force_l2_stac_resource +``` + + +#### Pystac client {.active} + +:::{.callout-warning} + +Due to backend limitations the `stac_document` parameter can only handle a document of limited length. This means it can only handle collections of very few items. +It is therefore advisable to either pass an URL (if possible) or to convert the item_collection returned by [`pystac_client.Client.search`](https://pystac-client.readthedocs.io/en/latest/api.html#pystac_client.Client.search) to a catalog. The item collection contains the entire content of each STAC item, whereas the catalog only contains a reference to the items, making a much shorter document. + +::: + +In case the backend limitation will be fixed, the transformation step will no longer be necessary. +It is already possible to pass `item_collection.to_dict()` directly as `stac_document`, if the item collection is small. + +##### Convert Item Collection to Catalog + +```{python} +import pystac + +def transform_item_collection_to_catalog_with_links(item_collection): + catalog = pystac.Catalog( + id="item-collection-catalog", + description="Catalog from item collection", + ) + for item in item_collection.items: + item_link = pystac.Link( + rel=pystac.RelType.ITEM, + target=item.self_href, + media_type=pystac.MediaType.GEOJSON, + title=f"Item {item.id}", + ) + catalog.add_link(item_link) + return catalog + +query_catalog = transform_item_collection_to_catalog_with_links(item_collection) +``` + +##### Process Graph + +```{python} +from openeo.rest.stac_resource import StacResource +from openeo.internal.graph_building import PGNode + +force_l2_stac_resource = StacResource( + graph=PGNode( + process_id="force_level2", + arguments={ + "stac_document": query_catalog.to_dict(), + "name": processing_name, + "do_brdf": True, + } + ), + connection=connection, +) + +# Temporary fix +force_l2_stac_resource = StacResource( + graph=PGNode( + process_id="run_cwl_to_stac", + arguments=dict( + cwl="https://github.com/bcdev/apex-force-openeo/releases/latest/download/force-level2.cwl", + context={ + "stac_document": query_catalog.to_dict(), + "name": processing_name, + "do_brdf": True, + } + ), + ), + connection=connection, +) +# Optional when only processing level 2 +# If you intend to continue processing, or share the results with others, +# add the `export_workspace` call: +import getpass + +# Be careful: The merge determines the storage location of your results. +# You may overwrite your results or those of others, if the `merge` parameter is not +# Set to a unique value. +merge = f"{getpass.getuser()}/experiments/{processing_name}" +print("merge: ", merge) +force_l2_stac_resource = force_l2_stac_resource.export_workspace( + workspace="apex-force-results-workspace", + merge=merge, +) +``` + +```{python} +#| echo: false +force_l2_stac_resource_pystac_client = force_l2_stac_resource +``` + +::: + + +You can (optionally) inspect the process graph to visualize it. + +:::{.callout-note title="Process Graph JSON" collapse="true"} + +:::{.panel-tabset group="query_engine"} + +### openEO + +```{python} +#| echo: false +force_l2_stac_resource = force_l2_stac_resource_openeo +``` + +```{python} +print(force_l2_stac_resource.to_json()) +``` + +### Pystac client {.active} + +```{python} +#| echo: false +force_l2_stac_resource = force_l2_stac_resource_pystac_client +``` + +```{python} +print(force_l2_stac_resource.to_json()) +``` + + +::: + +::: + + +### Executing the Process Graph + +With the process graph in place, we can submit it and wait for it to complete. +You can [monitor the progress](processes.qmd#monitoring-progress) through the client library or the web interface. + + +```{.python} +l2_job = force_l2_stac_resource.create_job(title=f"FORCE level 2 {processing_name}") +l2_job.start() # use start_and_wait instead to block until the job is done +``` + +## Accessing results + +When the job has completed, we can inspect the generated STAC metadata of the FORCE data cube and download it. + +**Downloading is optional**, if you wish to proceed with [higher level processing](tsa.qmd). If your processing chain is complete after the level 2 module, you may download the results to continue your analysis. + +The data cube is persistent on cloud storage, so you may download it at a later time or run multiple higher level processes with the same level 2 cube. +The data cube is referenced by the URL to its STAC catalog (or item) when passing it to other processes. + + + +:::{.callout-warning} +While the cube will stay in place for multiple days, eventually the storage will be cleared. So please **make sure to download your processing results for long term storage**. +::: + + + + + +```{python} +#| echo: false +#l2_job = connection.job("j-260527114522498d959f20eff3fc9b78") +l2_job = connection.job("j-260702093443422db1ddead799544cff") +``` + +#### Inspecting STAC metadata {#sec-inspecting-stac} + +:::{.callout-tip} +This step is optional +::: + +The basic URL of the FORCE workspace `apex-force-results-workspace` is +. Together with the `merge` path defined for our processing, we can determine the path to the `catalog.json` for our FORCE level results: + +```{.python} +from pystac.catalog import Catalog + +WORKSPACE_URL = "https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me" + +l2_catalog_url = f"{WORKSPACE_URL}/{merge}/catalog.json" +l2_catalog = Catalog.from_file(l2_catalog_url) + +l2_catalog.set_self_href(l2_catalog_url) +l2_catalog +``` + + +```{python} +#| echo: false +WORKSPACE_URL = "https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me" +``` + + +#### Downloading files + +:::{.callout-tip} +This step is optional if you plan to continue processing with FORCE on CDSE +::: + +Assets can be downloaded using the standard openEO mechanism to access result assets. +The STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files). + +To inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead. + +```{.python} +l2_results = l2_job.get_results() +l2_results +``` + +```{.python} +l2_results.download_files("force-level2-results") +``` + + diff --git a/docs/force/guide/parametrization.qmd b/docs/force/guide/parametrization.qmd new file mode 100644 index 0000000..4009bd9 --- /dev/null +++ b/docs/force/guide/parametrization.qmd @@ -0,0 +1,156 @@ +--- +format: + html: + code-fold: false + page-navigation: true +execute: + freeze: true +--- + +# Parametrization + +The [FORCE openEO processes](processes.qmd) (`force_level2` and `force_tsa`) accept most of the parameters that can be passed using FORCE's native parametrization files ([level 2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). The exceptions to this rule are explained below. + +There are 4 types of parameters that do not correspond exactly to the FORCE parameter files + +1. Special parameters that require translation (e.g., the area of interest) +1. Additional parameters of the openEO integration (e.g., the processing name) +1. FORCE parameters that are not exposed through the openEO API (e.g., the number of processes used for processing) +1. Parameters that support only a subset of the possible values + +Parameter names follow the openEO convention and have `snake_case`. Otherwise, the names are unchanged from those of the original parameter files. +It is possible to inspect the processes (and see their parameters) using openEO's [`describe_process`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process) method. See [processes.qmd] for more. + +Parameter values may be strings (the same as in FORCE parameter files) or native python values which can be serialized to JSON. Values are converted to UPPERCASE before being passed into the parameter file. + +Parameter values are passed as a dictionary to the `arguments` parameter of `openeo.processes.process`: + + +```{.python} +graph = openeo.processes.process( + process_id="force_level2", + arguments=dict( + stac_document=stac_catalog, + name="Modena", + do_brdf=True, + # ... <--- more parameters + } +) + +``` + +## Examples + +| FORCE parameter file | openEO | +|----------------------|--------| +| `PROJECTION = GLANCE7` | `projection = GLANCE7`| +| `DO_BRDF = TRUE` | `do_brdf=True` +| `CLOUD_THRESHOLD = 0.225` | `cloud_threshold=0.255` + +## FORCE Level 2 + +### Special parameters + +- `aoi`: replaces the `FILE_AOI` parameter. Pass a geojson feature (instead of a shapefile). See also [the aoi parameter description of level2](level2.qmd#Area of extent) +- `dem`: replaces digital elevation model specific parameters (`FILE_DEM`, `DEM_RESAMPLING`, `DEM_NODATA`). Only possible value is `Copernicus_30m` + +### openEO specific parameters + +:::{.callout-tip} +Check out the [FORCE level 2](level2.qmd) section of the guide to learn more about selecting and passing inputs. +::: + +- `stac_document` and `stac_url`: STAC documents describing the input products. Can be either an URL `stac_url` or the content (`stac_document` as a dictionary) of a STAC document. One of these parameters must be set. +- `name`: Give a name to the FORCE processing task. Determines the name of the STAC item describing the output data cube. + +### Partially supported parameters + +- `projection`: Only GLANCE7 (the default) is tested. +- `output_format`: Supports `GTiff`, `COG` + +### Unsupported parameters + +Input/Output directories (managed by the integration) + +- `FILE_QUEUE` +- `DIR_LEVEL2` +- `DIR_LOG` +- `DIR_PROVENANCE` +- `DIR_TEMP` + +Parallel processing parameters (managed by the integration) + +- `NPROC` +- `NTHREAD` +- `PARALLEL_READS` +- `DELAY` +- `TIMEOUT_ZIP` + +## FORCE Time Series Analysis + +### Special parameters + +### openEO specific parameters + +### Partially supported parameters + +- `output_format`: Supports `GTiff`, `COG` +- `target_sensor`: Supports Sentinel-2 types: + - SEN2A + - SEN2B + - SEN2C + - SEN2L + + +### Unsupported parameters + +Input/Output directories (managed by the integration) + +- `DIR_LOWER` +- `DIR_HIGHER` +- `DIR_PROVENANCE` +- `DIR_MASK` + +Masking + +- `DIR_MASK` +- `BASE_MASK` +- `READ_ERROR_MASK` + +Output Options + +- `FILE_OUTPUT_OPTIONS` + +Parallel processing (managed by the integration) + +- `NTREHAD_READ` +- `NTHREAD_COMPUTE` +- `NTHREAD_WRITE` +- `STREAMING` (TRUE) +- `PRETTY_PROGRESS` + +Extent + +- `FILE_TILE` + + +Spectral Mixture Analysis + +- `FILE_ENDMEM` (NULL) +- `SMA_SUM_TO_ONE` +- `SMA_NON_NEG` +- `SMA_SHD_NORM` +- `SMA_ENDMEMBER` +- `OUTPUT_RMS` + +Python UDFs + +- `FILE_PYTHON` +- `PYTHON_TYPE` +- `OUTPUT_PYP` + +R UDFs + +- `FILE_RSTATS` +- `RSTATS_TYPE` +- `OUTPUT_RSP` diff --git a/docs/force/guide/processes.qmd b/docs/force/guide/processes.qmd new file mode 100644 index 0000000..7251e74 --- /dev/null +++ b/docs/force/guide/processes.qmd @@ -0,0 +1,107 @@ +--- +format: + html: + code-fold: false + page-navigation: true +execute: + # Cache is necessary to avoid having to authenticate to build the documentation + freeze: true +--- + +# openEO processes + +Using an openEO client, we can control FORCE through a number of [openEO processes](https://openeo.org/documentation/1.0/datacubes.html). The FORCE openEO integration is specific to the backend deployed on CDSE, so the processes described here are not listed among the [standard processes](https://openeo.org/documentation/1.0/processes.html) and are generally experimental. + +The processes wrapping FORCE functionality support most of the options that can be set in the FORCE parameter file ([level2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). See the article on [parametrization](parametrization.qmd) for more details on the supported options and defaults. + + +To discover the processes, we must first connect to the CDSE openEO backend: + +```{python} +import openeo +connection = openeo.connect("https://openeo.dataspace.copernicus.eu") +connection.authenticate_oidc() +``` + + +```{python} +#| echo: false +connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() +``` + + + +If you don't have your [credentials set as environment variables](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables), you will be prompted to log in in a browser window by default. See also the openEO documentation on authentication using Open ID connect ([overview](https://openeo.org/documentation/1.0/authentication.html#device-flow), [Python client](https://openeo.org/documentation/1.0/authentication.html)) + + +## Query + + +We can use either the openEO process `query_stac` or another method to select input products for processing with FORCE. See the [next section](query.qmd) for an in depth discussion on the different methods how inputs may be specified. + + +:::{.callout-note collapse="true" title="query_stac process definition"} +```{python} +connection.describe_process("query_stac") +``` +::: + + + +We can use openEO's `describe_process` API call to discover the parameters of an openEO process. + + +## Level 2 + + +The openEO integration of FORCE provides a `force_level2` process wrapping the [FORCE level 2 processing system](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/). + +:::{.callout-note collapse="true" title="force_level2 process definition"} +```{python} +connection.describe_process("force_level2") +``` +::: + +The openEO processes exposes most of the options that can be set in the FORCE parameter file. + + + +## Time Series Analysis + +The `force_tsa` process wraps the [Time Series Analysis](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) module. + +:::{.callout-note collapse="true" title="force_tsa process definition"} +```{python} +connection.describe_process("force_tsa") +``` +::: + +# openEO job management + +Besides the openEO processes which make up the process graph, the openEO client provides functionality to submit, schedule and control jobs. +Please refer to the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html) (or the documentation of your preferred openEO client) for a detailed explanation of job management using openEO. + + +Name | Qualified name (Python client) | Purpose +-----|----------------|-------- +`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend +`start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion +`start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution +`get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files) +`download_files` | [`openeo.rest.job.JobResults.download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) | Download all assets (files) produced by a job +`download_file` | [`openeo.rest.job.JobResults.download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) | Download a singular asset (file) produced by a job + + +## Monitoring progress + +When using `start_and_wait` to start a job, status updates will automatically be printed to the command line or jupyter output cell. + +Alternatively, progress and logs can be monitored using the openEO web interface at [https://openeo.dataspace.copernicus.eu](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files) or use the client as explained e.g., in the [Python client documentation](https://open-eo.github.io/openeo-python-client/batch_jobs.html#create-a-batch-job). diff --git a/docs/force/guide/query.qmd b/docs/force/guide/query.qmd new file mode 100644 index 0000000..62f443e --- /dev/null +++ b/docs/force/guide/query.qmd @@ -0,0 +1,208 @@ +--- +format: + html: + page-navigation: true +execute: + freeze: true +--- + +# Selecting input data + +When running FORCE level 2 locally, we are responsible to manually provide the necessary input data and reference their full path in the `FILE_QUEUE` parameter of the [FORCE level 2 parameter file](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html). + +When running on CDSE, we do not have access to the excuting environment and its filesystem directly. +Downloads, file management and creation of the file queue are managed by the integration. +To specify which files should be processed, we provide a [STAC](https://stacspec.org/en) document ([Catalog](https://github.com/radiantearth/stac-spec/blob/master/catalog-spec/catalog-spec.md), [Collection](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md), [Item](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) or [ItemCollection](https://github.com/radiantearth/stac-api-spec/blob/main/fragments/itemcollection/README.md)) to the process. + +:::{.callout-warning} +Make sure that the STAC document provided to `force_level2` only references the files that should actually be processed. The FORCE integration will attempt to download all products referenced in the document, so it is inadvisable to provide a reference to the entire Sentinel-2 L1C collection on CDSE for example +::: + +## STAC + +The integrated FORCE and openEO more generally uses the [SpatioTemporal Asset Catalog (STAC) specification](https://stacspec.org/en) to identify (groups of) EO products and annotate them with metadata. + +If you are unfamiliar with STAC, we recommend that you familiarize yourself with the basic components of the specification (What are catalogs, items and feature collections?) [on the official STAC specification website](https://stacspec.org/en) before proceeding. + + +**The CDSE Sentinel-2 L1C STAC collection which we use as input for FORCE level 2 processing, is hosted at [https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c](https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c)** + +In the following, we show how this collection may be queried to to obtain inputs for FORCE level 2 processing. + + +## Querying STAC catalogs/collections + +FORCE level 2 can work with any STAC URL or document referencing input products of the correct type (Sentinel-2 L1C). The following describes how to make a query to the CDSE STAC catalog. +Be advised to query the CDSE STAC collection of L1C, as data access will be faster than external archives and the integration makes some assumptions about the metadata provided in the STAC items describing +the products. + +It is recommended to make the query using `pystac_client` for now, as using the `query_stac` process limits us to very small data sets See below for more details. + +### Area of interest + +As an example, we selected an area in northern Italy, around the city of Modena. + + +```{python} +w,s,e,n = 10.386, 44.437, 11.423, 44.973 +spatial_extent = { "west": w, "south": s, "east": e, "north": n} +# ATTENTION: Inclusion of the second date in the search depends on the query method (see below). +temporal_extent=["2026-04-17", "2026-05-01"] +``` + +:::{.callout-warning} +When defining the temporal extent as shown above, be aware that different methods may interpret time ranges differently. OpenEO's `query_stac` behaves just like [`load_collection`](https://openeo.org/documentation/1.0/processes.html#load_collection): The second instant is **excluded** from the search. + +Contrarily, `pystac_client`'s [`search`](https://pystac-client.readthedocs.io/en/stable/api.html#pystac_client.Client.search) method **includes** the second date when given as a list. Note that `pystac_client`'s search has many more options for specifying time ranges. +::: + +```{python} +#| code-fold: true +#| code-summary: "AOI plot" +import matplotlib.pyplot as plt +import contextily as ctx +import geopandas as gpd +from shapely import Polygon + +def plot_area_of_interest( + w, s, e, n, *, + context=5, + color="red", + linewidth=2, + title="Spatial Extent", + figsize=(12, 8), + show_tick_labels=True, + ): + polygon = Polygon([(w, s), (e, s), (e, n), (w, n)]) + gdf = gpd.GeoDataFrame(geometry=[polygon], crs="EPSG:4326") + fig, ax = plt.subplots(1, 1, figsize=figsize) + gdf.plot(edgecolor=color, facecolor="none", linewidth=linewidth, ax=ax) + gdf.plot(edgecolor=color, facecolor="none", linewidth=linewidth, ax=ax) + ax.set_xlim(w - context, e + context) + ax.set_ylim(s - context, n + context) + if not show_tick_labels: + ax.set_xticks([]) + ax.set_yticks([]) + ctx.add_basemap(ax, source=ctx.providers.OpenStreetMap.Mapnik, crs=gdf.crs) + fig.suptitle(title) + fig.tight_layout() + +plot_area_of_interest( + w=w, + s=s, + e=e, + n=n, + context=3, + figsize=(10, 6), + title=f"Spatial Extent: Modena", + show_tick_labels=False +) +``` + +### Querying a catalog + +:::{.panel-tabset group="query_engine"} + +#### openEO + +```{python} +import openeo + +L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c" +STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1" + +connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc() +query_pg = openeo.processes.process( + "query_stac", + arguments={ + "url": L1C_COLLECTION_URL, + "temporal_extent": temporal_extent, + "spatial_extent": spatial_extent, + } +) +``` + + + +```{python} +#| echo: false +connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() +``` + +We can directly use `query_pg` as a parameter for FORCE level 2, using openEO's process chaining. +To inspect the query results ahead of time, we may want to execute it ahead of time. +If we do not do this, execution of the query will be delayed until we submit the entire process graph (including processing with FORCE). + +```{python} +query_res = connection.execute(query_pg) + +print(f"Found products ({len(query_res['features'])}):") +print("\n".join([f["id"] for f in query_res["features"]]), "\n") +``` + + +#### Pystac client {.active} + + +```{python} +import pystac_client + +L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c" +STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1" + +client = pystac_client.Client.open("https://stac.dataspace.copernicus.eu/v1") +search = client.search( + datetime=temporal_extent, + collections=["sentinel-2-l1c"], + bbox=[w, s, e, n] +) +item_collection = search.item_collection() + +print("Found items:") +for item in search.items(): + print(item.id) + +``` + +::: + +:::{.content-visible when-format="html"} + +### Inspecting query results + +We may use pystac to inspect or process the feature collection further. In this example, we inspect the first of the returned items: + + +:::{.panel-tabset group="query_engine"} + +#### openEO + + +```{python} +import pystac + +item = pystac.Item.from_dict(query_res["features"][0]) +item +``` + +#### Pystac client {.active} + +```{python} +next(iter(search.items())) +``` + +::: +::: + + +## Other methods + +CDSE provides a [graphical STAC browser](https://browser.stac.dataspace.copernicus.eu/collections/sentinel-2-l1c) that can be used to search for and select singular items. diff --git a/docs/force/guide/tsa.qmd b/docs/force/guide/tsa.qmd new file mode 100644 index 0000000..4b53b4c --- /dev/null +++ b/docs/force/guide/tsa.qmd @@ -0,0 +1,313 @@ +--- +format: + html: + page-navigation: true +execute: + freeze: true +--- +# FORCE Time Series Analysis + + + +## Recap + +:::{.callout-warning} +You should use `pystac_client` based querying if your inputs encompass more than just a handful of items +::: + +:::{.callout-tip collapse="true" title="Selecting input products"} + +```{python} + +L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c" +STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1" + +w,s,e,n = 10.386, 44.437, 11.423, 44.973 +spatial_extent = { "west": w, "south": s, "east": e, "north": n} +# ATTENTION: Inclusion of the second date in the search depends on the query method (see below). +temporal_extent=["2026-04-17", "2026-05-01"] + +``` + +:::{.panel-tabset group="query_engine"} + + +## openEO + +```{python} +import openeo +connection = openeo.connect("openeo-staging.dataspace.copernicus.eu").authenticate_oidc() + +query_pg = openeo.processes.process( + "query_stac", + arguments={ + "url": L1C_COLLECTION_URL, + "temporal_extent": temporal_extent, + "spatial_extent": spatial_extent, + } +) +``` + +## Pystac client {.active} + +```{python} +import pystac_client + +client = pystac_client.Client.open(STAC_ROOT_URL) +search = client.search( + datetime=temporal_extent, + collections=["sentinel-2-l1c"], + bbox=[w, s, e, n] +) +item_collection = search.item_collection() +``` + +::: + +::: + +:::{.callout-tip collapse="true" title="level2 processing"} +:::{.panel-tabset group="query_engine"} + +#### openEO + +```{python} +import getpass +from openeo.rest.stac_resource import StacResource +from openeo.internal.graph_building import PGNode + +# Make sure to choose a unique name +processing_name = "Modena" + +force_l2_stac_resource = StacResource( + graph=PGNode( + process_id="force_level2", + arguments={ + "stac_document": query_pg, + "name": processing_name, + "do_brdf": True, + } + ), + connection=connection, +) + +merge = f"{getpass.getuser()}/experiments/{processing_name}" +force_l2_stac_resource.export_workspace( + workspace="apex-force-results-workspace", + merge=merge, +); +``` + +```{python} +#| echo: false +force_l2_stac_resource_openeo = force_l2_stac_resource +``` + +#### Pystac client {.active} + +```{python} +#| echo: false +import pystac + +def transform_item_collection_to_catalog_with_links(item_collection): + catalog = pystac.Catalog( + id="item-collection-catalog", + description="Catalog from item collection", + ) + for item in item_collection.items: + item_link = pystac.Link( + rel=pystac.RelType.ITEM, + target=item.self_href, + media_type=pystac.MediaType.GEOJSON, + title=f"Item {item.id}", + ) + catalog.add_link(item_link) + return catalog + +query_catalog = transform_item_collection_to_catalog_with_links(item_collection) +``` + + +```{python} +from openeo.internal.graph_building import PGNode +query_catalog = transform_item_collection_to_catalog_with_links(item_collection) + +processing_name = "Modena" + +force_l2_stac_resource = StacResource( + graph=PGNode( + process_id="force_level2", + arguments={ + "stac_document": query_catalog.to_dict(), + "name": "Modena", + "do_brdf": True, + } + ), + connection=connection, +) + +merge = f"{getpass.getuser()}/experiments/{processing_name}" +print(merge) +force_l2_stac_resource.export_workspace( + workspace="apex-force-results-workspace", + merge=merge, +) +``` + + +::: + +```{.python} +l2_job = force_l2_stac_resource.create_job() +l2_job.start() +``` + +```{python} +#| echo: false +l2_job = connection.job("j-260527114522498d959f20eff3fc9b78") +``` + + +::: + +## Time Series Analysis + +The openEO process for FORCE Time Series Analysis (TSA) is called `force_tsa`. Like the [level 2](level2.qmd) process, it takes most of the parameters usually passed to the [TSA module of FORCE](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/) using the [parameter file](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html). Parameters use `snake_case` format instead of `SCREAMING_SNAKE_CASE` (used in FORCE parameter files). + +Details of parametrization are discussed in the [dedicated article](parametrization.qmd). + +:::{.callout-note collapse="true" title="FORCE TSA parameters"} +```{python} +connection.describe_process("force_tsa") +``` +::: + +### Using FORCE level 2 results as an input + +The process graph for FORCE TSA built in the same way as the level 2 process graph is built. +The main input for FORCE TSA is the STAC catalog (or item) from the level 2 processing. + +We can access the STAC catalog produced by our level 2 job in the workspace, +described in [level 2 processing article](level2.qmd#Accessing results). + +It is important to remember the `merge` paramter passed to `export_workspace`, this is the path under which we can find the catalog. + + +```{python} +processing_name = "Modena" +merge = f"{getpass.getuser()}/experiments/{processing_name}" +print(merge) +``` + +Remember, the dedicated workspace for FORCE has the URL + + +```{python} +WORKSPACE_URL = "https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me" +``` + + +### TSA Parameters + + +Some parameters can be set a-priori or from prior knowledge of our level 2 processing parameters +```{python} +temporal_extent=["2026-04-17", "2026-04-27"] +``` + +For other parameters, we may want to inspect the metadata of the level 2 processing. +For example, we can extract the tiles from the asset names of the STAC item describing the level 2 datacube. + + +```{python} +merge = f"{getpass.getuser()}/experiments/{processing_name}" +l2_catalog_url = f"{WORKSPACE_URL}/{merge}/catalog.json" +print(l2_catalog_url) +l2_catalog = pystac.Catalog.from_file(l2_catalog_url) +l2_item = next(l2_catalog.get_items()) # there is just one item +l2_item +``` + +We can extract the tiles manually or programmatically +```{python} +tiles = set([k.split(".")[0] for k in l2_item.assets.keys() if k.endswith("BOA")]) +tile_x = set(int(tile[1:5]) for tile in tiles) +tile_y = set(int(tile[7:]) for tile in tiles) +tile_x, tile_y +``` + +### Process Graph + +The main process is `force_tsa`. + +```{python} +from openeo.rest.stac_resource import StacResource +from openeo.internal.graph_building import PGNode + + +force_tsa_stac_resource = StacResource( + graph=PGNode( + process_id="force_tsa", + arguments={ + "stac_url": l2_catalog_url, + "name": "Modena_TSA", + "date_range": temporal_extent, + "x_tile_range": [min(tile_x), min(tile_x)], + "y_tile_range": [min(tile_y), max(tile_y)], + "stm": ["AVG"], + "output_stm": True, + # ... more TSA parameters (optional) + } + ), + connection=connection, +) +``` + +We can then create and start the job as before: + +```{.python} +tsa_job = force_tsa_stac_resource.create_job(title=f"FORCE TSA {processing_name}") +tsa_job.start() # use start_and_wait to block until finished +``` + +Inspecting STAC metadata and downloading results works in the same way as for level 2 processing: + +```{python} +#| echo: false +tsa_job = connection.job("j-260515103226413e919ee5d9934c9693") +``` + +```{.python} +tsa_catalog_url = extract_catalog_url_from_job_logs(tsa_job.logs()) +tsa_catalog = pystac.Catalog.from_file(tsa_catalog_url) +tsa_item = next(tsa_catalog.get_items()) +tsa_item +``` + +Downloading assets as well: + +```{.python} +tsa_results = l2_job.get_results() +tsa_results +``` + +```{.python} +tsa_results.download_files("force-tsa-results") +``` + + + +### Further processing + +Because level 2 results are provided by a STAC URL to the TSA process, it is possible run multiple TSA jobs with the same level 2 input, without expensively reprocessing the level 2 data. +We just create a new TSA process graph and submit it, passing the same level 2 data cube URL. + +Remember that the job assets will be stored for some time on the backend which allows you to do processing over a span of a few days, but will not be stored permanently. Please download and archive results you wish to keep on your own infrastructure. diff --git a/docs/force/img/onion-concrete.svg b/docs/force/img/onion-concrete.svg new file mode 100644 index 0000000..8dd44a4 --- /dev/null +++ b/docs/force/img/onion-concrete.svg @@ -0,0 +1,347 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/force/img/src/force-eoap.d2 b/docs/force/img/src/force-eoap.d2 new file mode 100644 index 0000000..4b60841 --- /dev/null +++ b/docs/force/img/src/force-eoap.d2 @@ -0,0 +1,121 @@ +style { + fill: transparent +} +(*** -> ***)[*]: { + style { + stroke-width: 6 + font-size: 30 + stroke: "#111F3D" + font-color: black + } +} +(*** <- ***)[*]: { + style { + stroke-width: 6 + font-size: 30 + stroke: "#47B149" + font-color: white + } +} + +vars: { + d2-config: { + layout-engine: ELK + } +} +direction: right + +classes: { + workflow: { + style: { + font-size: 30 + fill: "#EAC868" + border-radius: 10 + } + } + CLT: { + style { + fill: "#f7e9c3" + font-size: 30 + border-radius: 10 + } + } + script: { + style { + fill: "#B5E0B6" + font-size: 30 + } + } + force: { + style { + fill: "#47B149" + font-size: 30 + #font-color: white + } + } + group: { + style { + fill: "#BCE6F8" + font-size: 30 + } + } + main: { + style { + fill: "#21ACE9" + bold: true + font-size: 40 + } + } +} + + +eoap { + label: Force EOAP + class: main + Docker { + class: group + force { + class: force + label: FORCE (base image) + } + l2-wrapper { + class: script + label: "Wrapper: Level 2" + } + tsa-wrapper { + class: script + label: "Wrapper: TSA" + } + } + CWL { + class: group + label: "CWL description" + l2-wf { + label: FORCE Level 2 Workflow + class: workflow + } + tsa-wf { + label: FORCE TSA Workflow + class: workflow + } + l2-clt { + label: FORCE Level 2 CommandLineTool + class: CLT + } + staging-clt { + label: Staging CommandLineTool + class: CLT + } + tsa-clt { + label: FORCE TSA CommandLineTool + class: CLT + } + tsa-wf -> tsa-clt + tsa-wf -> staging-clt + l2-wf -> l2-clt + l2-wf -> staging-clt + } +} + +eoap.cwl.l2-clt -> eoap.docker: references +eoap.cwl.tsa-clt -> eoap.docker: references diff --git a/docs/force/img/src/onion-concrete.typ b/docs/force/img/src/onion-concrete.typ new file mode 100644 index 0000000..dcba239 --- /dev/null +++ b/docs/force/img/src/onion-concrete.typ @@ -0,0 +1,77 @@ +#import "onion-def.typ" + +#set align(center) +#set page(fill: none, width: auto, height: auto) + +#let groups = ( + FORCE: blue, + EOAP: red, + openEO-backend: green, +) + +#let od = onion-def.onion-diagram( + radius: 7cm, + line-width: 2pt, + groups: groups, + layers: ( + ( + label: "FORCE", + group: "FORCE", + index: 0 + ), + ( + label: "Docker", + group: "EOAP", + index: 0 + ), + ( + label: "CWL-CLTool", + group: "EOAP", + index: 1 + ), + ( + label: "CWL-Workflow", + group: "EOAP", + index: 2 + ), + ( + label: "openEO CWL integration", + group: "openEO-backend", + index: 0 + ), + ( + label: "openEO process", + group: "openEO-backend", + index: 1 + ), + ( + label: "openEO client", + group: none, + ), + ), +) + +#od + +#let legend(groups) = { + let single-box(label, color) = box( + width: 2in, + height: 0.7in, + fill: color, + inset: 6pt, + radius: 6pt, + align(center + horizon)[ + #text(fill: white, weight: "bold", size: 18pt)[#label] + ], + ) + let boxes = () + for key in groups.keys() { + boxes.push(single-box(key, groups.at(key))) + } + stack( + dir: ttb, + spacing: 4pt, + ..boxes) +} + +#legend(groups) diff --git a/docs/force/img/src/onion-def.typ b/docs/force/img/src/onion-def.typ new file mode 100644 index 0000000..489b870 --- /dev/null +++ b/docs/force/img/src/onion-def.typ @@ -0,0 +1,146 @@ +#import "@preview/cetz:0.3.4": canvas, draw + +// Onion diagram template using CeTZ +// +// Features: +// - Concentric layered structure +// - Hue grouping for related layers +// - Independent layers visually distinct +// - Parametrizable overall size and line width +// - Configurable labels and colors +// +// Example usage: +// +// #onion-diagram( +// radius: 6cm, +// line-width: 1.5pt, +// layers: ( +// ( +// label: "Core", +// group: "platform", +// ), +// ( +// label: "Infrastructure", +// group: "platform", +// ), +// ( +// label: "Services", +// group: "platform", +// ), +// ( +// label: "Business Logic", +// group: "business", +// ), +// ( +// label: "UI", +// group: "business", +// ), +// ( +// label: "External APIs", +// group: none, +// ), +// ), +// ) + +#let onion-diagram( + radius: 5cm, + line-width: 1pt, + label-size: 10pt, + //font: "Linux Libertine", + // Group → base hue mapping + groups: ( + platform: blue, + business: green, + security: orange, + data: purple, + ), + // Layer specification + layers: (), +) = { + let n = layers.len() + + // Avoid division by zero + if n == 0 { + panic("onion-diagram requires at least one layer") + } + + let step = radius / n + + // Compute layer styling + let layer-style(i, layer) = { + let group = layer.at("group", default: none) + + if group == none { + ( + stroke: ( + paint: gray.darken(20%), + thickness: line-width, + ), + fill: luma(92%), + text-fill: black, + ) + } else { + let base = groups.at(group, default: blue) + + // Related layers share hue but vary in lightness + let t = layer.index + + let fill = if t == 0 { + base.lighten(35%) + } else if t == 1 { + base.lighten(50%) + } else { + base.lighten(65%) + } + + ( + stroke: ( + paint: base.darken(15%), + thickness: line-width, + ), + fill: fill, + text-fill: black, + ) + } + } + + canvas({ + import draw: * + + // Draw outer → inner + for i in range(n) { + let idx = n - i - 1 + let layer = layers.at(idx) + + let r = step * (idx + 1) + + let style = layer-style(idx, layer) + + circle( + (0, 0), + radius: r, + stroke: style.stroke, + fill: style.fill, + ) + + // Label position: + // placed at top of each ring + let label_r = if idx == 0 { + r / 2 + } else { + r - step / 2 + } + + content( + (0, label_r), + text( + size: label-size, + // font: font, + fill: style.text-fill, + weight: "bold", + layer.label, + ), + ) + } + }) +} diff --git a/docs/force/img/src/openeo-force-parameter-journey.d2 b/docs/force/img/src/openeo-force-parameter-journey.d2 new file mode 100644 index 0000000..4ef6cf8 --- /dev/null +++ b/docs/force/img/src/openeo-force-parameter-journey.d2 @@ -0,0 +1,168 @@ +style { + fill: transparent +} +(*** -> ***)[*]: { + style { + stroke-width: 6 + font-size: 30 + stroke: "#111F3D" + font-color: black + } +} +(*** <- ***)[*]: { + style { + stroke-width: 6 + font-size: 30 + stroke: "#47B149" + font-color: black + } +} + +vars: { + d2-config: { + layout-engine: ELK + } +} +#direction: right + +classes: { + stop: { + style: { + font-size: 30 + fill: "#EAC868" + border-radius: 10 + bold: true + } + } + CLT: { + style { + fill: "#f7e9c3" + font-size: 30 + border-radius: 10 + } + } + eoap: { + style { + fill: "#B5E0B6" + font-size: 30 + } + } + force: { + style { + fill: "#47B149" + font-size: 30 + #font-color: white + } + } + group: { + style { + fill: "#BCE6F8" + font-size: 30 + } + } + main: { + style { + fill: "#21ACE9" + bold: true + font-size: 40 + } + } +} + + +py-client { + class: stop + label: openEO Python client +} +py-parameters: |python + stac_resource = openeo.rest.stac_resource.StacResource( + graph=openeo.internal.graph_building.PGNode( + process_id="force_level2", + # Pass any parameters accepted by FORCE level 2 here + arguments={ + "stac_document": query_pg, + "do_brdf": True, + "output_ovv": True, + # ... + } + ), + connection=connection, + ) +| + +process-graph { + class: stop + label: Process Graph +} +pg-parameters: |json + { + "force_level2" { + "stac_document": "query_stac_1", + "do_brdf": true, + "output_ovv": true, + ... + } + } + | + +CWL { + class: stop + label: CWL input specification +} +cwl-parameters:|json +{ + "stac_document": "query_stac_1", + "do_brdf": true, + "output_ovv": true, + ... +} +| + + +EOAP { + class: eoap + staging { + class: stop + label: Staging tool + + } + wrapper { + class: stop + label: Level 2 wrapper script + } + staging-parameters: |bash + stac-staging \ + --string '{"stac_document": {"features": [{"assets": {"B01":[...]' \ + --method recursive + | + + wrapper-parameters: |bash + force-l2-wrapper.sh \ + --do_brdf true \ + --output_ovv true \ + # ... + | + + prm { + class: stop + label: FORCE parameter file + } + prm-parameters:|bash + FILE_QUEUE=inputs/tds.txt + DO_BRDF=TRUE + OUTPUT_OVV=TRUE + # ... + | + wrapper -> prm: wrapper +} + +py-client -> process-graph: "create_job" +process-graph -> cwl: "openEO CWL runner integration" +cwl -> EOAP.staging: "force-l2-workflow.cwl" +cwl -> EOAP.wrapper: "force-l2-workflow.cwl" + +py-parameters -> pg-parameters +pg-parameters -> cwl-parameters +cwl-parameters -> EOAP.staging-parameters +cwl-parameters -> EOAP.wrapper-parameters +EOAP.wrapper-parameters -> EOAP.prm-parameters diff --git a/docs/force/img/src/openeo-force-processes.d2 b/docs/force/img/src/openeo-force-processes.d2 new file mode 100644 index 0000000..657bfca --- /dev/null +++ b/docs/force/img/src/openeo-force-processes.d2 @@ -0,0 +1,104 @@ +style { + fill: transparent +} +(*** -> ***)[*]: { + style { + stroke-width: 6 + font-size: 26 + stroke: "#47B149" + font-color: white + } +} +(*** <- ***)[*]: { + style { + stroke-width: 6 + font-size: 26 + stroke: "#47B149" + font-color: white + } +} + +***.style.font-size: 24 +vars: { + d2-config: { + layout-engine: ELK + } +} +direction: right +classes: { + tl-component: { + style: { + font-size: 36 + fill: "#EAC868" + border-radius: 10 + bold: true + } + } + sub-component: { + style.fill: "#21ACE9" + } +} +#direction: right + +External STAC catalog { + #near: top-right + Sentinel2-L1C + class: tl-component +} + +openEO client: { + class: tl-component + Job Management { + class: sub-component + create_job + start + } + Result Access { + class: sub-component + get_results + download_files + } + + Process Graph definition { + class: sub-component + inner: { + label: "..." + } + } +} + +openEO backend { + class: tl-component + CWL runner : { + class: sub-component + } + openEO Processes { + class: sub-component + force_level2 + force_tsa + query_stac + + query_stac -> force_level2: provides input + } + + CWL runner { + FORCE EOAP + } + + CWL runner <- openEO Processes: control +} + + + +openEO client.Process Graph definition -> openEO backend: Control Workflow +openEO client.Job Management -> openEO backend +openEO client.Result Access <- openEO backend: outputs (FORCE datacube, STAC) + +openEO backend.openEO processes.query_stac -> External STAC catalog: { style.stroke-dash: 3 } + +CDSE S3 { + class: tl-component +} + +openEO backend.openEO processes.force_level2 -> CDSE S3: store results +openEO backend.openEO processes.force_tsa -> CDSE S3: Access results diff --git a/docs/force/index.qmd b/docs/force/index.qmd new file mode 100644 index 0000000..e9aed50 --- /dev/null +++ b/docs/force/index.qmd @@ -0,0 +1,99 @@ +--- +title: "FORCE with openEO on CDSE" +format: + html: + code-line-numbers: true +--- + +The integration of the [FORCE](https://force-eo.readthedocs.io/en/latest/) processing engine in CDSE provides access to [a subset of FORCE's processing functionality](feature_overview.qmd) via the [openEO API on CDSE](https://eoap.github.io/). + +The integrated FORCE can be used to process on cloud infrastructure using an openEO client ([Python](https://open-eo.github.io/openeo-python-client/), [R](https://open-eo.github.io/openeo-r-client/) or [Javascript](https://open-eo.github.io/openeo-js-client/latest/)) and the openEO web-interface to create and monitor jobs. +Input Staging and [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/en) generation are handled automatically. + +The integration has been performed by [Brockmann Consult GmbH](https://www.brockmann-consult.de/) in the context of the European Space Agency's (ESA) [*Application Propagation Environments* (APEx)](https://apex.esa.int/) initiative. + +## FORCE + +![Official FORCE logo, [https://force-eo.readthedocs.io/en/latest/_images/force.png](https://force-eo.readthedocs.io/en/latest/_images/force.png)](https://force-eo.readthedocs.io/en/latest/_images/force.png) + +FORCE (Framework for Operational Radiometric Correction for Environmental Monitoring) is an EO processing engine developed by [Prof. Dr. David Frantz](https://www.uni-trier.de/universitaet/fachbereiche-faecher/fachbereich-vi/faecher/geoinformatik/team/prof-dr-d-frantz) +(*Geoinformatics - Spatial Data Science*, Trier University). + +FORCE includes a large number of processing features, including the generation of Analysis Ready Data (ARD) Data Cubes and higher level data analysis processes. See the FORCE ['About' page](https://force-eo.readthedocs.io/en/latest/about.html) page for a full overview of FORCE's features. + + +::: {.callout-note} +## FORCE is … + +… an all-in-one processing engine for medium-resolution Earth Observation image archives. FORCE uses the data cube concept to mass-generate Analysis Ready Data, and enables large area + time series applications. With FORCE, you can perform all essential tasks in a typical Earth Observation Analysis workflow, i.e., going from data to information. + + +- [Force documentation](https://force-eo.readthedocs.io/en/latest/#force-is) +::: + +Details of FORCE functionality and validation are described in a number of [scientific publications](https://force-eo.readthedocs.io/en/latest/refs.html). + +## ...with openEO... + +This project makes it possible to use (the) FORCE through the [Copernicus Data Space Ecosystem (CDSE)](https://dataspace.copernicus.eu/) [openEO Service](https://dataspace.copernicus.eu/analyse/openeo). + +Using, for example, the [openEO Python client](https://openeo.org/documentation/1.0/python/) to interact with the backend, FORCE can be parametrized and executed using the familiar openEO machinery. + +::: {.callout-warning} +The integrated FORCE does not use the openEO data cube structure. +FORCE has its own file-system-based [data cube](https://force-eo.readthedocs.io/en/latest/howto/datacube.html) model. The openEO integration makes it possible to control FORCE through the openEO API. It does not merge the datacube models of FORCE and openEO, so foreign openEO processes cannot work directly on FORCE data cubes. +::: + + +```{.python} +import openeo + +connection = openeo.connect("https://openeo.dataspace.copernicus.eu").authenticate_oidc() + +stac_item_url = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521" +stac_resource = openeo.rest.stac_resource.StacResource( + graph=openeo.internal.graph_building.PGNode( + process_id="force_level2", + arguments=dict( + stac_url=stac_item_url, + do_brdf=True, + output_ovv=True, + # ... other parameters supported by FORCE level 2 + ) + ), + connection=connection, +) +job = stac_resource.create_job(title="FORCE level 2") +job.start_and_wait() +job.get_results() +l2_results.download_files("force-ARD") +``` + +For a full example, see [the guide](guide/intro.qmd). + +With the integrated FORCE, you can + +- Run the force level 2 processing system to generate Analysis Ready Data (example above) +- Run the Time Series Analysis module of the FORCE higher level processing system (hlps) +- Download data cubes in FORCE's native datacube format + +For an overview of the FORCE functionality made available via openEO, see [features](feature_overview.qmd). + +## ...on CDSE + +FORCE runs on the CDSE infrastructure using the same data archive available to other CDSE services. + + +## EO Application Package + +:::{.callout-note} +This section contains information on implementation details, you can +safely ignore it. +::: + + +FORCE is integrated into CDSE openEO using the built-in support for EO Application Packages following the [OGC Best Practice for Earth Observation Application Packages](https://docs.ogc.org/bp/20-089r1.html) (EOAP). +EOAPs are implemented as a combination of a Docker Image and a [Common Workflow Language](https://www.commonwl.org/) (CWL) document. + +![integration onion](img/onion-concrete.svg) + diff --git a/docs/force/styles.css b/docs/force/styles.css new file mode 100644 index 0000000..8c9cc4d --- /dev/null +++ b/docs/force/styles.css @@ -0,0 +1,14 @@ +/* css styles */ +.cell-output-pre { + background: #f6f8fa; + border-left: 4px solid #6c757d; + padding: 0.75rem 1rem; + border-radius: 0.25rem; + margin-top: 0.5rem; +} + +.cell-output { + margin-bottom: 1rem; + border-left: 4px solid #6c757d; + padding: 0.75rem 1rem; +}