🔴 downshape is an R research compendium exposing functions to search, download, preprocess and bias-adjust CMIP6 data discovered via the ESGF Search RESTful API. Major preprocess step made with CDO swofware. This compendium allow to download observed data come from Copernicus website or other website with URL of variable. We can choose to download bathymetry variable come from CDO.
graph LR
subgraph Graph
direction LR
xb5a86a9ebe0b7b97(["ano_vars"]):::uptodate --> x9b87618f04cad4de(["anomaly"]):::outdated
xd2707862cf793d3a(["connectPip_copernicus"]):::outdated --> x9b87618f04cad4de(["anomaly"]):::outdated
xd2707862cf793d3a(["connectPip_copernicus"]):::outdated --> x37179b61a203cbd3(["grad_copernicus"]):::outdated
xb5817e1daca13dcc(["bathy_vars"]):::uptodate --> xe44467b2b079fc18(["renameVar_copernicus"]):::outdated
x35b468daf9281b76(["http_data"]):::uptodate --> xe44467b2b079fc18(["renameVar_copernicus"]):::outdated
xd7bca5ba4e5f539d(["obs_data"]):::outdated --> xe44467b2b079fc18(["renameVar_copernicus"]):::outdated
xd15c82dcb79a7c2e(["renameVar"]):::uptodate --> xe44467b2b079fc18(["renameVar_copernicus"]):::outdated
xec6283d15a25ed08(["remapCDO_copernicus"]):::outdated --> xd2707862cf793d3a(["connectPip_copernicus"]):::outdated
x9289bfb53112cf3b(["concatenate_copernicus"]):::outdated --> xec6283d15a25ed08(["remapCDO_copernicus"]):::outdated
xd15c82dcb79a7c2e(["renameVar"]):::uptodate --> xec6283d15a25ed08(["remapCDO_copernicus"]):::outdated
xfc6ed28680e5db72(["speedCompo_copernicus"]):::outdated --> xec6283d15a25ed08(["remapCDO_copernicus"]):::outdated
xca459201a27e8460(["vars_speed"]):::uptodate --> xec6283d15a25ed08(["remapCDO_copernicus"]):::outdated
x9289bfb53112cf3b(["concatenate_copernicus"]):::outdated --> xfc6ed28680e5db72(["speedCompo_copernicus"]):::outdated
xca459201a27e8460(["vars_speed"]):::uptodate --> xfc6ed28680e5db72(["speedCompo_copernicus"]):::outdated
xe44467b2b079fc18(["renameVar_copernicus"]):::outdated --> x9289bfb53112cf3b(["concatenate_copernicus"]):::outdated
x4301c707c2ab0cdc(["tab_parameters"]):::uptodate --> xd7bca5ba4e5f539d(["obs_data"]):::outdated
x9b87618f04cad4de(["anomaly"]):::outdated --> x86e690a15fc5d647(["regrid_copernicus"]):::outdated
x37179b61a203cbd3(["grad_copernicus"]):::outdated --> x86e690a15fc5d647(["regrid_copernicus"]):::outdated
x71e926299248ce3f(["http_vars"]):::uptodate --> x35b468daf9281b76(["http_data"]):::uptodate
xf5c6fd225c377a3d(["bathy_CDO"]):::uptodate --> xb5817e1daca13dcc(["bathy_vars"]):::uptodate
x21b09bf88a2ff147(["baseline_period"]):::uptodate --> x21b09bf88a2ff147(["baseline_period"]):::uptodate
x084994fb0e480676(["current_period"]):::uptodate --> x084994fb0e480676(["current_period"]):::uptodate
x625f066a5f205ec8(["deep_level"]):::uptodate --> x625f066a5f205ec8(["deep_level"]):::uptodate
xac02e5e58926353b(["experiments"]):::uptodate --> xac02e5e58926353b(["experiments"]):::uptodate
x55a14a7f5821bbec(["resotempo"]):::uptodate --> x55a14a7f5821bbec(["resotempo"]):::uptodate
xe73cbbcc20086ecd(["spat_reso"]):::uptodate --> xe73cbbcc20086ecd(["spat_reso"]):::uptodate
end
- 1️⃣ Import files necessary into data folder (information bellow)
- 2️⃣ Edit targets of *_targets.R* file into project parameters part and check if all “skip” argument into targets function are “FALSE” (or “TRUE” if you want skip some targets).
- 3️⃣ Select the part the part of *_targets.R* script you want run. Commit all targets of the part not used. It’s impossible to run the two parts in same time.
- 4️⃣ Run pipeline launching make.R script
| Columns database | Description |
|---|---|
| my_variable_name | Your variable name (outside api command). Each name must be unique |
| dataset_id | Check on Copernicus website |
| dataset_version | Check API request on Copernicus website |
| longitude_min | Extend of map downloaded |
| longitude_max | Extend of map downloaded |
| latitude_min | Extend of map downloaded |
| latitude_max | Extend of map downloaded |
| date_min | Date. Format Year-Month-Day hours:minutes:seconds (2017-11-25 12:20:00) |
| date_max | Date. Format Year-Month-Day hours:minutes:seconds (2017-11-25 12:20:00) |
| depth_min | Numeric. Depth in meters. If you want to download one depth layer, same value for “depth_min” and “depth_max”. If it’s a 3D variable (with depth dimension) the depth must be provided otherwise leave the cell empty |
| depth_max | Numeric. Depth in meters. If you want to download one depth layer, same value for “depth_min” and “depth_max”. If it’s a 3D variable (with depth dimension) the depth must be provided otherwise leave the cell empty |
| variable | Short name of variable downloaded. If several variable by product create one line by variable in the table |
| DOI | Check on Copernicus website |
heavy_check_mark: [copernicus_parameters.csv]: CSV file. Parameters of the data we wish to download, with one row per variable to be downloaded. Please refer to the table structure requirements below. If a variable is large, the Copernicus API may not be able to download it, and it will be necessary to divide the variable into several smaller time periods (use the “divide”, “subvar”, and “septime” arguments of the copernicus_download_api() function). If a parameter in this table does not exist for your product (for example: depth), simply leave the cell empty. Columns “date_min” and “date_max” must be equal to or later than the current_period target (in the _targets.R file). To create a baseline for applying the change factor method to a CMIP variable, there must be an overlap between “date_min”-“date_max” and the historical_period target.
✔️ [Mask_PA_variable.shp] : Shapefile. Mask of study area to crop environmental rasters with CDO. This shapefile will be convert to NETCDF file with gdal before being used by CDO.
✔️ [experiments] : To select ssp scenario downloaded (check on esgf website menu)
✔️ [vars] : To select cmip6 variables want to download (check on esgf website menu). Add component and speed variables (the both) there are into “vars_speed_cmip” target.
✔️ [freq] : To select frequence of cmip6 variables want to download (check on esgf website menu).
✔️ [historical_period] : List start and end time. To define min and max time borns of cmip6 data historical period (cdo format: “YYYY-MM-DDThh:mm:ss”). Complet ssp “historical” period would you like. It must existe a time overlap with time_span target. Into CMIP6 ssp “historical” stop at 2014 years, but to match baseline and ssp “historical” periods it’s necessary to merge historical and a short ssp periods (it’s the goal of mergeHistorical_cmip function).
✔️ [future_period] : List start and end time. To define min and max time borns of cmip6 data future period (cdo format: “YYYY-MM-DDThh:mm:ss”). Complet ssp period would you like (without ssp “historical”). It must existe a time overlap with time_span target (period downloaded) and entire overlap with baseline_period target (to create same period between baseline and historical). “start” vector correspond at “end” vector of historical_period target +1 day.
✔️ [time_span] : To select min and max time of cmip6 variables downloaded (format example:“1982-01-01T00:00:00Z”). It must existe a time overlap with historical_period and future_period targets. If you want to apply change factor method, min born correspond to min born of historical_period target and max born corespond to max born of future_period target.
✔️ [vars_speed_cmip] : Same informations of vars_speed target but for cmip6 variables. All vectors of this target list can to be empty.
✔️ [climato_period] : List of vectors. Allow to name and define climatology periods use to apply change factor method (climatology on cmip6 ssp). “name” vector is how will be called climatology period and correspond to “start” and “end” vector define time bornes of climatology period (cdo format: “YYYY-MM-DDThh:mm:ss”). “month_choose” vector allow to choose month conserved into the climatology mean calcul (used into mean_month() function). It must existe a time overlap with future_period target.
✔️ [resotempo] : Give temporal resolution of all variable (copernicus or cmip or both) you want to mean by week with remapCDO(…, monthWeek = “week”). Complete “reso” list with “week” (if variable have one layer by week), “day” (if variable have one layer by day), “month” (one by month), “hour1” (one by one hour), “hour6” (one by 6 hours), “FIXE” one layer for all NetCDF file. Into “vars” list given variable name into “my_variable_name” column of parameters_copernicus.csv (same of “tabname” into [renameVar]). For speed variables give the output variable names inside [vars_speed] (called “name”). If BATHY variable are create with [bathy_CDO], add “BATHY” with “FIXE” in this target.
✔️ [http_vars] : List. List of download path (NetCDF file) and variable names. First (or n) element of “http” vector correspond to first (or n) element of “name” vector. “Name” vector will be the name used to rename file name of variable. To download variables outsite copernicus website. Leave empty if not used.
✔️ [current_period] : List start and end time. To define min and max time of copernicus data current period (cdo format: “YYYY-MM-DDThh:mm:ss”). Allow to calibrate model (weekly temporal resolution) and create current projection (monthly temporal resolution). This period must overlap all variable periods downloaded into copernicus_parameters.csv.
✔️ [baseline_period] : List start and end time. Complet this target if you want create a baseline to apply change factor method, otherwise write “NULL” into “start” and “end” arguments. It allow to define min and max time borns of baseline period (cdo format: “YYYY-MM-DDThh:mm:ss”).
✔️ [match_name] : List of two vectors. Allow to correspondance between “copernicus” and “cmip” variables during calcul of delta in change factor method. List uniquelly variables you want bias-corrected. These vectors must to contain all variables name create during process (variable processed by speed_vars and deep_level targets).
✔️ [spat_reso] : List of number. Spatial resolution of initial variables is degraded to “reso” degrees resolution (resolution must be higher than all initial resolutions). All variables are interpolated because their grids must match, so you must specify the number of rows and columns of the new grid, “grid_nrow” and “grid_ncol” respectively.
✔️ [deep_level] : List start and end deep level by file created. To split variable to several files by deep level. First (or n) element of “start” vector correspond to first (or n) element of “end” vector. vectors of the list can be empty.
✔️ [renameVar] : List of “my_variable_name” column value into “copernicus_parameters.csv” table and new name you want inside netcdf file. First (or n) element of “tabname” vector correspond to first (or n) element of “newname” vector. Allow to rename into NETCDF file the variable juste after downloading step. This vectors gathers all variable names used to rename cmip6 and copernicus data just after download step. In others words, it allows to change name of “variable” column into parameters_copernicus.csv and “Variable_id” column into selected_datasets.csv for cmip6 data. If variable name have a name with “_“,”-“,”#“,” “, use this target to rename with one word without specials characters (otherwise CDO command doesn’t work). All vectors of this target list can to be empty. If bathymetry is downloaded by downloadCDO_bathy() it will be called”BATHY” and variable name inside the NetCDF file will be not rename (called “topo”).
✔️ [vars_speed] : List first, second components and output variable name to calcul copernicus variables speed with two components. First (or n) element of “compo1” vector correspond to first (or n) element of “compo2” vector. All variables must have unique component names. The first and second vector give the variable names into “my_variable_name” column of parameters_copernicus.csv (same of “tabname” into [renameVar]). The last vector (“name”) give the new name of speed variable created.
✔️ [bathy_CDO] : Logical. If “TRUE” bathymetry variable from CDO are downloaded into “data_copernicus” folder. The name of variable is “topo_cdo_NOAA_ETOPO2_2006_brut.nc”. If “FALSE” bathymetry doesn’t downloaded.
✔️ [ano_vars] : character. Name of the variable with which you want to calculate the anomaly. This calculation utilizes the climatology data for the same variable (create manually into data folder), and the filename of the climatology file must match the variable name in the output folder according to the following format. Monthly and weekly climatology must to be availables: ‘SST0x100_climatology_XX_XXX_week.nc’ and ‘SST0x100_climatology_XX_XXX_month.nc’. NetCDF and GRD extension are supported.
🚩 If you want skip copernicus_download_api() function : You can put variables (.nc) into folder “output/data_copernicus” with file called “copernicus_parameters_modified” create during downloading data. Variable want to be named NameVar_XXXX_XXX_XXX_XXXX_DateBegining-DateFinal.nc..
🚩 If you want skip download_cmip_data() function : You can put variables (.nc) into folder “output/data_cmip6” with subfolders by model and experiment. Variable want to be named NameVar_XXXX_model_experiment_XXXX_XXX_DateBegining-DateFinal.nc.
ℹ️ Variable with “1x100”, “0x1”, etc… into file name are integrated (depth mean) between values before and after the “x”. Variable file without “x” pattern into number are integrated on all depth available (if depth are available for the variable).
- 📂 output –[make.r]–
- 📄 dataset_found_before_filter.csv –[select_dataset()]–
- 📄 selected_datasets.csv –[select_dataset()]–
- 📂 data_cmip6 –[download_cmip_data()]–
- 📂 Model_name_download
–[download_cmip_data()]–
- 📂 Experiment_name_download
–[download_cmip_data()]–
- 📄 ModelName_ExperimentName_VarsName.sh –[download_cmip_data()]–
- 📄 Vars1.nc –[download_cmip_data()]–
- 📄 Vars2.nc –[download_cmip_data()]–
- 📄 VarsX.nc … –[download_cmip_data()]–
- 📂 Experiment_name_download
–[download_cmip_data()]–
- 📂 Model_name_download
–[download_cmip_data()]–
- 📂 data_cmip6_change_factor –[climato_cmip()]–
- 📂 climatology –[climato_cmip()]–
- 📂 Name_climatology (ex : 2030)
- 📂 Model_name_download
- 📂 Experiment_name_download
- 📄 Vars1.grd
- 📄 Vars2.grd
- 📄 VarsX.grd …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.grd
- 📂 Experiment_name_download
- 📂 Model_name_download
- 📂 Name_climatology (ex : 2030)
- 📂 historical_ssp_merged
–[mergeHistorical_cmip()]–
- 📂 Model_name_download
- 📂 Experiment_name_download
- 📄 Vars1.grd
- 📄 Vars2.grd
- 📄 VarsX.grd …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.grd
- 📂 Experiment_name_download
- 📂 Model_name_download
- 📂 variables_bias-corrected –[deltaCF()]–
- 📂 Name_climatology (ex : 2030)
- 📂 Model_name_download
- 📂 Experiment_name_download
- 📄 Vars1.grd
- 📄 Vars2.grd
- 📄 VarsX.grd …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.grd
- 📂 Experiment_name_download
- 📂 Model_name_download
- 📂 Name_climatology (ex : 2030)
- 📂 variables_bias-corrected_mean –[meanMod()]–
- 📂 Name_climatology (ex : 2030)
- 📂 Experiment_name_download
- 📂 Gradient –[grad_cmip()]–
- 📄 Vars1.grd
- 📄 Vars2.grd
- 📄 VarsX.grd …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.grd
- 📂 Experiment_name_download
- 📂 Name_climatology (ex : 2030)
- 📂 climatology –[climato_cmip()]–
- 📂 data_cmip6_remapped –[remapCDO_cmip()]–
- 📂 Model_name_download –[remapCDO_cmip()]–
- 📂 Experiment_name_download
–[remapCDO_cmip()]–
- 📄 Vars1.nc –[remapCDO_cmip()]–
- 📄 Vars2.nc –[remapCDO_cmip()]–
- 📄 VarsX.nc …–[remapCDO_cmip()]–
- 📄 VarsDepth1x100.nc …–[remapCDO_cmip()]–
- 📄 Vars_speed.nc –[speedCompo_cmip()]–
- 📂 Experiment_name_download
–[remapCDO_cmip()]–
- 📂 Model_name_download –[remapCDO_cmip()]–
- 📂 data_copernicus –[copernicus_download_api()]–
- 📄 copernicus_parameters_modified.csv –[copernicus_download_api()]–
- 📄 Vars1.nc –[copernicus_download_api()]–
- 📄 Vars2.nc –[copernicus_download_api()]–
- 📄 VarsX.nc … –[copernicus_download_api()]–
- 📂 data_copernicus_remapped
–[remapCDO_copernicus()]–
- 📂 month
- 📄 Vars1.nc –[remapCDO_copernicus()]–
- 📄 Vars2.nc –[remapCDO_copernicus()]–
- 📄 VarsX.nc …–[remapCDO_copernicus()]–
- 📄 VarsDepth1x100.nc –[remapCDO_copernicus()]–
- 📄 Vars_speed.nc –[speedCompo_copernicus()]–
- 📂 GRD –[connectPip()]–
- 📄 Vars1.nc
- 📄 Vars2.nc
- 📄 VarsX.nc …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.nc
- 📂 Gradient –[grad_copernicus()]–
- 📄 GVars1.nc
- 📄 GVars2.nc
- 📄 GVarsX.nc …
- 📄 GVarsDepth1x100.nc
- 📄 GVars_speed.nc
- 📂 week
- 📄 Vars1.nc –[remapCDO_copernicus()]–
- 📄 Vars2.nc –[remapCDO_copernicus()]–
- 📄 VarsX.nc … –[remapCDO_copernicus()]–
- 📄 VarsDepth1x100.nc –[remapCDO_copernicus()]–
- 📄 Vars_speed.nc –[speedCompo_copernicus()]–
- 📂 GRD –[connectPip()]–
- 📄 Vars1.nc
- 📄 Vars2.nc
- 📄 VarsX.nc …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.nc
- 📂 Gradient –[grad_copernicus()]–
- 📄 GVars1.nc
- 📄 GVars2.nc
- 📄 GVarsX.nc …
- 📄 GVarsDepth1x100.nc
- 📄 GVars_speed.nc
- 📂 baseline
- 📄 Vars1.nc –[remapCDO_copernicus()]–
- 📄 Vars2.nc –[remapCDO_copernicus()]–
- 📄 VarsX.nc … –[remapCDO_copernicus()]–
- 📄 VarsDepth1x100.nc –[remapCDO_copernicus()]–
- 📄 Vars_speed.nc –[speedCompo_copernicus()]–
- 📂 GRD –[connectPip()]–
- 📄 Vars1.nc
- 📄 Vars2.nc
- 📄 VarsX.nc …
- 📄 VarsDepth1x100.nc
- 📄 Vars_speed.nc
- 📂 Gradient –[grad_copernicus()]–
- 📄 GVars1.nc
- 📄 GVars2.nc
- 📄 GVarsX.nc …
- 📄 GVarsDepth1x100.nc
- 📄 GVars_speed.nc
- 📂 month
- 📂 data_copernicus_final (Same sub-folder as data_copernicus_remapped) –[regrid_copernicus()]–
- 📂 data_copernicus_final (Same sub-folder as data_cmip6_remapped) –[regrid_cmpi()]–