The Stack Data Uploader

📝 Note: In the commands and file snippets below placeholders are indicated using angled brackets, for example <STACK NAME>. You will need to substitute in appropriate values before running any commands.

📝 Note: Unless otherwise stated all paths listed in this readme are relative to the Deploy/stacks/dynamic/stack-data-uploader directory in the TheWorldAvatar repository.

Table of Contents
Introduction
Running the Stack Data Uploader
Datasets and Subsets
The Dataset Configuration File
Data Types
OBDA Mapping file
Using Specific Data Sets
Value by File Name
Processing Data Without Upload
Debugging the Stack Data Uploader in VSCode
Developing the Stack Data Uploader in VSCode
Troubleshooting

Introduction

The Stack Data Uploader is designed to make it easier to ingest static data files into a stack.

Running the Stack Data Uploader

0. Prerequisites

These are the same as listed in The Stack Manager.

1. Initialise the stack

You should initialise the stack by following the instructions in Spinning up a Stack.

2. Check the examples

Example dataset files can be found in the examples/datasets directory. Descriptions of each example can be found in the Example datasets section.

3. Copy in data files

The source files need to be copied into the inputs/data/ directory. The structure of this directory is described in the Datasets and subsets section. All data must be in a subdirectory two levels below data folder.

4. Create a configuration file

Create a JSON file in the inputs/config/ directory to define how the data is to be uploaded. The structure of this file is described in the The Dataset configuration file section.

5. Running the data uploader

From a terminal in the stack-data-uploader directory, start the stack-data-uploader container by running the following:

./stack.sh start <STACK NAME>

Datasets and Subsets

Data files are grouped into datasets, each of which has its own configuration file and data directory.

By default all dataset configuration files in the inputs/configs/ are read by the data uploader. When a dataset's name matches with that of the stack then only that configuration file and its external datasets will be loaded.

Below is an example where there are two datasets. One of which (dataset1) contains one data subset and another (dataset2) that contains two data subsets, each with their own subdirectory. Note that every data file exists in a subdirectory, even if there are no sibling data on the same level. Data will not be uploaded unless it is two levels below data in a subdirectory.

inputs/
  config/               # Directory in which the dataset configuration files should be stored
    dataset1.json       # Configuration file for dataset1
    dataset2.json       # Configuration file for dataset2
  data/                 # Directory in which the data files should be stored
    dataset1/           # Data directory for dataset1
      datasubset1/     # Data subdirectory for data subset1
        data.csv        # Data file for dataset1
    dataset2/           # Data directory for dataset2
      datasubset2/      # Data subdirectory for data subset2
        polygon.geojson # Data file for dataset2
      datasubset2/      # Data directory for data subset2
        table.csv

Example Datasets

There are several example configuration files in the examples/datasets directory. You can follow the instructions in the README.md to load in one of the example datasets. The following table provides a description of each example:

Example	Description
building-bavaria	Uploads a set of CityGML files into the Postgres database of the stack. [b3dm files] and [a geoserver layer] are generated automatically with default settings for visualisation. There is also a OBDA mapping file (citydbOntop.obda), which provides an example of how to make the uploaded data queryable through the Ontop SPARQL endpoint.
building-cambridge	Uploads a set of GDB folders and a CSV file into the Postgres database of the stack. [b3dm files] and [a geoserver layer] are generated automatically with default settings for visualisation. There is also a OBDA mapping file (ontop_v2.obda), which provides an example of how to make the uploaded data queryable through the Ontop SPARQL endpoint.
building-hongkong	Uploads a GeoJSON file into the Postgres database of the stack. [b3dm files] and [a geoserver layer] are generated automatically with default settings for visualisation.
cropmap-reduced	Uploads a set of Shapefiles into the stack as single vector layer, which is served using the default style by GeoServer. This is a reduced styling (cropmap-reduced.sld) of the cropmap to make it clearer to view with pylon data.
cropmap-simple	Uploads a set of Shapefiles into the stack as single vector layer, which is served using the default style by GeoServer.
cropmap	Uploads a set of Shapefiles into the stack as single vector layer along with several .csv files that contain auxiliary data. Some of the auxiliary data is then used by a custom style (cropmap.sld) to dynamically colour the polygons when served through GeoServer. There is also a OBDA mapping file (ontop_with_comments.obda), which provides an example of how to make the uploaded data queryable through the Ontop SPARQL endpoint. Uses reference to file name for SQL query.
elevation	Uploads a set of GeoTiff files into the stack as a single raster layer, which is served using the custom `elevation` style via GeoServer.
forestry	Uploads a ShapeFile into the stack as a vector layer, along with a .csv file that defines a colour for each category. The layer is served using the colour mapping and a custom style (forestry.sld) through GeoServer.
forestry-reduced	Uploads a set of Shapefiles into the stack as single vector layer, which is served using the default style by GeoServer. This is a reduced styling (forestry-reduced.sld) of the forestry data to make it clearer to view with pylon data.
ng-pylons	Uploads a set of Shapefiles into the stack as multiple vector layers along a .csv file that contain auxiliary data. Some of the auxiliary data is then used by custom styles (overhead-lines.sld and underground-cables.sld) to dynamically style the lines, towers, and underground cables when served through GeoServer.
population	Uploads a GeoTiff file into the stack as a raster layer, which is served using the default style via GeoServer.
pylons	An example of how to use the `"externalDatasets"` node to load multiple datasets by name.
pylons-and-veg	An example of how to use the `"externalDatasets"` node to load multiple datasets by name of another config referencing other `"externalDatasets"`.
treesAndHills	An example of how to use the `"externalDatasets"` node to load multiple datasets by name.
ukpn-pylons	Uploads a set of Shapefiles into the stack as multiple vector layers along with a .csv file that contain auxiliary data. Some of the auxiliary data is then used by a custom style (overhead-lines.sld to dynamically style the lines and towers when served through GeoServer. There is also a OBDA mapping file (ukpn_ontop.obda), which provides an example of how to make the uploaded data queryable through the Ontop SPARQL endpoint.
rdf	A wrapper around a collection of examples of loading in RDF data. This includes triples with and without inference; quads; and using a properties file.
rml	An example of uploading YARRRML rules and a CSV file into a custom Blazegraph namespace.
rml-advanced	An example of uploading more advanced YARRRML rules (involving conditions and functions) and a CSV file into a custom Blazegraph namespace.
tboxcsv	An example of loading a TBox from a CSV file into a custom Blazegraph namespace.

The Dataset Configuration File

Each dataset should have its own JSON configuration file located in the inputs/config/ directory. The following table shows the top level nodes allowed in a configuration file.

Key	Required?	Default value	Description
`"name"`	No	The filename without the .json extension	The name of the dataset
`"datasetDirectory"`	No	The dataset's name	The directory within `inputs/data/` that contains the data files associated with this dataset
`"skip"`	No	`false`	If set to `true` this dataset will be ignored by the data uploader
`"database"`	No	The dataset's name	The name of the database within Postgres to which appropriate data will be uploaded
`"workspace"`	No	The dataset's name	The GeoServer workspace into which any 2D geospatial data layers, vector and raster, will be added
`"namespace"`	No	The dataset's name	The Blazegraph namespace into which RDF data will be added. The long syntax can be used to specify properties if the namespace needs to be created
`"externalDatasets"`	No*	`[]`	A list of other datasets' names. Each listed dataset will also be loaded if this dataset is loaded by name
`"additionalMetadata"`	No	`{}`	Additional metadata describing the dataset
`"dataSubsets"`	No*	`[]`	A list of data subset objects
`"styles"`	No*	`[]`	A list of GeoServer style file definition objects
`"mappings"`	No*	`[]`	A list of Ontop mapping (OBDA) file names
`"rules"`	No	`[]`	A list of inference rules file names
`"ontopLenses"`	No	`[]`	A list of Ontop "lenses" file names
`"staticGeoServerData"`	No	`null`	An object describing static data to be served by GeoServer

* At least one of these needs to be populated.

`"name"`

This is the name of the dataset. If left unspecified it is set to the name of the configuration file, without the .json extension. This is the value that should be specified in the "externalDatasets" node of another dataset.

📝 Note: If the dataset's name matches the name of the stack it is being loaded into then only that dataset and its associated external datasets will be loaded. More information on that in Using Specific Data Sets.

`"datasetDirectory"`

The directory within inputs/data/ that contains the data files associated with this dataset.

`"skip"`

Setting the "skip" value of a dataset to true will cause the data uploader to not load any of the data or files listed in that dataset.

`"database"`

The name of the database within Postgres to which appropriate data will be uploaded.

📝 Note: The database will be created if it does not already exist.

⚠️ Warning: Ontop can only access the default 'postgres' database so it is usually best not to change this value.

`"workspace"`

The GeoServer workspace into which any 2D geospatial data layers, vector and raster, will be added.

📝 Note: The workspace will be created if it does not already exist.

`"namespace"`

The Blazegraph namespace into which RDF data will be added.

📝 Note: The namespace will be created if it does not already exist.

short syntax

Just the name of the namespace. This is the recommend form when referring to an exiting namespace. For example:

"namespace": "kb"

long syntax

This syntax should be used when creating a new namespace. It allows the specification of properties that will be used when creating the Blazegraph namespace. The properties can be specified in a .properties file and/or directly in the .json config file, the values in the config file take precedence. For example, to load the properties from a file called myquads.properties that is stored in the /inputs/config directory:

"namespace": {
  "name": "quads",
  "propertiesFile": "myquads.properties"
}

To specify properties directly under a "properties" key, for example:

"namespace": {
  "name": "quads",
  "properties": {
    "com.bigdata.journal.AbstractJournal.bufferMode": "DiskRW",
    "com.bigdata.rdf.store.AbstractTripleStore.quadsMode": true
  }
}

Information about the different properties is distributed throughout the Blazegraph code repository. Some basic configurations are mentioned in the Blazegraph wiki in theConfiguring_Blazegraph and InferenceAndTruthMaintenance sections.

The most comprehensive documentation of the available properties is located in the Javadocs/comments for the various Options interfaces. Most (if not all) of the Options classes can be found by looking at the list of interfaces that the BigdataSail.Options interface extends (Superinterfaces). The source code for the most commonly used options can be found in in the following files: AbstractTripleStore Options, BigdataSail Options and Journal Options.

It is also possible to create namespaces, with a small selection of options, through the Blazegraph web interface and then inspect the properties. Be aware though that some of the property keys contain the namespace's name so can not just be copied for a different namespace.

⚠️ Warning: The properties will be ignored if the namespace already exists.

`"externalDatasets"`

Any datasets that are named under this node will be included if this dataset is loaded by name, either because the stack has the same name or because it appears in the "externalDatasets" list of another dataset that is loaded by name.

`"additionalMetadata"`

Additional metadata describing the dataset (or data subset). For example additional information that corresponds to concepts in the dcat or prov-o ontologies. Any triples added here are in addition to the basic metadata that is added by default. All metadata triples are added to the kb namespace in Blazegraph.

Key	Required?	Default value	Description
`"prefixes"`	No	`{}`	Key-value map of prefixes to IRIs
`"triplePatterns"`	No	`""`	Turtle formatted triples as a single-line string with double-quote characters escaped. The dataset IRI can be accessed as `?dataset`. For data subsets the variable is `?dataSubset`

Example:

"additionalMetadata": {
        "prefixes": {
            "ex": "<https://dcat.example.org/>",
            "dct": "http://purl.org/dc/terms/",
            "dcat": "http://www.w3.org/ns/dcat#",
            "xsd": "http://www.w3.org/2001/XMLSchema#"
        },
        "triplePatterns": "?dataset dct:publisher ex:finance-ministry ; dct:spatial <http://sws.geonames.org/6695072/> ; dct:temporal [ a dct:PeriodOfTime ; dcat:startDate \"2011-07-01\"^^xsd:date ;  dcat:endDate \"2011-09-30\"^^xsd:date ; ] ."
    },

`"dataSubsets"`

This node should contain a list of data subset objects. Each dataset should generally contain at least one data subset. If the files in a dataset are of multiple different types, or represent different geospatial layers, they should be divided into multiple data subsets, with one for each type/layer. Each data subset should then have its own subdirectory. These specify how to load the data from a particular set of files. Each data subset must have the following values specified:

Key	Required?	Default value	Description
`"name"`	No	Last component of the subdirectory	The name of the data subset
`"type"`	Yes	N/A	The type of the data
`"subdirectory"`	Yes	N/A	The subdirectory within the dataset directory that contains the data in this data subset
`"skip"`	No	`false`	If set to `true` this data subset will be ignored by the data uploader
`"sql"`	No	N/A	If the data is being loaded into the PostgreSQL database then the query provided here is run straight after the data is loaded 📂
`"additionalMetadata"`	No	`{}`	Additional metadata describing the data subset

`"name"`

This is the name of the data subset.

`"type"`

This controls which functions are used to load the data. More information about the different data types can be found here.

`"subdirectory"`

If there are multiple data subsets then each one must have a separate subdirectory set. If there is only one data subset, this still have to be set, otherwise no upload will be performed.

`"styles"`

A list of GeoServer style file definition objects. The styles defined here will be loaded into the GeoServer workspace associated with the dataset. Each entry requires the following values to be specified:

Key	Description
`"name"`	The name of the style, it will need to be prefixed by the dataset's workspace when referenced in GeoServer
`"file"`	The style file's path, relative to the dataset directory

Currently only .sld style files are supported. Worked examples of different SLD styles can be found in the GeoServer manual. If required in the future support for other style formats might be add as GeoServer does support several other formats natively and a few more if the required plugins are loaded.

An example of a style specification in the configuration file is:

"styles": [
    {
        "name": "elevation",
        "file": "elevation.sld"
    }
]

`"ontologyDataset"`

Specify a list of datasets that serve as ontologies for this dataset. These are treated like "externalDatasets" but the contents of the Blazegraph endpoint of that dataset is added as an Ontop ontology and/or loaded into the relevant triplestore namespace.

`"mappings"`

A list of Ontop mapping file definition objects provided as paths relative to the "datasetDirectory". Currently only the Ontop native format (.obda) is supported as it is much easier for both humans and Ontop to work with. Ontop also supports the R2RML (.ttl) OBDA file standard but the data uploader would need changes to include matching support.

The OBDA file for the cropmap example (ontop_with_comments.obda) shows the Ontop OBDA format. The Ontop OBDA file format is also described in detail in the OBDA mapping file section.

`"rules"`

Specify list of .toml files provided as paths relative to the "datasetDirectory". These .toml files specify a list of sparql insert queries in the Ontop rules format. An example of such a file can be found here. For Blazegraph these insert queries are run once after data is uploaded. For Ontop they are assigned as Ontop rules.

`"ontopLenses"`

Specify list of .json files provided as paths relative to the "datasetDirectory". These .json files specify a list of Ontop "lenses" in Ontop's custom format. Documentation for lenses can be found on the Ontop website.

`"staticGeoServerData"`

A description of static data to be loaded into and served by GeoServer. The icons can be found at GEOSERVER_URL/www/icons and the "other files" (being any regular files or folders) can be found at GEOSERVER_URL/www/static_data.

Key	Description
`"iconsDir"`	Directory relative to the `"datasetDirectory"` where icons files can be found
`"otherFiles"`	A list of "other files" with the `source` relative to the `"datasetDirectory"` and the `target` the location relative to `GEOSERVER_URL/www/icons` from which would would like this data is to be served

"staticGeoServerData": {
  "iconsDir": "icons",
  "otherFiles": [
    {
      "source": "my_additional_data/index.html",
      "target": "additional_data/index.html"
    },
    {
      "source": "my_other_additional_data_directory",
      "target": "other_additional_data"
    }
  ]
}

Notes:

If you want to reference icons uploaded in this way from a GeoServer .sld file, you will need to use stack-internal URLs, e.g.:

<OnlineResource xlink:type="simple" xlink:href="http://localhost:8080/geoserver/www/icons/myicon.png" />

The behaviour is similar to the cp -R command.
Relative file-system paths containing .. are not supported.

Data Types

The following data types are supported: vector, raster, tabular, rdf, rml, tboxcsv, citygml, xbuilding, and osmrouting. A description of how each is processed and a summary of the available configuration options are provided below.

Vector Data

The "vector" data type should be used to load 2D point, line or polygon geospatial data. The data loader does two things when uploading vector data:

It uses the GDAL ogr2ogr tool to read in data from a wide variety of file formats and output it to the PostgreSQL database in the stack. The full list of file formats that ogr2ogr supports is given here although some of these might not be available depending on the exact GDAL Docker image being used, see here for details.
It uses the GeoServer REST API to create a new layer in GeoServer that can be used to visualise the newly uploaded geometries.

The options for these two processes are set using the following json objects within the respective data subset object in the dataset configuration file:

GDAL Options

In most situations the default ogr2ogr settings will be sufficient to upload the data but sometimes some extra options need to be supplied. 📂 These can be specified within an "ogr2ogrOptions" object under the following keys:

`"sridIn"`

If the input dataset does not have an SRID/CRS/SRS specified then it can be specified as the value for the "sridIn" key. When specifying an EPSG code for the SRS it needs to include the authority as well as the ID, for example "EPSG:4296" rather than just 4296 or "4296". This sets the value of the -a_srs argument passed to ogr2ogr. If an SRID is set with this node, the uploader will not use GDAL to try attempt incoming SRID automatically.

`"sridOut"`

If you want to reproject the coordinates the target SRID/CRS/SRS can be set as the value for the "sridOut" key. When specifying an EPSG code for the SRS it needs to include the authority as well as the ID, for example "EPSG:4296" rather than just 4296 or "4296". This sets the value of the -t_srs argument passed to ogr2ogr. It also means any value specified for "sridIn" is passed as the value of the -s_srs argument, rather than -a_srs.

`"inputDatasetOpenOptions"`

Some data source formats require additional options to be set for the geometries and their metadata to be loaded correctly. These can be set as key-value pairs within a "inputDatasetOpenOptions" object. These options are format specific and are generally described in a section with the heading "Open options" or "Dataset open options" on the relevant driver documentation page. All of the vector drivers are listed here with links to their documentation. The values are passed to the ogr2ogr tool as NAME=VALUE pair arguments of the -oo option.

`"layerCreationOptions"`

All vector datasets are loaded into the PostGIS database within the stack with each data subset being loaded as a separate layer/table. In general these options should not need to be set explicitly as the ogr2ogr tool can usually work them out from the source dataset, or use default values. However, setting one or more of them may be required to fix specific problems with the input dataset. The layer creation options provided by the PostGIS driver are described here. The values are passed to the ogr2ogr tool as NAME=VALUE pair arguments of the -lco option.

`"outputDatasetOpenOptions"`

The PostGIS driver has a few options to control how the ogr2ogr tool connects to the database. However, all of the essential information (database name, port number, username, etc.) is set automatically so these settings should only be changed if absolutely necessary. The dataset open options provided by the PostGIS driver are described here. The values are passed to the ogr2ogr tool as NAME=VALUE pair arguments of the -doo option.

`"configOptions"`

Many options that can be set via environment variables can also be specified as "config" options. The values are passed to the ogr2ogr tool as NAME VALUE pair arguments of the --config option. A list of possible options can be found on the config options page.

`"otherOptions"`

Several non-driver specific options are also available. These can be set as key-array-valued pairs within an "otherOptions" object. This allows for multiple values per option (["value1", "value2"]) but requires that single values are still placed within an array (["value"]) and valueless flags are paired with an empty array ([]). A list of possible options can be found on the vector common options and ogr2ogr options pages.

`"envVars"`

A few aspects of some of the drivers can also be set via environment variables. These can be set as key-value pairs within an "envVars" object.

Common Drivers

Comma Separated Value (.csv)
ESRI Shapefile / DBF
PostgreSQL / PostGIS (mainly as the output)

GeoServer Options

For vector data you can add a geoServerSettings node within the relevant data subset in the configuration json. These settings are generally only required to add dynamic (value-based) styling to the layers for visualisation. Within that the following nodes can be added.

"virtualTable" creates a SQL View which is specified as follows:
- "name" a name is required.
- "sql" an SQL query that defines the virtual table is required. 📂
- "keyColumn" specify column for parameter key.
- "escapeSql" is Boolean true or false. This concerns the handling of special characters in column names such as setting single-quotes to doubled single-quotes.
- "geometry" specifies the geometry with the following key:value pairs.
  - "name" name of column with the geometry.
  - "type" one of Point, LineString, LinearRing, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometryCollection.
  - "srid" EPSG code as an integer, for example 4296 rather than "EPSG:4296" or "4296". Note that this is different from the GDAL Options.
- "parameters" specify individual parameters as a list of nodes with the following key:value pairs.
  - "name" parameter name.
  - "defaultValue" default value of parameter.
  - "regexpValidator" validation regular expression.
"defaultStyle" name of style within GeoServer that will be the style if of this layer if no other style is specified.

These are the most commonly used options, for more see the examples here and here.

Raster Data

The "raster" data type should be used to load raster/coverage geospatial data. The data loader does three things when uploading raster data:

It uses the GDAL gdal_translate tool to read in data from a wide variety of file formats and output it to Cloud Optimized GeoTIFF (COG) files stored in the stack. This is an extension of the GeoTIFF format and both are very efficient to read. The full list of file formats that gdal_translate supports is given here although some of these might not be available depending on the exact GDAL Docker image being used, see here for details.
It uses the PostGIS raster2pgsql tool to register the GeoTIFF files in the PostGIS database. The raster2pgsql tool also automatically divides the data into tiles in the database to make geospatial searching more efficient.
It uses the GeoServer REST API to create a new coverage layer in GeoServer that can be used to visualise the newly uploaded data.

GDAL Options

In most situations the default gdal_translate settings will be sufficient to upload the data but sometimes some extra options need to be supplied. 📂

📝 Note: If issues are encountered, due to non-standard projections or strips of the raster being flipped vertically in the visualisation, then it is possible to use gdalwarp tool by specifying "gdalWarpOptions" instead of "gdalTranslateOptions". Due to the potential loss of accuracy when re-projecting it is better to use gdal_tanslate whenever possible.

These can be specified within an "gdalTranslateOptions" object (previously just called "options") under the following keys:

`"sridIn"`

If the input dataset does not have an SRID/CRS/SRS specified then it can be specified as the value for the "sridIn" key. When specifying an EPSG code for the SRS it needs to include the authority as well as the ID, for example "EPSG:4296" rather than just 4296 or "4296". A full explanation of the acceptable SRS formats is given here. This sets the value of the -a_srs argument passed to gdal_translate.

`"sridOut"`

If you want to reproject the coordinates the target SRID/CRS/SRS can be set as the value for the "sridOut" key. When specifying an EPSG code for the SRS it needs to include the authority as well as the ID, for example "EPSG:4296" rather than just 4296 or "4296". A full explanation of the acceptable SRS formats is given here. This sets the value of the TARGET_SRS creation option passed to gdal_translate. This is an option specific to the COG raster driver when using gdal_translate, although we could use gdalwarp to handle this more efficiently in the future.

If there is no sridIn set in config file and gdal does not recognise the EPSG SRID (i.e. gdalsrsinfo returns an EPSG:-1), then the projection is assumed to be a custom one and will be appended to the spatial ref system in postGIS and GeoServer. This must still include an authority (although not EPSG) and a new number. The uploader will throw an error if the number exists already in the table.

e.g. if the sridOut is set to TWA:101000 and the projection is not recognised, this will be used as the SRID and authority in the newly specified custom projection.

`"inputDatasetOpenOptions"`

Some data source formats require additional options to be set for the geometries and their metadata to be loaded correctly. These can be set as key-value pairs within a "inputDatasetOpenOptions" object. These options are format specific and are generally described in a section with the heading "Open options" or "Dataset open options" on the relevant driver documentation page. All of the raster drivers are listed here with links to their documentation. The values are passed to the gdal_translate tool as NAME=VALUE pair arguments of the -oo option.

`"creationOptions"`

All raster datasets are loaded into the PostGIS database within the stack with each data subset being loaded as a separate layer/table. In general these options should not need to be set explicitly as the gdal_translate tool can usually work them out from the source dataset, or use default values. However, setting one or more of them may be required to fix specific problems with the input dataset. The creation options provided by the COG driver are described here. The values are passed to the gdal_translate tool as NAME=VALUE pair arguments of the -co option.

`"configOptions"`

Many options that can be set via enviornment variables can also be specified as "config" options. The values are passed to the gdal_translate tool as NAME VALUE pair arguments of the --config option. A list of possible options can be found on the config options page.

`"otherOptions"`

Several non-driver specific options are also available. These can be set as key-array-valued pairs within an "otherOptions" object. This allows for multiple values per option in an array ("--option": ["value1", "value2"]); single values either within an array ("--option": ["value"]) or as a string ("--option": "value"); and valueless flags are paired with an empty string ("--option": ""), array ("--option": []), or object ("--option": {}). A list of possible options can be found on the raster common options and gdal_translate options pages.

`"mdimSettings"`

This is for specific info about multidimensional geospatial files. Consider running gdalinfo and gdalmdiminfo on your file to get these parameters if they are not known.

"layerArrayName" is the array of the multidimensional file with the required data to iterate over and generate rasters
"timeOptions" contains information for geoserver to parse the time dimension. "arrayName" should have the name of the array with times.
- The "format" needs to be set to the standard format understood by java and geoserver e.g. "yyyyMMddHH". Be careful that m is minute and M is month etc. See above link to correctly format your datetime string. This can include a timezone offset but will be overridden by timeZone parameter.
- The regex should be used in conjuction with the format to parse the filename. e.g. "regex": ".*([0-9]{10}).*" will parse 10 digits beside each other anywhere in the filename.
- The timeZone is passed as a ZoneId in java and can be something like "GMT" or "Europe/Paris", see more here.

Common Drivers

GeoTIFF
COG – Cloud Optimized GeoTIFF (mainly as the output)

netCDF Files

netCDF files are commonly used in climate science projection data. The uploader can recognise these and will copy them into postgis and subsequently iterate over internal bands to create individual geotiffs for each value in the time series band. It is important to set mdimSettings

GeoServer Options

For raster data you can add a geoServerSettings node within the relevant data subset in the configuration json. Within that the following nodes can be added.

"layerSettings"
- "defaultStyle": name of style within GeoServer that will be the style if of this layer if no other style is specified.

Tabular Data

The "tabular" data type should be used to load non-geospatial data. The data loader just does one thing when uploading tabular data:

It uses the GDAL ogr2ogr tool to read in data from a wide variety of file formats and output it to the PostgreSQL database in the stack. As the data is intended to be non-geospatial, this is most useful for reading in data from comma separated value (.csv), and Microsoft Excel's XLS and XLSX formatted files. The full list of file formats that ogr2ogr supports is given here although some of these might not be available depending on the exact GDAL Docker image being used, see here for details.

GDAL Options

These are the same as listed in the vector GDAL Options although obviously the options specific to geospatial data will not be relevant.

Common Drivers

RDF Data

The "rdf" data type should be used to load RDF data (triples or quads) from common file formats. The full list of file formats that are supported is given here. The data loader does the following when uploading RDF data:

It uses the RemoteStoreClient::uploadFile method to read in RDF triple and quad data to the Blazegraph database in the stack.

There are no configurable options for this process, the namespace the data is added to is always the one defined in the parent dataset.

RML Data

The "rml" data type should be used to load RDF triples from CSV files using YARRRML rules, into a specified SPARQL endpoint namespace (defined in the parent dataset). Users must deploy both the rml-mapper and yarrrml-parser services within their stack, and provide resources as matching .yml and .csv file pairs with identical names for the loader to associate them.

⚠️ Warning: Do NOT include sources or targets field in your YARRRML rules. The loader will automatically populate these fields.

📝 Note: Users can make use of the default functions in RML Mapper as functions in their YARRRML rules. Please read the test .yml files if you require compliant samples.

The data loader performs the following steps when uploading csv data and rules:

YARRRML loading: The loader automatically populates the sources and targets fields following the csv data file names and the namespace defined in the parent dataset.
YARRRML to RML: It converts the generated YARRRML rules into RML rules using the YARRRML Parser tool.
RML upload: It reads the RML rules using the RMLMapper tool to convert the csv data files into RDF triples. These triples are then subsequently uploaded to the specified namespace to the Blazegraph database within the stack.

TBox CSV Data

The "tboxcsv" data type should be used to load TBox triples from CSV files that adhere to the TBox Generator format, example files can be found here. The data loader does the following when uploading RDF data:

It uses the TBoxGeneration::generateTBox method to generate an OWL file from the contents of the CSV file.
It uses the RemoteStoreClient::uploadFile method to uploads the contents of the OWL file to the Blazegraph database in the stack.

There are no configurable options for this process, the namespace the data is added to is always the one defined in the parent dataset.

CityDB Data

The "CityDB" data type should be used to load CityGML and CityJSON data. The data loader does the following things by default when uploading data:

It uses the 3DCityDB Importer impexp import tool to read in data from CityGML and CityJSON files and output it to the PostgreSQL database in the stack using the 3DCityDB schema. The full list of file formats that impexp import supports is given here.
Building footprints and heights are added to the uploaded data if they do not exist.
It writes the processed data that has been uploaded to PostgreSQL out to a compressed CityGML file using the 3DCityDB Importer impexp import tool.
It uses the GeoServer REST API to create a new layer in GeoServer that can be used to visualise the newly uploaded geometries (in Mapbox).
It uses the py3dtiler tool to create 3DTile sets that can be used to visualise the newly uploaded geometries (in Cesium).

These tilesets are written to folders in a Docker volume and served on the /3dtiles path of the stack. The full URL for a tileset.json file, which should be specified in the data.json visualisation file, is <server base address>/3dtiles/<database name>/<database schema>/<spec>/tileset.json. The components of this are as follows:

Placeholder	Description
`<server base address>`	e.g. `http://localhost:3838`
`<database name>`	The name of the database, as specified at the top-level of the dataset config file
`<database schema>`	The database schema, this is fixed as `citydb` for now.
`<spec>`	There are currently three hardcoded specs: `lod2-features`, `lod2-buildings` and `lod1_lod2-buildings`. Each one is generated using different options passed to the `py3dtiler` tool.

The options for these processes are set using the following json objects within the respective data subset object in the dataset configuration file:

Import Options

The only option that is required is "sridIn". Other than "sridIn", in most situations the default impexp import settings will be sufficient to upload the data but sometimes some extra options need to be supplied. Options can be specified within an "importOptions" object under the following keys:

`"sridIn"` (Required)

The SRID/CRS/SRS of the input dataset does not get picked up automatically so needs to be specified as the value for the "sridIn" key. When specifying an EPSG code for the SRS just the ID is required as a string, for example "4296" rather than 4296 or "EPSG:4296". Because the SRID is set for each database schema Each dataset should write to its own PostgreSQL database and all of the city data in a dataset must use the same SRID. The 3DCityDB importer doesn't support reprojection but this could be added in the future.

`"options"`

An "options" node can be added with a map of options to be passed to the impexp import command-line interface. The format is the same as the one used for ogr2ogr otheroptions. The list of avaliable options can be found here. The "Database connection options" are set automatically by the stack-data-uploader so can be ignored.

`"augmentData"`

This boolean flag controls whether footprint and height will be calculated and added to the existing data, if not already present. It is assumed to be true if omitted.

`"discoverThematicSurface"`

If this boolean flag is set to true, the uploader will attempt to identify untagged surfaces as roof, wall or ground surfaces. It is assumed to be false. Note that this is dependent on "augmentData".

`"createTile"`

This boolean flag controls whether 3D tiles will be generated for uploaded building data. It is assumed to be true if omitted.

`"parallelTiling"`

Three sets of 3D tiles with different settings will be generated by the tiler. This boolean flag controls whether the tiling process will be run in parallel or not. Running in parallel will be faster but more suspectible to running out of memory. It is assumed to be true if omitted. Note that this is dependent on "createTile".

X building data

The "XtoCityDB" data type should be used to load LoD1 (footprint with height) building data that are not CityGML or CityJSON. The data loader does the following when uploading data:

It uses the GDAL ogr2ogr tool to read in data from a wide variety of file formats and output it to the PostgreSQL database in the stack. The full list of file formats that ogr2ogr supports is given here although some of these might not be available depending on the exact GDAL Docker image being used, see here for details.
It uses the 3DCityDB Importer impexp import tool to initialise the 3DCityDB schema in the PostgreSQL database in the stack.
It uses two SQL scripts to convert the building data uploaded by GDAL and populate the 3DCityDB tables sequentially. The building data will be instantiated as LoD2 buildings with thematic surfaces.
Building footprints and heights are added to the uploaded data.
It writes the processed data that has been uploaded to PostgreSQL out to a compressed CityGML file using the 3DCityDB Importer impexp import tool.
It uses the GeoServer REST API to create a new layer in GeoServer that can be used to visualise the newly uploaded geometries (in Mapbox).
It uses the py3dtiler tool to create 3DTile sets that can be used to visualise the newly uploaded geometries (in Cesium).

Check here for details about configuring the GDAL, here for details about the 3DTile set and here for details about the Geoserver layer. "sridIn" and "augmentData" for the 3DCityDB Importer are read.

"table" will be the name of the resulting table of the GDAL ogr2ogr tool, whereas "name" will be the name of the GeoServer layer.

Building data processing with SQL

Two SQL scripts are executed to convert the building data to conform with the 3DCityDB schema. The following information must be contained by (or can be derived from) the table uploaded by GDAL:

"IDname", "IDval": an ID for each building
"footprint": the footprint of each building
"elevation": the ground height of each building
"height": the height of each building

A "columnMap" can be specified in the input configuration to inform the stack which columns to look for these information. "IDval" is the name of the column that contains the building ID, whereas "IDname" is the name of the ID that will be stored in the 3DCityDB tables.

The first SQL script creates two tables from the original table: "raw_building_XtoCityDB" and "raw_surface_XtoCityDB". It assumes that the original data is uploaded to the public schema. The "raw_building_XtoCityDB" table has the following columns:

"IDval": the ID of each building from the original data
"gmlid": a randomly generated GML ID of each building
"geom": the 3D solid geometry of each building, generated by extruding the "footprint" with "height" and shifting it with "elevation"
"mh": the height of each building

The "raw_surface_XtoCityDB" table has the following columns:

"building_gmlid": the GML ID of the parent building of a surface
"gmlid": a randomly generated GML ID of each surface of each building
"class": an integer indicating whether a surface is a roof, wall or ground
"geom": the 3D polygon geometry of each surface

In some cases, the original data require more sophisticated processing. Users can supply a custom SQL script with the "preprocessSql" keyword using File by Value Name. The query must create the "raw_building_XtoCityDB" and "raw_surface_XtoCityDB" tables in the public schema from the uploaded original data with the column names stated above. In this case, only "IDname" and "IDval" will take effect and needed to be specified in "columnMap".

The second SQL script populates the 3DcityDB schema with preprocessed building data.

OSM Data

The "osmrouting" data type should be used to load Open Street Map (OSM) files in a form that is compatible with pgRouting. These can be .osm or .pbf files. By default, three tables and three GeoServer layers are created; DATA_SUBSET_NAME_ways, DATA_SUBSET_NAME_ways_vertices_pgr, and DATA_SUBSET_NAME_pointsofinterest.

osm2pgrouting Options

For OSM data you can add a osm2PGRoutingOptions node within the relevant data subset in the configuration json. This can be used to configure the osm2pgrouting tool as specified here.

"flags": a list of flags without arguments e.g. [--attributes, --addnodes]
"options": a node containing the key value pairs of options with arguments e.g. {"--chunk": "40000"}

OSM GeoServer Options

For OSM data you can add the nodes waysGeoServerSettings, verticesGeoServerSettings, and poiGeoServerSettings nodes within the relevant data subset in the configuration json. The nodes that can be added within each are the same as the GeoServer options for vector data.

Ontop (OBDA)

The stack uses Ontop to enable running SPARQL queries on data stored in relational databases.

The stack-data-uploader will spin up a new Ontop container for each Dataset that specifies an Ontop (OBDA) mapping. The name of the new Ontop container will include the name of the Dataset.

📝 Note: The stack-data-uploader won't attempt to pull the Ontop image so it must be present on the system before the stack-data-uploader is run.

OBDA Mapping File

The general layout of the file is as follows, see the official guide for more details:

Comments

Full-line comments are officially supported in the Ontop OBDA format, specified by placing a ; character at the beginning of the line.

As comments were not previously officially avaliable the data uploader also supports using the # character, however these comments will be stripped out before the mapping is passed to Ontop. Comments are started by a # character and can appear at the start of a line, that contains no "code", or at the end of one that does. When a comment follows "code" the # character must be preceded by at least one white-space character. For example:

# Comment at the start of a line, whole line.
SELECT var1 var2 # Comment following some "code"

Prefix Declarations

This is where the RDF prefixes should be defined, these can then be used when specifying triple patterns in the mappings. It starts with a [PrefixDeclaration] tag, followed by the prefix-IRI base pairs, without angled-brackets <>.

[PrefixDeclaration]
rdf:    http://www.w3.org/1999/02/22-rdf-syntax-ns#
ex:     http://example.org/

Mapping Declarations

The mapping declarations section starts with this line: [MappingDeclaration] @collection [[ and is closed by the following line: ]]

Each mapping has three parts:

Label	Description
mappingId	The unique name of the mapping
target	Template of the Turtle format triple patterns that will exist in Ontop's virtual knowledge graph. Placeholders for values extracted from the PostGIS database are specified as `{value}`.
source	SQL query used to determine the values to replace the placeholders in the target block.

Ontop effectively takes the result of the SQL SELECT query written in the source block and for each row creates a set of virtual triples by substituting the SQL variables into the Turtle formatted template in the target block. In practice Ontop performs direct mappings, query rewriting, and other optimisations to improve the efficiency of running SPARQL queries over these virtual triples.

A simple example of a mapping is:

mappingId   exampleMapping

target      ex:building/{id} ex:hasName "{name}"^^xsd:string;
                ex:hasId {id}^^xsd:integer .

source      SELECT id, name
            FROM buildings

Here the the PostgreSQL table buildings is assumed to contain the columns id (containing integers) and name (containing strings). For each row in that table two virtual triples will be created ex:building/{id} ex:hasName "{name}"^^xsd:string and ex:building/{id} ex:hasId {id}^^xsd:integer.

For example if the buildings table was as follows then the subsequent virtual triples would be created.

id	name	....
1234	"building 1"	...
1235	"building 2"	...

ex:building/1234 ex:hasName "building 1"^^xsd:string

ex:building/1234 ex:hasId 1234^^xsd:integer

ex:building/1235 ex:hasName "building 2"^^xsd:string

ex:building/1235 ex:hasId 1235^^xsd:integer

SPARQL Queries via Ontop

Ontop supports a wide range of SPARQL 1.1 and GeoSPARQL 1.0 features. The cropmap example OBDA file shows how to use the PostGIS function ST_ASTEXT and the http://www.opengis.net/ont/geosparql#wktLiteral to make it possible to run GeoSPARQL queries.

Using Specific Data Sets

If you do not want the use every config file you can either use "skip"=true or name your stack so that the relevant config file is named <STACK NAME>.json. If you want to use a few config files you can create one master config file named <STACK NAME>.json with the following.

{
    "name": "<STACK NAME>",
    "externalDatasets": [
        "name of one config file (no .json)",
        "name of another config file",
        // Other external datasets
    ]
}

Value by File Name

The stack uploader supports file referencing in the config file on certain values denoted here by 📂. This an be done by giving a value of '@' followed by the name of the file containing the text to be used for that value. For example one can avoid long SQL queries in their configs by putting them in a file in the inputs/config directory and referencing that file in the following way.

{
    ...
    "sql": "@/inputs/config/my-sql-query.sql"
    ...
}

Note that this file path is the path inside the container.

Processing Without Upload

If a subdirectory is not specified in a dataSubset node, no data will be uploaded, however for some types of data this is useful to run only the post processing. For example it is possible to run arbitrary sql by specifying sql queries or pointing to .sql files and specifying a table. It is also useful to create layers in GeoServer on already uploaded data.

Debugging the Stack Data Uploader in VSCode

In the Run and Debug side panel of VSCode run the Debug (stack-data-uploader) configuration.

Developing the Stack Data Uploader in VSCode

You will need permission to push to the CMCL package repository to be able to build the stack-data-uploader project

Follow the instructions in step 1. of Debugging the Stack Data Uploader in VSCode
Create two files called repo_username.txt and repo_password.txt in the stack-data-uploader/docker/credentials directory. Populate the files with your GitHub username and access token (with scope to write packages), respectively.
In the Run and Debug side panel of VSCode run the Build and Debug (stack-data-uploader) configuration.

Troubleshooting

The best place to start is to look at the container logs. These can be seen by right clicking on the containers on VSCode or through Docker Desktop.
It is often helpful to run the uploader after each step after step 3 of Running the Stack Data Uploader. This way you can look at look at the user interfaces of the various services (see step 5 of Spinning up a Stack) and be able to see your data sets being populated here.
When writing SQL queries you can try them in the Adminer first.
It is recommended that you write and validate GeoServer styles in the GeoServer UI, saving it to a .sld file.

Quick Fixes

General

For certain vector geometries (e.g. MULTILINESTRING and LINESTRING) it is necessary to use -nlt to specify the geometry in the following way.
```
"ogr2ogrOptions": {
    "otherOptions": {
        "-nlt": ["<GEOMETRY TYPE>"]
    }
}
```
To upload only specific properties/fields from a source dataset their names can be specified as comma-separated values for the "-select" option under "otherOption". In the example below only the fields with the names field1, field3, field4 and field8 would be uploaded with other fields, for example field2 and field5, being ignored.
```
"ogr2ogrOptions": {
    "otherOptions": {
        "-select": [
            "field1,field3,field4,field8"
        ]
    }
}
```
The cropmap example shows this being used to remove some fields (containing calculated areas and perimeters) that were not constantly named across all of the crop-map-of-england-crome-2020 Shapefiles.

ESRI File Geodatabase

As described in the GDAL documentation ESRI File Geodatabase datasets must be stored in a directory/folder with a name that ends with the .gdb extension. For improved efficiency this folder can be added to a zip file with the .gdb.zip extension. For example:

inputs/
  data/
    dataset1/
    datasubset1/
      layer.gdb.zip           # Compressed zip file
      layer.gdb/              # Special .gdb folder containing ESRI File Geodatabase files
        a0000000a.gdbtablx
        a0000000a.gdbtable
        a0000000a.gdbindexes
        a0000000a.freelist
        a0000000a.spx
        ...

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

The Stack Data Uploader

Table of Contents

Introduction

Running the Stack Data Uploader

0. Prerequisites

1. Initialise the stack

2. Check the examples

3. Copy in data files

4. Create a configuration file

5. Running the data uploader

Datasets and Subsets

Example Datasets

The Dataset Configuration File

"name"

"datasetDirectory"

"skip"

"database"

"workspace"

"namespace"

short syntax

long syntax

"externalDatasets"

"additionalMetadata"

"dataSubsets"

"name"

"type"

"subdirectory"

"styles"

"ontologyDataset"

"mappings"

"rules"

"ontopLenses"

"staticGeoServerData"

Data Types

Vector Data

GDAL Options

"sridIn"

"sridOut"

"inputDatasetOpenOptions"

"layerCreationOptions"

"outputDatasetOpenOptions"

"configOptions"

"otherOptions"

"envVars"

Common Drivers

GeoServer Options

Raster Data

GDAL Options

"sridIn"

"sridOut"

"inputDatasetOpenOptions"

"creationOptions"

"configOptions"

"otherOptions"

"mdimSettings"

Common Drivers

netCDF Files

GeoServer Options

Tabular Data

GDAL Options

Common Drivers

RDF Data

RML Data

TBox CSV Data

CityDB Data

Import Options

"sridIn" (Required)

"options"

"augmentData"

"discoverThematicSurface"

"createTile"

"parallelTiling"

X building data

`"name"`

`"datasetDirectory"`

`"skip"`

`"database"`

`"workspace"`

`"namespace"`

`"externalDatasets"`

`"additionalMetadata"`

`"dataSubsets"`

`"name"`

`"type"`

`"subdirectory"`

`"styles"`

`"ontologyDataset"`

`"mappings"`

`"rules"`

`"ontopLenses"`

`"staticGeoServerData"`

`"sridIn"`

`"sridOut"`

`"inputDatasetOpenOptions"`

`"layerCreationOptions"`

`"outputDatasetOpenOptions"`

`"configOptions"`

`"otherOptions"`

`"envVars"`

`"sridIn"`

`"sridOut"`

`"inputDatasetOpenOptions"`

`"creationOptions"`

`"configOptions"`

`"otherOptions"`

`"mdimSettings"`

`"sridIn"` (Required)

`"options"`

`"augmentData"`

`"discoverThematicSurface"`

`"createTile"`

`"parallelTiling"`