From 69e1e79b915690a22bea4a3ffc88652bbbf4589f Mon Sep 17 00:00:00 2001 From: Esteban Zimanyi Date: Wed, 24 Jun 2026 22:47:23 +0200 Subject: [PATCH] Load the AIS chapter from the bundled data file The AIS chapter loads the ais.csv file bundled with the workshop, which includes the Geom column, and lists the explicit columns for the full daily files from the Danish Maritime Authority that omit it. --- docs/AIS.xml | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/AIS.xml b/docs/AIS.xml index c82b238..38e1235 100644 --- a/docs/AIS.xml +++ b/docs/AIS.xml @@ -36,7 +36,7 @@
Data - The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format here. The columns in the CSV are listed in . This module uses the data of one day June 1st 2023. The CSV file size is 582 MB, and it contains more than 11 M rows. + The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format here. The columns in the CSV are listed in . This module uses the data of one day. A ready-to-use one-day extract (January 4th 2018, 1,209,962 rows) is bundled with the workshop in the data directory and, in the Docker image, under /Workshop/ais_data/ais.csv, so the tutorial can be run without downloading the full dataset. The full daily files, each about 580 MB with more than 11 M rows, can be downloaded from the link above. The result counts shown in this chapter correspond to a full day's data and will be smaller for the bundled extract. AIS columns @@ -138,24 +138,28 @@ CREATE TABLE AISInput(
Loading the Data - For importing CSV data into a PostgreSQL database one can use the COPY command as follows: + The bundled file ais.csv includes the Geom column, so it is loaded into all columns of the table with the COPY command as follows: -COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, - ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, - TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, - SizeA, SizeB, SizeC, SizeD) -FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER; +COPY AISInput +FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER; It is possible that the above command fails with a permission error. The reason for this is that COPY is a server capability, while the CSV file is on the client side. To overcome this issue, one can use the \copy command of psql as follows: -psql -d DanishAIS -c "\copy AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, SizeA, SizeB, SizeC, SizeD) FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER;" +psql -d DanishAIS -c "\copy AISInput FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER;" - In addition, if you downloaded the CSV file from this repo's data , then you will need to add the column 'geom' to the command. - This import took about 1 minute and 30 seconds on my machine, which is a development laptop. The CSV file has 11,809,593 rows, all of which were correctly imported. For bigger datasets, one could alternative could use the program pgloader. + The full daily files published by the Danish Maritime Authority do not include the Geom column, which is computed below. Such a file is loaded by listing the columns explicitly so that Geom is left empty: + +COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, + ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, + TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, + SizeA, SizeB, SizeC, SizeD) +FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER; + + For bigger datasets, one could alternatively use the program pgloader. We clean up some of the fields in the table and create spatial points with the following command.