diff --git a/docs/AIS.xml b/docs/AIS.xml index c82b238..38e1235 100644 --- a/docs/AIS.xml +++ b/docs/AIS.xml @@ -36,7 +36,7 @@
Data - The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format here. The columns in the CSV are listed in . This module uses the data of one day June 1st 2023. The CSV file size is 582 MB, and it contains more than 11 M rows. + The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format here. The columns in the CSV are listed in . This module uses the data of one day. A ready-to-use one-day extract (January 4th 2018, 1,209,962 rows) is bundled with the workshop in the data directory and, in the Docker image, under /Workshop/ais_data/ais.csv, so the tutorial can be run without downloading the full dataset. The full daily files, each about 580 MB with more than 11 M rows, can be downloaded from the link above. The result counts shown in this chapter correspond to a full day's data and will be smaller for the bundled extract. AIS columns @@ -138,24 +138,28 @@ CREATE TABLE AISInput(
Loading the Data - For importing CSV data into a PostgreSQL database one can use the COPY command as follows: + The bundled file ais.csv includes the Geom column, so it is loaded into all columns of the table with the COPY command as follows: -COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, - ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, - TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, - SizeA, SizeB, SizeC, SizeD) -FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER; +COPY AISInput +FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER; It is possible that the above command fails with a permission error. The reason for this is that COPY is a server capability, while the CSV file is on the client side. To overcome this issue, one can use the \copy command of psql as follows: -psql -d DanishAIS -c "\copy AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, SizeA, SizeB, SizeC, SizeD) FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER;" +psql -d DanishAIS -c "\copy AISInput FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER;" - In addition, if you downloaded the CSV file from this repo's data , then you will need to add the column 'geom' to the command. - This import took about 1 minute and 30 seconds on my machine, which is a development laptop. The CSV file has 11,809,593 rows, all of which were correctly imported. For bigger datasets, one could alternative could use the program pgloader. + The full daily files published by the Danish Maritime Authority do not include the Geom column, which is computed below. Such a file is loaded by listing the columns explicitly so that Geom is left empty: + +COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, + ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, + TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, + SizeA, SizeB, SizeC, SizeD) +FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER; + + For bigger datasets, one could alternatively use the program pgloader. We clean up some of the fields in the table and create spatial points with the following command.