Skip to content

Load the AIS chapter from the bundled data file #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
estebanzimanyi merged 1 commit into from
Jun 24, 2026
Merged

Load the AIS chapter from the bundled data file #32

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 14 additions & 10 deletions docs/AIS.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
<section>
<title>Data</title>
<para>
The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format <ulink url="https://web.ais.dk/aisdata/">here</ulink>. The columns in the CSV are listed in <xref linkend="tabdata"/>. This module uses the data of one day June 1<superscript>st</superscript> 2023. The CSV file size is 582 MB, and it contains more than 11 M rows.
The Danish Maritime Authority publishes about 3 TB of AIS routes in CSV format <ulink url="https://web.ais.dk/aisdata/">here</ulink>. The columns in the CSV are listed in <xref linkend="tabdata"/>. This module uses the data of one day. A ready-to-use one-day extract (January 4<superscript>th</superscript> 2018, 1,209,962 rows) is bundled with the workshop in the <ulink url="https://github.com/MobilityDB/MobilityDB-workshop/tree/master/data">data</ulink> directory and, in the Docker image, under <varname>/Workshop/ais_data/ais.csv</varname>, so the tutorial can be run without downloading the full dataset. The full daily files, each about 580 MB with more than 11 M rows, can be downloaded from the link above. The result counts shown in this chapter correspond to a full day's data and will be smaller for the bundled extract.
</para>

<table pgwide='1' width='100%' id='tabdata' frame='all'><title>AIS columns</title>
Expand Down Expand Up @@ -138,24 +138,28 @@ CREATE TABLE AISInput(
<section>
<title>Loading the Data</title>
<para>
For importing CSV data into a PostgreSQL database one can use the <varname>COPY</varname> command as follows:
The bundled file <varname>ais.csv</varname> includes the <varname>Geom</varname> column, so it is loaded into all columns of the table with the <varname>COPY</varname> command as follows:
<programlisting language="sql" xml:space="preserve">
COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus,
ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length,
TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType,
SizeA, SizeB, SizeC, SizeD)
FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER;
COPY AISInput
FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER;
</programlisting>
</para>
<para>
It is possible that the above command fails with a permission error. The reason for this is that <varname>COPY</varname> is a server capability, while the CSV file is on the client side. To overcome this issue, one can use the <varname>\copy</varname> command of <varname>psql</varname> as follows:
<programlisting language="bash" xml:space="preserve">
psql -d DanishAIS -c "\copy AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus, ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length, TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType, SizeA, SizeB, SizeC, SizeD) FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER;"
psql -d DanishAIS -c "\copy AISInput FROM '/Workshop/ais_data/ais.csv' DELIMITER ',' CSV HEADER;"
</programlisting>
In addition, if you downloaded the CSV file from <ulink url="https://github.com/MobilityDB/MobilityDB-workshop/tree/master/data">this repo's data</ulink> , then you will need to add the column 'geom' to the command.
</para>
<para>
This import took about 1 minute and 30 seconds on my machine, which is a development laptop. The CSV file has 11,809,593 rows, all of which were correctly imported. For bigger datasets, one could alternative could use the program <ulink url="https://github.com/dimitri/pgloader">pgloader</ulink>.
The full daily files published by the Danish Maritime Authority do not include the <varname>Geom</varname> column, which is computed below. Such a file is loaded by listing the columns explicitly so that <varname>Geom</varname> is left empty:
<programlisting language="sql" xml:space="preserve">
COPY AISInput(T, TypeOfMobile, MMSI, Latitude, Longitude, NavigationalStatus,
ROT, SOG, COG, Heading, IMO, CallSign, Name, ShipType, CargoType, Width, Length,
TypeOfPositionFixingDevice, Draught, Destination, ETA, DataSourceType,
SizeA, SizeB, SizeC, SizeD)
FROM '/home/mobilitydb/DanishAIS/aisdk-2023-06-01.csv' DELIMITER ',' CSV HEADER;
</programlisting>
For bigger datasets, one could alternatively use the program <ulink url="https://github.com/dimitri/pgloader">pgloader</ulink>.
</para>
<para>
We clean up some of the fields in the table and create spatial points with the following command.
Expand Down
Loading