OpenAlex Watcher

OpenAlex Watcher monitors the OpenAlex S3 manifest for new works data files, downloads the relevant JSONL archives, and processes them through citescoop to produce temporary outputs for authors, institutions and works.

Features

• Watch the OpenAlex S3 manifest for updates
• Download new OpenAlex works files from S3
• Stream gzipped JSONL files into citescoop
• Combine processed temporary entity files into PBF outputs

Requirements

• Python 3.11 or newer
• citescoop-cli binary
• Network access to the OpenAlex S3 bucket

Usage

Basic usage

1. Install requirements

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt

2. Run the watcher:

   python3 watcher.py \
     --citescoop /path/to/citescoop-cli \
     --output-dir /path/to/output \
     --max-parallel 2

3. The watcher will process OpenAlex works files and write temporary outputs to output-dir. After processing, it will combine entity files into:

   • authors.pbf
   • institutions.pbf
   • works.pbf

Running periodically

You can run the watcher on a schedule using cron or systemd timers.

Cron example (run daily at 02:00):

0 2 * * * /usr/bin/python3 /path/to/watcher.py --citescoop /path/to/citescoop-cli --output-dir /path/to/output --max-parallel 2 >> /var/log/openalex-watcher.log 2>&1

systemd timer example:

Create a service unit at /etc/systemd/system/openalex-watcher.service:

[Unit]
Description=OpenAlex Watcher

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /path/to/watcher.py --citescoop /path/to/citescoop-cli --output-dir /path/to/output --max-parallel 2

Create a timer unit at /etc/systemd/system/openalex-watcher.timer to run daily at 02:00:

[Unit]
Description=Run OpenAlex Watcher daily

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable and start the timer:

sudo systemctl daemon-reload
sudo systemctl enable --now openalex-watcher.timer

Configuration

The watcher reads its last processed date from a config file, which must contain a [watcher] section and a date value in ISO 8601 format.

Example:

[watcher]
date = 2025-01-01T00:00:00

Use --force to reprocess the current manifest even if its date is not newer than the configured date.

Contributing

See the CONTRIBUTING document.

Licensing

This repo uses the REUSE standard in order to communicate the correct license for the file. For those unfamiliar with the standard the license for each file can be found in one of three places. The license will either be in a comment block at the top of the file, in a .license file with the same name as the file, or in the dep5 file located in the .reuse directory. All files committed to this repo must contain valid licensing information or the pull request can not be accepted.