Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@
openspecimen_install_type: "{{ openspecimen_install_type | default('upgrade') }}"

pre_tasks:
# Detect deploy vs rollback direction. On downgrade, the rollback flow
# is invoked automatically and the play ends — the openspecimen role
# below does not run.
- name: Detect direction (deploy or rollback)
ansible.builtin.include_role:
name: openspecimen
tasks_from: direction

# Verify the release zip and every paid/customer plugin zip exists on the
# control node before the openspecimen role runs. Fails fast if missing.
- name: Pre-flight artifact check
Expand Down
65 changes: 48 additions & 17 deletions docs/DEPLOY-UPGRADE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,23 @@ Day-1 fresh install, Day-2 upgrade, and rollback procedures for OpenSpecimen.

## How version detection works

The roles use the marker file `/usr/local/openspecimen/.release` on the target
node to determine what is currently installed:
The playbooks use the marker file `/usr/local/openspecimen/.release` on the
target node to determine what is currently installed, then compare against the
requested `openspecimen_release` using natural version sort (`sort -V`):

- **File absent** → fresh install (all five roles run in order)
- **File present, same version** → no-op (idempotent run)
- **File present, lower version** → upgrade (backup → deploy → restart)
- **File present, higher version** → downgrade — **blocked** with a clear error
- **Marker absent** → fresh install (all five roles run in order)
- **Marker present, same version** → re-deploy (idempotent run)
- **Marker present, requested > installed** → upgrade (backup → deploy → restart)
- **Marker present, requested < installed** → automatic rollback (see "Rollback" below) —
the requested version's backup is restored and the play ends. No separate job needed.

`openspecimen_release` is always supplied at run time via `-e openspecimen_release=<name>`.
It is **not** stored in inventory `group_vars`.
`openspecimen_release` is always supplied at run time via
`-e openspecimen_release=<name>`. It is **not** stored in inventory `group_vars`.

Direction detection runs as the **first** pre_task in both `site.yml` and
`deploy.yml` — `roles/openspecimen/tasks/direction.yml`. When a downgrade is
detected, the play dispatches to `tasks_from: rollback` (target_version =
requested release) and ends; the remaining roles never run.

---

Expand Down Expand Up @@ -190,32 +197,56 @@ Each timestamped directory holds a complete snapshot:

## Day-2: Rollback

Use the `rollback.yml` playbook. It restores the WAR, all three plugin tiers,
the MySQL connector JAR (when backed up), and the `.release` marker from a
timestamped backup, then waits for the health check. Target: ≤ 5 minutes.
Rollback has two paths — both backed by the same `roles/openspecimen/tasks/rollback.yml`:

### A. Automatic (via deploy job — recommended)

### Roll back to the most recent backup (default)
Just pick a lower release in `site.yml` / `deploy.yml` (or in the Jenkins
deploy job). Direction detection notices the requested version is older than
what's installed, finds the backup whose `.release` matches the request,
runs the rollback, and ends the play. **No separate rollback job or playbook
invocation required.**

```bash
ansible-playbook -i inventory/customers/<name>/ rollback.yml \
# Installed: openspecimen_v12.2.RC12, want to go back to RC8
ansible-playbook -i inventory/customers/<name>/ site.yml \
-e openspecimen_release=openspecimen_v12.2.RC8 \
-e @secrets/<name>.yml --vault-password-file .vault-pass
```

### Roll back to a specific timestamped backup
If no backup of the requested version exists, the play fails with a list of
available backups and the operator can either pick a version that does have
one or override with `-e allow_downgrade=true`.

### B. Explicit — direct rollback.yml invocation

List available backups:
Use this when you want to roll back to a specific backup directory (or the
most recent one) rather than a specific version.

**Most recent backup (default):**

```bash
ansible-playbook -i inventory/customers/<name>/ rollback.yml \
-e @secrets/<name>.yml --vault-password-file .vault-pass
```

**Specific timestamped backup** (list first, then pass):

```bash
ansible -i inventory/customers/<name>/ openspecimen -b \
-a "ls /usr/local/openspecimen/backup/"
# example output: 17052026_093247 18052026_104530 19052026_110042

ansible-playbook -i inventory/customers/<name>/ rollback.yml \
-e backup_timestamp=17052026_093247 \
-e @secrets/<name>.yml --vault-password-file .vault-pass
```

Then pass the chosen timestamp:
**Specific OpenSpecimen version** (let the playbook locate the matching backup):

```bash
ansible-playbook -i inventory/customers/<name>/ rollback.yml \
-e backup_timestamp=17052026_093247 \
-e target_version=openspecimen_v12.1.RC8 \
-e @secrets/<name>.yml --vault-password-file .vault-pass
```

Expand Down
67 changes: 67 additions & 0 deletions roles/openspecimen/tasks/direction.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
# Detect deploy/rollback direction from openspecimen_release vs the on-target
# .release marker. When a downgrade is detected, the rollback flow is invoked
# automatically (using openspecimen_release as the target_version), then the
# play ends — site.yml's other roles do not run.
#
# Called as the FIRST pre_task from site.yml and deploy.yml.

- name: Validate openspecimen_release is set (required for direction detection)
ansible.builtin.fail:
msg: |
✗ openspecimen_release is not set.

What to do:
Pass at run time: -e openspecimen_release=openspecimen_v12.2.RC12
From Jenkins: select RELEASE_FILE in the deploy job.
when: not (openspecimen_release | default('') | trim)

- name: Read installed version marker on target
ansible.builtin.slurp:
src: "{{ openspecimen_release_marker }}"
register: _direction_marker
failed_when: false
changed_when: false

- name: Set installed release fact
ansible.builtin.set_fact:
_installed_release: >-
{{ (_direction_marker.content | b64decode | trim)
if _direction_marker.content is defined else '' }}

# Use sort -V (natural version sort) so RC12 > RC8 sorts correctly.
- name: Detect direction (deploy / rollback)
ansible.builtin.shell: |
INSTALLED="{{ _installed_release }}"
REQUESTED="{{ openspecimen_release }}"
if [ -z "$INSTALLED" ] || [ "$INSTALLED" = "$REQUESTED" ]; then
echo deploy
elif [ "$(printf '%s\n%s\n' "$INSTALLED" "$REQUESTED" | sort -V | head -1)" = "$REQUESTED" ]; then
echo rollback
else
echo deploy
fi
register: _direction
changed_when: false

- name: Announce detected direction
ansible.builtin.debug:
msg: >-
{% if _direction.stdout == 'rollback' %}
⤺ Downgrade detected: installed {{ _installed_release }} → requested {{ openspecimen_release }}.
Switching to ROLLBACK flow — will restore from the backup whose .release matches {{ openspecimen_release }}.
{% elif _installed_release %}
→ Deploy: installed {{ _installed_release }} → requested {{ openspecimen_release }}.
{% else %}
→ Fresh install: requested {{ openspecimen_release }}.
{% endif %}

- name: Run rollback flow when downgrade detected
ansible.builtin.include_tasks: rollback.yml
vars:
target_version: "{{ openspecimen_release }}"
when: _direction.stdout == 'rollback'

- name: End play after rollback (skip deploy roles)
ansible.builtin.meta: end_play
when: _direction.stdout == 'rollback'
Loading