Support offline operation (no internet access)

[gsanchietti]

Description

NS8 has been designed to work with an internet connection. This is a new requirement from a special scenario: support offline/no-internet operation. Currently, several components assume continuous connectivity and degrade poorly:

1. Applications page / Cluster status UI ("Available updates", "Installed applications" cards) hang ~2 minutes before rendering when the repo host is unreachable.
2. Backups fail when the restic container image needs to be re-pulled but there's no internet.
3. Related: journal noise from modules checking subscription/repo connectivity.

The goal is to make NS8 operational in air-gapped or intermittently-offline clusters by:

• Failing fast instead of blocking on network timeouts
• Using locally cached data/images as fallback
• Optionally introducing a "cluster offline" mode flag so code can degrade gracefully

Alternative solutions

1. Do nothing: NS8 requires internet. Users who need offline mode must either provide connectivity or accept non-functional features.
2. Quick timeout fix only: Add explicit timeout= to HTTP calls so unreachable hosts fail in seconds instead of ~2 min. Does not address backup image re-pulls or provide graceful degradation.
3. Full offline mode with flag: Introduce a "cluster offline" or "air-gapped" configuration flag. Code checks this flag to skip network calls entirely and serve cached data. Requires more design (when/how to set the flag, what falls back to cache, what remains broken).
4. Hybrid: Fix HTTP timeouts immediately (fast fail), redesign image cleanup + caching for restic, document what works offline vs what requires internet.

Root cause details (for implementers)

UI slowness (modules.py)

• core/imageroot/usr/local/agent/pypkg/cluster/modules.py:128-140 _get_http_session() sets osession.timeout = 15, which is a no-op on requests.Session — the actual .get() call (line 154, inside _list_repository_modules) passes no timeout= kwarg, so each retry attempt (1 initial + 3 retries) blocks on the OS-level TCP timeout instead of failing fast.
• Called from UI via list-modules/list-core-modules cluster actions (ClusterStatus.vue:762-878, ApplicationsCenter.vue:855+).
• Frontend has no independent timeout guard — UI spinner waits indefinitely for the backend task.

Backup failures (restic image)

• core/imageroot/var/lib/nethserver/node/update-core.d/96cleanup_restic_images unconditionally removes all cached restic image tags on every core update.
• The next backup then has no local image and must pull from ghcr.io with no offline fallback.
• prepare_restic_command() (agent/pypkg/agent/__init__.py:215-271) never specifies --pull=missing or similar protection.

Tested workaround (proves the hang location)

--- a/modules.py
+++ b/modules.py
@@ -147,6 +147,7 @@
     url = _urljoin(repository_url, "repodata.json")
     hsubscription = rdb.hgetall("cluster/subscription") or None
     try:
+        return []  # HACK: offline mode
     with _get_http_session() as osession:

Current logs (offline environment)

Jul 07 16:06:59 ns8 agent@node[3314501]: Trying to pull ghcr.io/nethserver/restic:3.20.0...
Jul 07 16:07:29 ns8 agent@node[3314501]: time="2026-07-07T16:07:29+02:00" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://ghcr.io/nethserver/restic:3.20.0: pinging container registry ghcr.io: Get \"https://ghcr.io/v2/\": dial tcp 140.82.121.33:443: i/o timeout"
Jul 07 16:08:00 ns8 agent@node[3314501]: time="2026-07-07T16:08:00+02:00" level=warning msg="Failed, retrying in 1s ... (2/3). Error: ...same..."
Jul 07 16:08:31 ns8 agent@node[3314501]: time="2026-07-07T16:08:31+02:00" level=warning msg="Failed, retrying in 1s ... (3/3). Error: ...same..."
Jul 07 16:09:03 ns8 podman[924198]: ... image pull-error ghcr.io/nethserver/restic:3.20.0 unable to copy from source docker://ghcr.io/nethserver/restic:3.20.0: ...i/o timeout
Jul 07 16:09:03 ns8 agent@node[3314501]: [ERROR] restic stats failed for 86d1a8ac-ef89-557a-8e19-8582ab86b7c4/traefik/e410ec36-ab9d-4eb9-9dc0-4e3ae4710825: Command '['podman', 'run', ...]' returned non-zero exit status 125.

Jul 07 16:05:19 ns8 promtail[3314125]: ts=2026-07-07T14:05:19.253352728Z level=info msg="reporting Alloy stats" date=2026-07-07T14:05:19.253Z
Jul 07 16:05:19 ns8 node_exporter[3313990]: time=2026-07-07T14:05:19.520Z level=ERROR source=textfile.go:225 msg="inconsistent metric help text" collector=textfile metric=node_backup_status original_help_text="Status of the backup (0 = failure, 1 = success, -1 = unknown)" new_help_text="Status of the backup (0 = failure, 1 = success, 2 = conflict, -1 = unknown)" file=/host/run/node_exporter/backup.prom

References

• Customer ticket: https://helpdesk.nethesis.it/a/tickets/203497