opensearch role: unit inactive, authenticated API unreachable after 600s

[yakari007]

Summary

Deployed a VM with the opensearch service role attached, with a real
generated admin_password submitted (16 chars, mixed upper/lower/digit/
special - meets the role's own documented complexity requirement), on the
Ubuntu 24.x "with PlusClouds Service" image. After SSH login succeeded,
systemctl is-active opensearch reported inactive, and
https://127.0.0.1:9200/ (with basic auth against the submitted password,
falling back to plain http if https wasn't reachable at all) never became
reachable, even after waiting an extended 600s budget (to account for the
JVM startup + vm.max_map_count sysctl step this role's description
mentions).

Caveat: I have not independently confirmed the actual OpenSearch
systemd unit name on this image - opensearch is the standard name from
its official apt/dnf packaging, but if this role installs it under a
different unit name, systemctl is-active opensearch would report a
"not active"-ish result for the wrong reason. The HTTP check failing too
(a name-independent signal) is what makes this worth filing regardless,
but worth confirming service naming during triage. Also worth checking:
whether a valid admin_password was actually accepted/applied, since the
role's description says it "fails fast with a clear error" on an invalid
one - if it silently treated a technically-valid-but-unexpected password
as invalid, that would explain a stalled install.

Repro

POST /leo/iaas/virtual-machines/create-wizard with
service_roles: {"opensearch": {"config": {"admin_password": "<16-char password meeting the complexity rule>"}}}
against the Ubuntu 24.x "with PlusClouds Service" image, then after boot
(waited up to 600s):

systemctl is-active opensearch                               # -> inactive
curl -k -u admin:<password> https://127.0.0.1:9200/            # -> unreachable

The VM was deployed and torn down automatically as part of the test run,
so the live VM/install-script output is gone - filed from the test's
captured output, not a live inspection.

Found via leo4.tester's iaas-service-roles automated test scenario.

[yakari007]

Investigated but did not find a reproducible code-level bug, so no change made yet.

Checked the OpenSearch apt repo/key URLs (still live, package present), and downloaded+inspected the real opensearch-3.7.0-linux-x64.deb package's postinst script. One thing worth noting even though it doesn't look like the actual bug: OpenSearch's own postinst deliberately does not auto-enable/start the service on a fresh install (it only does so on upgrades) — it just prints instructions telling you to run systemctl enable --now yourself. The role already compensates for this correctly with its own explicit "Enable and start OpenSearch service" task, so that's not a gap. OPENSEARCH_INITIAL_ADMIN_PASSWORD is the correct, documented env var name and the role's fail-fast check matches. OpenSearch is JVM-heavy with its own vm.max_map_count and memory-lock requirements (already handled by the role) — if this keeps reproducing, I'd check actual available RAM on the test VM and journalctl -u opensearch from a live instance before assuming it's a role config bug.