Redact sensitive values from command output

[jordanenglish]

Description

Some API responses carry credentials, and tfctl prints them. A created token is returned once in full, and a state version carries hosted-state-download-url and hosted-json-state-download-url, which grant the state itself without a token. State holds every value Terraform wrote, marked sensitive or not.

The caller was authorized to fetch these, so this is not a permission failure. The problem is that a credential outlives the moment it was needed: output persists in scrollback and shell history, is captured wholesale by CI logs, and is read into context by coding agents, which this project ships a skill for. The download URLs are the sharpest case, because they are bearer credentials, so a copy of the output is a copy of the access.

AGENTS.md already states the rule:

Never include credentials or sensitive values in a displayer payload. JSON output serializes the full payload, not only the displayed field templates.

JSONAPIDisplayer cannot honor it, because it carries whatever the server returned. resource.ExcludeColumns does not help either: it feeds FieldTemplates, which drives table and pretty output, while outputJSON marshals the raw envelope. orderedFields then appends every attribute that is not excluded, so a plain get renders these URLs too, not only --json.

This adds internal/pkg/redact and applies it in format.Outputter.Display, before the format is selected, so every format agrees and a --jq filter cannot reach a value that --json would have hidden. It covers Outputter.CopyRaw for bodies no displayer handles, such as plan JSON, and the --dry-run request preview. This is a hygiene layer in the same sense as inputguard, not a security boundary: #85 did it for telemetry paths and #60 covered transmission and file permissions, leaving stdout as the gap.

Behavior

Three modes, set by the new redact profile property or TFCTL_REDACT. Precedence is --no-redact, then TFCTL_REDACT, then the profile property, then the default.

Mode	Masks
strict (default)	Known secret fields, declared-sensitive values, and values whose name or shape indicates a credential
known	Known secret fields and declared-sensitive values only. The escape hatch when a heuristic hides something needed, without turning masking off
off	Nothing. --no-redact is off for a single command

What counts as sensitive:

Rule	Matches	Mode
Known secret field	token, secret, private-ssh-key, encryption-password, and the authorization / proxy-authorization headers	all
Capability URL	Any name containing download-url, upload-url, or log-read-url, which covers state versions, configuration versions, plan exports, and plan or apply log reads	all
Declared sensitive	value where the object sets "sensitive": true, or where the name the object gives itself indicates a credential, as in variables.db_password.value	all
Recognizable shape	PEM private key, JWT, *.atlasv1.*, Vault hv[sbr]., GitHub gh?_, AWS AKIA/ASIA, or a presigned URL carrying X-Amz-Signature	strict

The shape rules are deliberately narrow. They recognize formats rather than guess at entropy, so an unnamed field holding an unrecognizable secret is not caught.

Notes:

• Withheld values stay withheld. A null stays null rather than becoming a placeholder, so output still shows that the API held the value back.
• Names that describe a credential are kept. oauth-token-id still renders; masking it would break every workflow that needs the VCS connection.
• The sensitive marker is never masked, since it is what drives the declared-sensitive rule.
• The placeholder is (redacted), matching the style Profile.String uses for the stored token. Angle brackets are avoided because encoding/json escapes them.
• Masked fields are reported once on stderr, naming the flag needed to show them.
• A bad redact value falls back to strict and reports why, rather than failing every command.
• Copying is on write. A 4.6 MB plan JSON holding no credential costs 51 MB of allocations against 4.6 MB for streaming it unmasked, and the walk itself allocates zero.

Example Output

A state version download URL is an opaque credential: it grants the state without a token. Before, it renders in --json, in --jq, and in the default output. The URL is elided here:

$ tfctl api /workspaces/{workspace}/current-state-version -p workspace=my-workspace \
    --jq '.data.attributes["hosted-state-download-url"]'
https://app.terraform.io/.../<opaque token, no credential required>

After:

$ tfctl api /workspaces/{workspace}/current-state-version -p workspace=my-workspace \
    --jq '.data.attributes["hosted-state-download-url"]'
(redacted)
WARNING: masked 2 sensitive fields: hosted-json-state-download-url, hosted-state-download-url. Use --no-redact to show them.

The default output of a single get is masked the same way, since it renders every attribute that is not excluded:

$ tfctl get sv sv-abc123
Serial:                          42
Status:                          finalized
Hosted Json State Download URL:  (redacted)
Hosted State Download URL:       (redacted)
...

A --dry-run preview no longer echoes the value being set:

$ tfctl api /workspaces/{workspace}/vars -p workspace=my-workspace \
    -X POST -a key=db_password -a value=s3cr3t --dry-run
DRY RUN: would send POST request
> POST https://app.terraform.io/api/v2/workspaces/ws-abc123/vars
> Accept: */*
> Content-Type: application/vnd.api+json

{
  "data": {
    "attributes": {
      "key": "db_password",
      "value": "(redacted)"
    },
    "type": "vars"
  }
}
WARNING: masked 1 sensitive field: value. Use --no-redact to show it.

Names that only describe a credential are untouched, and an ordinary workspace renders exactly as before:

$ tfctl get ws my-workspace --jq '.data.attributes["vcs-repo"]'
{
  "branch": "main",
  "identifier": "my-org/my-repo",
  "oauth-token-id": "ot-abc123"
}

Tests

• internal/pkg/redact: table tests for each rule in both directions. Values that must be masked, and values that must survive, including oauth-token-id, the sensitive marker, an ordinary URL, and a value the server already withheld.
• internal/pkg/format: a synthetic corpus of nine responses (state version download URLs, a created token, an upload URL, a declared-sensitive variable, six variables holding secrets nobody marked sensitive, an OAuth client secret, a log read URL, a workspace that must survive intact, and a raw plan JSON body). For each, the invariant is asserted in every output format, including the default format and a --jq filter aimed directly at the attribute. Every credential in the corpus is invented: either a published documentation example or a literal that says it is not real.
• TestRedactCorpus_LeaksWithoutTheRedactor renders the same corpus with no redactor and requires every credential to appear, so a corpus that was empty or misspelled cannot make the masking test pass vacuously.
• Copy-on-write is pinned: the input is never written to, untouched subtrees are shared rather than copied, and a document with nothing to mask walks with zero allocations.
• internal/commands/api: the command path against a routed test server for a state version, a created token, a raw JSON body, the --dry-run preview, a sensitive header, --no-redact, and a body that cannot be parsed.
• internal/pkg/cmd: mode resolution across flag, environment, and profile, including the fallback on an unusable value.
• gofmt, golangci-lint run, go test ./..., and go test ./... -race clean.

PR Checklist

• Run npx changie new or install changie to prepare a new changelog entry for the next set of release notes.
  • Added three entries: BUG FIXES for the leak, ENHANCEMENTS for the redact setting and --no-redact flag, and NOTES describing the change to default output.
• Ensure any command changes are sensitive to these global flags:
  • --json — Force machine readable output to stdout. Does not apply to stderr.
  • --markdown — Force markdown output to stdout. Does not apply to stderr.
  • --dry-run — Don't make any actual writes or other mutations. Describe what would have changed to stderr.
  • --quiet — Only render essential content.
  • Masking is applied in Display before the format is selected, so --json, --markdown, pretty, table, and agent output are all covered and stay consistent with each other; --jq cannot reach around it. --dry-run previews are masked with the same rules and remain on stderr. The report of masked fields goes to ErrUnessential(), so --quiet suppresses it; the warning that masking is off is essential and survives --quiet. Byte-for-byte raw output is preserved when nothing was masked.
• Get the logging interface from the context and add debug logging for interesting conditions and nonfatal situations.
  • Added for the two nonfatal paths: a JSON:API envelope that cannot be masked, and a raw body whose content type claims JSON but does not parse. An unusable redact value is reported to the user on stderr rather than only logged, because it silently changes protection.
• Run make gen/screenshot if the root command output changes.
  • Regenerated: this adds the --no-redact global flag, which changes root command output.
• Add the Autocomplete field to positional arguments and flags to assist shell autocomplete.
  • --no-redact is a boolean flag with no value to complete, and is registered in AutocompleteGlobalFlags alongside the other global flags. Profile.Predict completes the redact property and its three values.

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

  • Reverting the PR fully removes the change; no additional revert steps or data migration. A user who needs the previous output without a revert can set redact = "off" in their profile or pass --no-redact per command.

• If applicable, I've documented the impact of any changes to security controls.

  Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

  • This adds a control rather than modifying one. Nothing about authentication, authorization, or token handling changes, and no request is altered: masking applies only to what is rendered, so the requests tfctl sends are byte-for-byte what they were before. It is a hygiene layer and not a security boundary: the API still decides what a token may read, and masking can be disabled per command or per profile, so it must not be relied on as an enforcement point. It does change default output: a script that reads a state version download URL or a newly created token out of tfctl output will read (redacted) until it passes --no-redact. The stderr report names the masked field and the flag, and a NOTES changelog entry records the change.

[shwetamurali]

Awesome, thanks for contributing this!