Add check_multi

[lgmu]

By default 'CheckMulti' is enabled, but you can disable it in the '[/modules]' section of the snclient_local.ini.
You can also set 'max checks' in the '[/settings/check/multi]' section of the snclient_local.ini, which limits
the number of checks that can be configured.

When using the inline mode, you can only use available commands (run 'check_index' to get a full list).

You can also define custom check sections in the config file, for example:
[/settings/check/multi/mycheck]
command[alias1] = check_process process=123
command[alias2] = check_process process=345

This can be executed with 'check_multi "config=mycheck"'.

It's also possible to use custom scripts in the config section, for example:
[/settings/check/multi/myscript]
command[alias1] = /path/to/plugin1
command[alias2] = /path/to/plugin2
command[alias3] = /path/to/plugin3

This can be executed with 'check_multi "config=myscript"'.

---

Examples: 

    check_multi "command[check_process]=check_process 'process=firefox'" "command[check_memory]=check_memory 'type=physical' 'crit=used_pct gt 80%'"
OK - 2 plugins checked, 2 ok |'check_process::count'=1;;;0 ... 'check_memory::physical %'=78.7%;;;0;100
[check_process] OK - all 1 processes are ok.
[check_memory] OK - physical = 12.59 GiB/16.00 GiB (78.7%)

You can define 'warning' and 'critical' conditions based on the number of checks in a certain state (see attributes below):

check_multi "command[check_dummy1]=check_dummy 0 'OK - check works'" "command[check_dummy2]=check_dummy 1 'WARNING - problem found'" "critical=problem_count gt 0"
CRITICAL - 2 plugins checked: 1 ok, 1 warning, 0 critical, 0 unknown - warning(check_dummy2: WARNING - problem found)
[check_dummy1] OK - check works
[check_dummy2] WARNING - problem found

You can also override the 'top-syntax' and use IF ELSE statements to get a certain output based on the results:

check_multi "command[check_dummy1]=check_dummy 0 'OK'" "command[check_dummy2]=check_dummy 2 'CRITICAL'" \
			"top-syntax={{ if ok_count gt 0 }}OK - %(ok_count)/%(count) checks are OK {{ ELSE }}CRITICAL - all checks failed{{ END }}"
OK - 1/2 checks are OK
[check_dummy1] OK
[check_dummy2] CRITICAL

[lgmu]

Edit: Done

[Copilot]

Pull request overview

Adds a new check_multi command that can run multiple checks (inline or from config sections) and aggregate status/output/perfdata, along with config and documentation updates to support it.

Changes:

• Introduce new pkg/snclient/check_multi.go check implementation plus comprehensive tests.
• Extend INI parsing/serialization to allow “bare lines” in /settings/check/multi/* sections (raw command/script lines).
• Enhance list-macro generation in CheckData (unknown counts, warning/critical aliases) and adjust an existing check_files expectation accordingly.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
README.md	Adds check_multi to the documented command list.
pkg/snclient/config.go	Allows raw/bare command lines in /settings/check/multi/* INI sections and serializes them without =.
pkg/snclient/checkdata.go	Adds unknown/warning/critical list macros and disables filter for checks that opt out.
pkg/snclient/check_multi.go	Implements check_multi execution/aggregation (inline + config modes).
pkg/snclient/check_multi_test.go	Adds test coverage for inline/config modes, limits, disabling, and check_index visibility.
pkg/snclient/check_files_test.go	Updates expected output after list-macro counting changes.
packaging/snclient.ini	Enables CheckMulti by default and documents max checks.
Makefile	Includes check_multi in doc generation command list.
docs/checks/commands/check_multi.md	Adds user-facing documentation for check_multi.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Suppressed comments (9)

pkg/snclient/check_multi.go:186

• Config.Section creates and registers a section when it does not exist. Because config is check input, repeated probes of unique missing names permanently grow config.sections and may later serialize those empty sections. Look up the section without the creating accessor.

	sec := snc.config.Section(secName)

	if len(sec.keys) == 0 {

pkg/snclient/check_multi.go:269

• Preserving each child state in _state causes CheckData.setStateFromMaps to unconditionally promote any nonzero crit_count/warn_count, after the user-defined aggregate conditions run. Consequently, a critical child still makes the result CRITICAL with critical=none, so custom conditions cannot lower or remap child severities as advertised. Add a check_multi-specific way to disable this implicit list-state promotion and let its aggregate thresholds determine the result.

			"_state":  fmt.Sprintf("%d", res.State),

pkg/snclient/check_multi.go:278

• The shallow copy retains each built-in child's Warning and Critical condition lists. Parent check.Finalize() calls CheckMetrics, re-evaluates those child thresholds, and can escalate the aggregate result independently of check_multi's configured conditions. This also makes critical=none ineffective for metric-based child failures even after list-state promotion is addressed. Preserve the perfdata thresholds but skip metric threshold evaluation for the aggregate check.

		for _, m := range res.Metrics {
			metricCopy := *m
			metricCopy.Name = fmt.Sprintf("%s::%s", tag, m.Name)
			allMetrics = append(allMetrics, &metricCopy)

pkg/snclient/check_multi.go:260

• Only the child's primary Output is aggregated, so checks that return additional diagnostic lines in CheckResult.Details silently lose them. Use BuildOutputString() to include the complete child output.

		detailsList = append(detailsList, fmt.Sprintf("[% 2d] %s %s", idx+1, tag, res.Output))

pkg/snclient/checkdata.go:119

• Adding unknownThreshold makes unknown= available to every check, but the existing threshold-normalization paths were not extended: transformKeywordsUsingAttributes, disableGenerallizedConditionsUsingAttributes, and the drive-specific blacklist/keyword transforms still process only warning, critical, and OK lists. Unknown conditions using aliases or drive-specialized keywords therefore behave differently and may not match the same entries. Include unknownThreshold in those normalization paths or scope the argument to checks that support it.

	unknownThreshold              ConditionList
	defaultUnknown                string

pkg/snclient/config.go:352

• This detects only a literal space. A valid command separated from its first key=value argument by a tab (for example, check_process\tprocess=123) falls through as a named INI entry and later tries to execute 123 instead of check_process. Treat tabs as command whitespace too.

		if isMultiSection && strings.Contains(val[0], " ") {

pkg/snclient/config.go:340

• SetRaw interprets any key ending in + as INI append syntax and strips the suffix. Here the entire line is a command, so a bare executable/argument ending in + is silently changed before execution. Store raw command keys without applying key-append semantics.

			if err := currentSection.SetRaw(line, ""); err != nil {
				parseErrors = append(parseErrors, fmt.Errorf("config error in %s:%d: %s", iniPath, lineNr, err.Error()))
			}

pkg/snclient/config.go:355

• This raw-command path also passes the whole command through SetRaw, which strips a trailing + as append syntax. For example, check_dummy message=+ is persisted and executed as check_dummy message=. Store the command key without applying INI key operators.

			if err := currentSection.SetRaw(line, ""); err != nil {
				parseErrors = append(parseErrors, fmt.Errorf("config error in %s:%d: %s", iniPath, lineNr, err.Error()))
			}

docs/checks/commands/check_multi.md:105

• This states that the attributes can be used in filters, but check_multi explicitly rejects both filter and filter+ (disableFilter: true). Update the help/documentation generator for filter-disabled checks so this section describes threshold keywords only; otherwise users are directed to an option that always returns UNKNOWN.

### Filter Keywords

these can be used in filters and thresholds (along with the default attributes):

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (6)

pkg/snclient/check_multi.go:183

• max checks limits only this invocation's immediate children. Because child commands can themselves be check_multi, a valid config can fan out to 20^6 executions before the depth guard, and fast handlers such as check_dummy ignore an expired context. Track a shared remaining budget in the context and decrement it for every child so nested configurations cannot bypass the configured execution limit.

	if int64(len(childChecks)) > maxChecks {
		return &CheckResult{
			State:  CheckExitUnknown,
			Output: fmt.Sprintf("number of checks (%d) exceeds max checks limit (%d)", len(childChecks), maxChecks),
		}, nil

pkg/snclient/config.go:801

• This serialization branch emits a bare key, but ParseINI rejects every non-comment line without = (lines 327-332). An empty value in a multi section therefore cannot survive a save/reload round trip. Keep the normal key = representation unless matching parser support for bare commands is added.

				if strings.HasPrefix(cs.name, "/settings/check/multi/") {
					data = append(data, key)
				} else {
					data = append(data, fmt.Sprintf("%s =", key))

pkg/snclient/check_multi.go:128

• A missing CheckMulti key is treated as disabled because GetBool returns false when absent, while DefaultConfig["/modules"] does not define this module. Minimal/custom configurations therefore contradict the documented “enabled by default” behavior (the new tests all explicitly enable it). Add the module to the programmatic defaults, not only the packaged INI.

	enabled, _, _ := snc.config.Section("/modules").GetBool("CheckMulti")
	if !enabled {

pkg/snclient/check_multi.go:309

• CheckResult.Output excludes Details; the public BuildOutputString method is what combines them. Consequently, nested checks that produce detail output silently lose those lines here, and the output attribute also sees incomplete output. Build one combined child output and use it for both aggregation and the entry attribute.

		firstLine := strings.TrimSpace(strings.Split(res.Output, "\n")[0])
		detailsList = append(detailsList, fmt.Sprintf("[%s] %s", tag, res.Output))

		entryState := fmt.Sprintf("%d", res.State)
		entry := map[string]string{
			"name":        tag,
			"tag":         tag,
			"command":     chk.cmdStr,
			"state":       entryState,
			"status":      res.StateString(),
			"shortoutput": firstLine,
			"output":      res.Output,

pkg/snclient/check_multi.go:336

• These shallow copies retain each child metric's Warning and Critical condition lists. Parent check.Finalize() then calls CheckMetrics on them, so a child metric can re-escalate the aggregate even when check_multi's custom count/entry thresholds intentionally classify that child as OK; metric-free children behave differently. Preserve the child thresholds for perfdata rendering, but prevent inherited child conditions from participating in the parent's state calculation.

		for _, m := range res.Metrics {
			metricCopy := *m
			metricCopy.Name = fmt.Sprintf("%s::%s", tag, m.Name)
			allMetrics = append(allMetrics, &metricCopy)

pkg/snclient/check_multi.go:81

• The PR description advertises inline check=... arguments and untagged config entries, but this API only registers command as a tagged list and requires command[tag]=...; the advertised invocations return UNKNOWN. Either retain compatibility with the described syntax or update the PR contract and examples to make this breaking syntax choice explicit.

			"command": {value: &l.commands, description: "Check command to execute with mandatory unique tag, e.g. command[tag]=..."},
			"config":  {value: &l.config, description: "Config section name under [/settings/check/multi/< section >] to execute"},

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

pkg/snclient/checkdata.go:1125

• Bracket suffixes are normalized before checking the destination type, so this makes tagged spellings valid for every existing check argument, not just command. For example, path[x]=... is now accepted as path, and differently tagged list arguments can repeatedly clear one another because hasArgsSupplied uses the full keyword. Reject bracket syntax unless the resolved argument is a TaggedCommandList.

	if before, rest, found := strings.Cut(keyword, "["); found && strings.HasSuffix(keyword, "]") {
		lookupKey = before
		tag = rest[:len(rest)-1]

pkg/snclient/check_multi.go:312

• Child output is copied into CheckResult.Details, which CheckData.finalizeOutput later processes with CheckResult.Finalize. As a result, literal child text such as %(count), %(status), or {{ IF ... }} is interpreted using the parent's macros and can be rewritten or removed. Aggregate plugin output as literal text rather than passing untrusted child output back through the template engine.

		firstLine := strings.TrimSpace(strings.Split(childOutput, "\n")[0])
		detailsList = append(detailsList, fmt.Sprintf("[%s] %s", tag, childOutput))

pkg/snclient/check_multi.go:259

• Config-mode tags are not trimmed, unlike inline tags in parseTaggedCommand. Consequently command[foo] and command[ foo ] are treated as distinct tags, bypassing the uniqueness check and producing whitespace-dependent detail/performance labels. Normalize the extracted config tag before validation and duplicate detection.

		tag := strings.TrimSuffix(strings.TrimPrefix(key, "command["), "]")