Manage CloudWatch status-check alarms with VM lifecycle#21
Open
frank-veriff wants to merge 3 commits into
Open
Manage CloudWatch status-check alarms with VM lifecycle#21frank-veriff wants to merge 3 commits into
frank-veriff wants to merge 3 commits into
Conversation
Create per-VM status-check alarms at create, delete at terminate/gc, plus an alarm-sync command. Config-driven (cloudwatch_alarm_checks, _prefix, _topic, _desc_*); no-op when unconfigured, non-fatal on AWS errors.
Create per-VM status-check alarms at create, delete them at terminate/gc, prune orphans of externally-deleted VMs during gc. Adds alarm-sync command. Config-driven (cloudwatch_alarm_checks, _prefix, _topic, _desc_*); off by default, non-fatal on AWS errors.
Author
|
tests ran through, followed existing patterns. |
| client = self.get_cloudwatch() | ||
| try: | ||
| for pos in range(0, len(names), 100): | ||
| client.delete_alarms(AlarmNames=names[pos:pos + 100]) |
Collaborator
There was a problem hiding this comment.
while it's unlikely, with large volumes this may get throttled
| OKActions=actions) | ||
| except Exception as ex: | ||
| eprintf("WARNING: alarm create failed (%s/%s): %s", vm_id, check, ex) | ||
| return |
Collaborator
There was a problem hiding this comment.
maybe we should continue instead of returning?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional per-VM CloudWatch status-check alarms, managed with the VM
lifecycle like DNS/EIP/ENI attachments:
of VMs deleted outside vmtool
alarm-synccommand: re-ensure alarms for running role VMsConfig-driven, off by default:
cloudwatch_alarm_checks- comma list of system, instance, ebs(unset = feature fully inert)
cloudwatch_alarm_prefix- alarm name prefix (default vm-status-)cloudwatch_alarm_topic- optional SNS topic for system/ebs alarm actionscloudwatch_alarm_desc_<check>- description text per checkCompatibility: purely additive, no existing lines modified; without the
config keys every code path no-ops; alarm API failures warn on stderr and
never fail create/terminate. Verified with live create/terminate/gc
cycles; passes the test suite and .pylintrc lint.