Fix CI workflow syntax and restore green build - #41
Open
alekhyaakkiraju-droid wants to merge 66 commits into
Open
Fix CI workflow syntax and restore green build#41alekhyaakkiraju-droid wants to merge 66 commits into
alekhyaakkiraju-droid wants to merge 66 commits into
Conversation
RPM-2 Fixes #34 Do not run tests during package building
Add PEP 621 packaging metadata with hatchling build backend, Python 3.12+ target, modern runtime dependencies, optional test extras, and tool configs for ruff, mypy, black, isort, and pytest. Forge: WO-4581be7b Co-authored-by: Cursor <cursoragent@cursor.com>
Add centralized hacklog/config.py with typed settings for syslog, SMTP, scoring, database, and security. Load from YAML with environment variable overrides, fail fast on missing SMTP secrets, and document variables in .env.example. Includes 9 pytest unit tests. Forge: WO-7dc26aa5 Co-authored-by: Cursor <cursoragent@cursor.com>
Add logging_config module with JSON output, ISO timestamps, secret redaction, and configurable PII masking. Replace stdlib logging in server.py and algorithm.py and add structured logs to services.py and accessdata.py. Includes 6 pytest unit tests. Forge: WO-992cc2d6 Co-authored-by: Cursor <cursoragent@cursor.com>
Define operational counters, histograms, and gauges in hacklog/metrics.py, add optional /metrics HTTP exposition on port 9090 controlled by HACKLOG_METRICS_ENABLED, and include 7 pytest unit tests. Forge: WO-5e41c0f2 Co-authored-by: Cursor <cursoragent@cursor.com>
Refactor EmailService to accept SmtpConfig from ConfigManager, wire startup through load_config_or_exit(), and remove all hardcoded SMTP secrets and addresses. Adds email service unit tests, required HACKLOG_SMTP_SENDER env var, and bandit/grep verification. Forge: WO-6b24f477 Co-authored-by: Cursor <cursoragent@cursor.com>
Add hacklog/security.py with CIDR allowlisting, message size limits, token-bucket rate limiting, Prometheus drop counters, and structlog rejection logging. Config merges HACKLOG_ALLOWED_CIDRS from env. 13 new tests; full suite 39 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
Use MutableDict.as_mutable(JSON) on Days/Hours/Servers/IpAddress profile columns. Add Alembic migration with pre-migration backup, batch_alter_table pickle-to-JSON conversion, and downgrade. Add JSON entity and migration tests; fix accessdata characterization tests for Python 3.12. 56 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace Py2 imports (ConfigParser/Queue/thread), add explicit UTF-8 UDP decode, IntEnum constants, and type hints on public APIs. Disable configparser interpolation for regex patterns. Add twisted and mockito deps. All 71 tests pass on Python 3.12. Co-authored-by: Cursor <cursoragent@cursor.com>
Delete tests/compat.py, replace mockito with unittest.mock, remove _Compat mixins, simplify tests/__init__.py. All 71 tests pass on Python 3.12. Co-authored-by: Cursor <cursoragent@cursor.com>
Validate CSV CLI paths with resolve_csv_input_path to block directory traversal. Centralize demo random PID/port helpers with NOSONAR for non-cryptographic syslog replay traffic. Co-authored-by: Cursor <cursoragent@cursor.com>
(WO-012) Migrate test suite to unittest.mock
…tocol Added hacklog/syslog_server.py with SyslogProtocol, bounded asyncio.Queue(10000), message_consumer coroutine, WO-009 security validation, and SIGINT/SIGTERM graceful shutdown. Rewrote server.py to use asyncio.run(). Removed twisted from pyproject.toml, setup.py, and hacklog.spec. Added tests/test_syslog_server.py with unit, UDP integration, and WO-002 corpus E2E tests. 80/80 tests passing. User Story: Implement asyncio SyslogProtocol replacing Twisted UDP Priority: P0 Status: in_progress Co-authored-by: Cursor <cursoragent@cursor.com>
Replace declarative_base with DeclarativeBase, Session.query with select() + Session.execute(), and wrap all DAO operations in context managers. Update test_entities_json.py to use select(). Add mergeEntity characterization test. 81/81 tests pass with SQLALCHEMY_WARN_20=1. User Story: Upgrade SQLAlchemy to 2.0 with modern patterns Priority: P0 Status: in_progress Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce ScoringEngine with injected UpdateService and EmailService. Remove algorithm.py globals and setServices(). create_db_engine now returns an Engine; wiring happens in SyslogServer.start(). Message queue is a SyslogServer instance attribute passed to syslog_server. Added test_scoring_engine.py and test_scoring_pipeline.py. 88/88 passing. User Story: Eliminate global mutable state with dependency injection Priority: P1 Status: in_progress Co-authored-by: Cursor <cursoragent@cursor.com>
Add ProfileRepository, UserRepository, and AuditRepository with injected session factories and context-managed sessions. Update UpdateService to use repositories; keep accessdata DAO wrappers for WO-003 compatibility. Added test_repositories.py with CRUD, rollback, and DI tests. 96/96 passing. User Story: Implement repository pattern for data access layer Priority: P1 Status: in_progress Co-authored-by: Cursor <cursoragent@cursor.com>
Add AlertService with aiosmtplib, circuit breaker, retry with backoff, and dead letter queue. Remove legacy EmailService and wire AlertService into the scoring pipeline and server startup. Co-authored-by: Cursor <cursoragent@cursor.com>
[WO-cf00b894] Rewrite AlertService with async SMTP and circuit breaker
Rename camelCase methods to snake_case, Servers to Server with Alembic migration, parse_config/db_file typo fixes, read_csv module rename, and black/isort/ruff formatting across the codebase. Co-authored-by: Cursor <cursoragent@cursor.com>
SonarCloud flagged path construction in scripts/wo018_rename.py as a blocker vulnerability on new code. The bulk rename is complete, so the dev-only script is removed from the repository. Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce validators module to reject invalid usernames, IP addresses,
and hostnames before EventLog creation. Invalid messages log a warning,
increment messages_dropped_total{reason=invalid_field}, and return None
from the parser. Includes unit and integration tests with injection fixtures.
Co-authored-by: Cursor <cursoragent@cursor.com>
The validator module was added but parse_log_line never called validate_parsed_fields before returning EventLog instances. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace deprecated Travis CI with a matrix workflow (Python 3.12/3.13) running ruff, black, isort, mypy on typed modules, bandit, and pytest with coverage. Add dev optional dependencies and document branch protection rules in README. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace editable pip install with locked requirements-ci.txt and --only-binary=:all: to resolve SonarCloud S8541/S8544 findings. Co-authored-by: Cursor <cursoragent@cursor.com>
[WO-432d5be9] Fix naming conventions typos and enforce PEP 8
…zation [WO-f1e93c38] Add input sanitization for parsed syslog fields
Fix render_event_dict return type for mypy on Python 3.13, add --follow-imports=skip to typed mypy step, and use single locked pip install without pip self-upgrade for Sonar S8544. Co-authored-by: Cursor <cursoragent@cursor.com>
[WO-5d05121b] Create GitHub Actions CI/CD pipeline
… alert events [tests: failing] Implements WO-025: append-only audit trail for scoring and alerting events. Adds AuditRecord entity and migration (003_create_audit_table), extends AuditRepository with save_audit_record (no update/delete), integrates audit record creation into ScoringEngine (score_calculated, scare_count_updated, scare_count_reset) and AlertService (alert_sent, alert_suppressed). Audit events are emitted as structlog entries with audit=True tag for external aggregation and optionally persisted to DB. All 13 new tests cover unit, integration, and system-level scenarios. ## How to Test cd /workspace/f757b220/repo && python -m pytest tests/test_audit.py tests/test_repositories.py tests/test_scoring_engine.py tests/test_alerting.py -v
…purge [tests: failing] Implements WO-026: configurable data retention with automated purge. Adds RetentionConfig to ConfigManager (env vars: HACKLOG_EVENT_RETENTION_DAYS, HACKLOG_PROFILE_INACTIVITY_DAYS). DataRetentionService in hacklog/retention.py provides batched physical DELETE for event logs and inactive user profiles, structlog audit logging with audit=True tag, AuditRecord persistence via WO-025's AuditRepository, and asyncio daily scheduling at a configurable UTC hour. 17 tests cover all acceptance criteria including retention boundary tests, idempotency, batch processing, audit records, and scheduler smoke testing.
…cumentation [tests: failing] Rewrites README.md with project description, 6-layer architecture diagram, Docker and bare-metal quick starts, complete HACKLOG_* configuration reference (including WO-026 retention vars), scoring algorithm documentation, and GitHub Actions badge. Creates CONTRIBUTING.md enabling 30-minute contributor onboarding with Python 3.12+ setup, pytest instructions, code style guide (black/isort/ruff/mypy), architecture overview, and PR process. All deprecated tool references (Travis CI, Python 2, RPM, init.d, setup.py) removed.
…11c-run3-3wo [Forge Batch] 3 work orders — forge/hacklog-0e55b11c-run3-3wo
- Delete setup.py (replaced by pyproject.toml) - Delete hacklog/run.sh and hacklog/stop.sh (replaced by systemd/Docker) - Confirm tests/compat.py and .travis.yml were already removed in prior WOs - Remove 'from __future__ import annotations' from all Python files (Python 3.12 native support) - Clean up blank lines left by import removal - Verified: no twisted, mockito, ConfigParser, Queue, thread imports remain - Verified: no python2/py2/Python 2 patterns remain
…11c-remove-legacy-files-and-deprec-WO-028 [WO-028] Remove legacy files and deprecated dependencies
User Story: End-to-end integration test for full pipeline Priority: P1 Status: in_progress Also fixes: test_audit.py import paths (hacklog.* alignment for SmtpConfig isinstance checks)
[WO-d9f64597] End-to-end integration test for full pipeline
…n, f-strings, required field checks, and unit tests User Story: Modernize readCSV syslog replay utility with proper data types and validation Priority: medium Status: in_progress
…rnization [WO-709d97fd] Modernize readCSV syslog replay utility
…rmat, and graceful row skipping User Story: Normalize and validate date-time parsing in CSV ingestion module Priority: medium Status: in_progress
…-validation [WO-b8802a73] Normalize CSV date-time parsing with logging and config
…asyncio message consumer; clarify is-not-None guard User Story: Implement graceful server shutdown and robust None-handling in message processing loop Priority: medium Status: in_progress
…tdown [WO-8b13aec7] Graceful shutdown and None-handling in message consumer
…k, signal handler cleanup, and DB dispose on server exit User Story: Implement graceful server shutdown with signal handling and resource cleanup Priority: medium Status: in_progress
…anup [WO-d8baa519] Graceful shutdown with resource cleanup
Replace ad-hoc process management with SIGTERM-based graceful shutdown, ConfigManager/.env loading, and documented make dev-start/dev-stop targets.
WO-039: Standardized dev run/stop scripts and Makefile
Cap calculate_subscore at 1.0 for normalized unit interval; add boundary and persistence tests; refresh golden vector for capped-frequency edge case.
…alization WO-040: Fix scoring subscore normalization and add behavioral tests
Extract SSH payload without treating the first data token as host; use SyslogMsg.host for server identity. Add characterization tests for the SyslogMsg parser interface.
…-msg WO-041: Refactor parse_log_line to accept SyslogMsg entity
Confirm hacklog/run.sh and hacklog/stop.sh are removed; document Makefile, scripts/, Docker, and systemd as replacements.
…ts-removal WO-042: Replace legacy run/stop scripts with modern process management
Confirm parse_log_line SyslogMsg interface and call-site wiring; extends WO-041 coverage with signature and message_consumer contract tests.
…-msg-contract WO-043: Verify SyslogMsg parser interface and call sites
Replace Days/Hours/Server/IpAddress ORM classes with Profile + ProfileType. Add Alembic migration 004 to merge legacy tables. Move IP classification to IpLocation. Fix inactive-user purge to delete EventLog rows.
WO-044: Consolidate profile entity classes into unified Profile model
Quote the pip install command in GitHub Actions to avoid YAML parse errors, apply lint/format fixes, update migration tests for the unified profiles table, and stabilize read_csv logging tests after Alembic runs. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pip install --only-binary=:all:command in.github/workflows/ci.ymlso GitHub Actions can parse the workflow (CI was failing in 0s with a YAML syntax error on line 32).ProfileTypeimport inretention.py.profilestable after WO-044.read_csvlogging tests by mocking the module logger instead of relying oncaplog, which breaks after Alembic runs in the full suite.Test plan
hacklog-0.0.5)Made with Cursor