From feaed744d9e13e4211847970d2a50fcc85932d45 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A9d=C3=A9ric=20Hurier=20=28Fmind=29?= Date: Mon, 10 Aug 2026 19:04:48 +0200 Subject: [PATCH] docs: teach the stack the repositories actually run, and add AGENTS.md The chapters described the previous stack. Every command, flag, path, and version in the rewritten pages was read out of the reference implementations. - 6.2 gains substantial AGENTS.md coverage, which the course had none of: what the format is and that the Agentic AI Foundation stewards it under the Linux Foundation, the README-for-humans / AGENTS.md-for-agents split, what belongs in each, a complete skeleton, the boundary where its authority stops, and the drift lesson from the vendored skill copies the package used to ship - 5.1 documents the `all` task as the named gate, mise.lock, and run_auto_install; the task table drops a `watch` task the package never had - 5.2 explains the 10/20/30 hook priorities and the three-tier secret scan - 5.3 carries the real hardened workflows, why the job is named `checks`, why the porcelain assertion beats git diff, job-level permissions replacing rather than merging, and a new section on linting workflows with actionlint + zizmor - 5.4 wires hadolint to check:dockerfile and explains the pinned image and the numeric user - 5.5 and 5.6 teach MLflow on SQLite, including that MLflow 3.14 already defaults to it, and the fixture that keeps a migrated database per session - 4.4 corrects `trivy config .` to `trivy --config trivy.yaml fs .` and explains both bugs; 4.1 and 4.5 cover Ruff 0.16; 4.2 adds the coverage gate - every chapter that said `uv add --group check` now says `--group dev`, which is the group all four repositories actually use Also pins the security workflow runner to ubuntu-24.04 so it matches CI. --- .github/workflows/security.yml | 2 +- docs/0. Overview/0.5. Assistants.md | 7 + docs/1. Initializing/1.3. uv (project).md | 41 ++- docs/3. Productionizing/3.0. Package.md | 18 +- docs/4. Validating/4.0. Typing.md | 28 +- docs/4. Validating/4.1. Linting.md | 74 ++++- docs/4. Validating/4.2. Testing.md | 134 +++++++++- docs/4. Validating/4.4. Security.md | 237 +++++++++++++++- docs/4. Validating/4.5. Formatting.md | 80 +++++- docs/4. Validating/index.md | 9 +- docs/5. Refining/5.1. Task Automation.md | 136 ++++++++-- docs/5. Refining/5.2. Pre-Commit Hooks.md | 59 +++- docs/5. Refining/5.3. CI-CD Workflows.md | 268 +++++++++++++++++-- docs/5. Refining/5.4. Software Containers.md | 78 +++++- docs/5. Refining/5.5. AI-ML Experiments.md | 126 +++++++-- docs/5. Refining/5.6. Model Registries.md | 43 ++- docs/5. Refining/index.md | 8 +- docs/6. Sharing/6.2. Readme.md | 178 +++++++++++- docs/6. Sharing/6.3. Releases.md | 4 +- docs/6. Sharing/6.4. Templates.md | 116 ++++++-- 20 files changed, 1474 insertions(+), 172 deletions(-) diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index 9f8ce23..bc18cc2 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -14,7 +14,7 @@ concurrency: jobs: scan: name: Full-history scan - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - name: Checkout complete history diff --git a/docs/0. Overview/0.5. Assistants.md b/docs/0. Overview/0.5. Assistants.md index 7e5da66..fb5d056 100644 --- a/docs/0. Overview/0.5. Assistants.md +++ b/docs/0. Overview/0.5. Assistants.md @@ -34,3 +34,10 @@ Access to the assistant is available for a monthly fee of $10. To subscribe, ple While our assistant is highly accurate—with an estimated 90% precision rate—it is crucial to approach its output with a critical mindset. Like any AI, it can occasionally make mistakes or generate suggestions that are not optimal for your specific context. Always verify critical information and use its output as a guide, not an absolute directive. For situations requiring guaranteed expert oversight, we highly recommend our [human mentoring](./0.4. Mentoring.md) service, which provides a perfect blend of AI-driven efficiency and human expertise. + +### What about the coding agent working inside my repository? + +The MLOps Coding Assistant answers questions *about* the course. It is a different tool from the coding agent that edits files in your own project, and that agent needs its own guidance to follow the practices taught here: + +- **`AGENTS.md`**: the open, tool-agnostic file that tells any coding agent your project's exact commands, definition of done, conventions, and layout. [6.2. Readme](../6. Sharing/6.2. Readme.md#what-is-agentsmd) explains what belongs in it, what it deliberately cannot enforce, and how to keep it from drifting away from your code. +- **[MLOps Coding Skills](https://github.com/MLOps-Courses/mlops-coding-skills)**: the methodology of this course packaged as [Agent Skills](https://agentskills.io/home), one skill per chapter, that your agent loads on demand. diff --git a/docs/1. Initializing/1.3. uv (project).md b/docs/1. Initializing/1.3. uv (project).md index a0ecd62..572656c 100644 --- a/docs/1. Initializing/1.3. uv (project).md +++ b/docs/1. Initializing/1.3. uv (project).md @@ -77,21 +77,24 @@ Your `pyproject.toml` becomes the single source of truth for your project's conf [project] name = "bikes" -version = "4.1.0" +version = "6.0.0" description = "Predict the number of bikes available." authors = [{ name = "Médéric HURIER", email = "github@fmind.dev" }] readme = "README.md" -license = { file = "LICENSE.txt" } +license = "MIT" +license-files = ["LICENSE.txt"] keywords = ["mlops", "python", "package"] requires-python = ">=3.14" dependencies = [ "loguru>=0.7.3", "matplotlib>=3.11.0", "mlflow>=3.14.0", + # numba/numpy/pyarrow/shap are ABI-coupled: keep permissive floors so uv's universal + # (all-platform) resolution stays satisfiable; the lockfile pins the latest tested versions. "numba>=0.61.0", "numpy>=2.1.3", "omegaconf>=2.3.1", - "pandas>=2.3.3", # MLflow 3.x pins pandas<3 + "pandas>=2.3.3", # MLflow 3.x pins pandas<3; pandas 3.0 is held back by MLflow compat "pandera>=0.32.1", "plotly>=6.8.0", "plyer>=2.1.0", @@ -122,12 +125,16 @@ bikes = 'bikes.scripts:main' # SYSTEMS [build-system] -requires = ["uv_build>=0.9.0"] +# Keep the upper bound at least one minor ahead of the pinned `uv` tool: without it +# `uv build` warns, and a future breaking `uv_build` would silently break the sdist. +requires = ["uv_build>=0.9,<0.13"] build-backend = "uv_build" ``` The `[build-system]` table tells packaging tools how to turn your source tree into a distributable artifact. This course uses [`uv_build`](https://docs.astral.sh/uv/concepts/build-backend/), uv's own fast, built-in build backend, so a single tool manages your dependencies, environment, and packaging. With it in place, `uv build` produces the wheel and source distribution ready to publish. +Note the **upper bound** on the requirement: `uv_build>=0.9,<0.13` rather than a bare `uv_build>=0.9.0`. Semantic versioning only promises a stable interface from 1.0 onwards; below it, a minor release is allowed to break things. `uv_build` is still pre-1.0, so an unbounded requirement means a future `0.13` could change how your source tree is packaged the next time anyone builds your project — and you would find out from a broken wheel, not from a failed resolution. `uv build` itself warns when the bound is missing. Keep the ceiling at least one minor ahead of the `uv` version you have pinned, and raise it deliberately after you have tested the new backend. + ## How do you manage project dependencies with `uv`? `uv` simplifies adding, removing, and updating dependencies. It automatically updates your `pyproject.toml` file and re-syncs your environment. @@ -171,6 +178,32 @@ The `uv.lock` file is a lockfile that records the exact versions of every packag While `pyproject.toml` might specify a version range (e.g., `pandas>=2.2`), `uv.lock` pins a specific version (e.g., `pandas==2.2.3`). When you run `uv sync`, `uv` will use the lockfile if it exists, ensuring that every developer on your team and every CI/CD run uses the exact same set of package versions. This prevents the "it works on my machine" problem and ensures that your builds are deterministic and stable over time. +## What do you do when a dependency blocks a security fix? + +Sooner or later a package you depend on will declare an upper bound that keeps you on a vulnerable version of one of *its* dependencies. This is not hypothetical: at the time of writing, MLflow 3.15.1 still declares `cryptography<50`, while the fix for `PYSEC-2026-3552` — a PKCS#7 Bleichenbacher oracle — only ships in `cryptography` 50.0.0. Resolve without intervention and you get 49.0.0, and your vulnerability scan fails. + +You have two ways out, and they are not equivalent: + +- **Suppress the finding**: add the advisory to an ignore list so the scanner stops reporting it. The vulnerable code is still installed and still running. You have removed the alarm, not the risk. +- **Override the constraint**: tell the resolver to ignore the stale upper bound and install the patched library anyway, then let your test suite prove the combination still works. The vulnerability is genuinely gone; the risk you have taken on is an untested version pairing, and tests are exactly the tool for that. + +`uv` supports the second option with [`override-dependencies`](https://docs.astral.sh/uv/concepts/resolution/#dependency-overrides), which replaces a declared constraint during resolution: + +```toml +[tool.uv] +# MLflow 3.15 still declares `cryptography<50`, but the fix for PYSEC-2026-3552 +# (PKCS#7 Bleichenbacher oracle) only ships in 50.0.0. Overriding the stale upper +# bound installs the patched library; the test suite is what proves MLflow still +# works with it. Drop this override once MLflow relaxes the constraint upstream. +override-dependencies = ["cryptography>=50"] +``` + +Three habits make an override safe rather than reckless: + +1. **Document why it exists**, including the advisory identifier, directly above the entry. +1. **Prove it with tests**: an override is only defensible if your suite exercises the code paths that use the overridden package. +1. **Write down its exit condition**: state what has to happen upstream for the override to be deleted, and remove it when that happens. An undated override quietly becomes a permanent fork of someone else's dependency graph. + ## How do you run commands in the `uv` managed environment? `uv` provides the `uv run` command to execute scripts within the context of your project's virtual environment, so you don't need to manually activate it (e.g., `source .venv/bin/activate`). diff --git a/docs/3. Productionizing/3.0. Package.md b/docs/3. Productionizing/3.0. Package.md index adf5f13..e26b53a 100644 --- a/docs/3. Productionizing/3.0. Package.md +++ b/docs/3. Productionizing/3.0. Package.md @@ -56,13 +56,14 @@ Here is an example with explanations for each section: # Core project metadata used by PyPI and installation tools. [project] name = "bikes" -version = "3.0.0" +version = "6.0.0" description = "Predict the number of bikes available." authors = [{ name = "Médéric HURIER", email = "github@fmind.dev" }] readme = "README.md" requires-python = ">=3.14" dependencies = [] # List your production dependencies here -license = { file = "LICENSE.txt" } +license = "MIT" # an SPDX expression, per PEP 639 +license-files = ["LICENSE.txt"] keywords = ["mlops", "python", "package"] # URLs that appear on your package's PyPI page. @@ -77,16 +78,21 @@ Changelog = "https://github.com/fmind/bikes/blob/main/CHANGELOG.md" [project.scripts] bikes = 'bikes.scripts:main' -# Configures uv to install optional dependency groups by default during development. -[tool.uv] -default-groups = ["checks", "commits", "dev", "docs", "notebooks"] +# Development-only dependencies, kept out of the production install. +[dependency-groups] +dev = ["pytest>=9.1.1", "ruff>=0.16.2", "ty>=0.0.69,<0.1"] +notebook = ["ipykernel>=6.29.5", "nbformat>=5.10.4"] # Specifies the build backend (uv_build, in this case) to create the package. [build-system] -requires = ["uv_build>=0.9.0"] +# Keep the upper bound at least one minor ahead of the pinned `uv` tool: without it +# `uv build` warns, and a future breaking `uv_build` would silently break the sdist. +requires = ["uv_build>=0.9,<0.13"] build-backend = "uv_build" ``` +The upper bound on `uv_build` is not decoration. A pre-1.0 project makes no compatibility promise across minor releases, so an unbounded `uv_build>=0.9.0` lets a future `0.13` change how your source tree is packaged behind your back; `uv build` warns about exactly this. Bound it, and raise the ceiling deliberately once you have built and tested against the new backend. The same reasoning explains the `ty>=0.0.69,<0.1` range above. + ## Where should you structure the source code for your package? Always place your package's source code inside a `src` directory. This is known as the [**`src` layout**](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) and is a best practice for several reasons: diff --git a/docs/4. Validating/4.0. Typing.md b/docs/4. Validating/4.0. Typing.md index 554e12b..f52e3ff 100644 --- a/docs/4. Validating/4.0. Typing.md +++ b/docs/4. Validating/4.0. Typing.md @@ -133,16 +133,27 @@ class GridCVSearcher(pdt.BaseModel): The recommended tool for static type checking is [ty](https://docs.astral.sh/ty/), a new type checker from [Astral](https://astral.sh/) (the makers of `uv` and `Ruff`). Like its siblings, `ty` is written in Rust and is exceptionally fast, so it can run on every save and on every commit without slowing you down. It can be run from the command line or integrated directly into your [editor](https://docs.astral.sh/ty/editors/) through its language server. ```bash -# Install ty into your "check" dependency group -uv add --group check ty +# Install ty into your "dev" dependency group +uv add --group dev ty # Type-check your source and test code uv run ty check ``` -By default, `ty check` checks the whole project; you can also pass explicit paths (e.g. `uv run ty check src/ tests/`). +By default, `ty check` checks the whole project; you can also pass explicit paths (e.g. `uv run ty check src/ tests/`). The MLOps Python Package wires this command to `mise run check:types`. -**Note**: `ty` is still **pre-1.0** and moving fast. Pin it to a compatible range (e.g. `ty>=0.0.56,<0.1`) rather than an open-ended version, and treat it as a local and CI check that you upgrade deliberately, since new releases can add or refine diagnostics. +**Note**: `ty` is still **pre-1.0** and moving fast, so both reference repositories declare it as a bounded range rather than a simple floor: + +```toml +[dependency-groups] +dev = [ + "ty>=0.0.69,<0.1", # pre-1.0: pin a compatible range until it stabilizes +] +``` + +The reason is the versioning contract. Under [semantic versioning](https://semver.org/), a pre-1.0 project makes no compatibility promise at all: the `0.0.x` segment is free to add diagnostics, rename rules, or change what counts as an error. A plain `ty>=0.0.69` therefore lets `uv sync` pick up a release that reports brand-new errors on code you never touched, and your build turns red for a reason unrelated to your commit. The `<0.1` upper bound keeps that upgrade a deliberate act: you raise the floor, run `uv run ty check`, and fix or silence the new diagnostics in a dedicated change. + +Apply the same reasoning to any pre-1.0 tool you put in a gate. Once a project reaches 1.0 and promises backward compatibility within a major, a simple floor (`>=`) is enough again. Several other type checkers exist, and you will encounter them in older projects: @@ -167,9 +178,12 @@ python-version = "3.14" # ty (pre-1.0) does not model every dynamic ML library yet; relax the # categories they trigger so ty still gates real errors. [tool.ty.rules] -invalid-argument-type = "ignore" # dynamic ML/MLflow call sites -invalid-type-form = "ignore" # pandera typed Series / dynamic type forms -unresolved-attribute = "ignore" # dynamic attributes on MLflow/pandera objects +invalid-argument-type = "ignore" # dynamic ML/MLflow call sites +invalid-frozen-dataclass-subclass = "ignore" # ty reads frozen Pydantic models as frozen dataclasses +invalid-type-form = "ignore" # pandera typed Series / dynamic type forms +not-subscriptable = "ignore" # optional dynamic MLflow attributes +possibly-missing-submodule = "ignore" # lazily-loaded mlflow.* submodules +unresolved-attribute = "ignore" # dynamic attributes on MLflow/pandera objects ``` If you need to bypass a diagnostic for a specific line, add a `# ty: ignore` comment with the rule name: diff --git a/docs/4. Validating/4.1. Linting.md b/docs/4. Validating/4.1. Linting.md index 9bba290..f3b1572 100644 --- a/docs/4. Validating/4.1. Linting.md +++ b/docs/4. Validating/4.1. Linting.md @@ -30,8 +30,8 @@ Key advantages of Ruff include: - **VS Code Extension**: The official [Ruff VS Code extension](https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff) integrates these features directly into your editor. ```bash -# Install Ruff into your "check" dependency group -uv add --group check ruff +# Install Ruff into your "dev" dependency group +uv add --group dev ruff # Run Ruff to lint your codebase uv run ruff check src/ tests/ @@ -43,22 +43,43 @@ To keep your repository clean, remember to add the `.ruff_cache/` directory to y Linter configurations are typically placed in the `pyproject.toml` file. This allows you to define project-wide rules, customize behavior, and ensure every developer uses the same settings. -Here is a sample configuration for Ruff: +Here is the configuration used by the MLOps Python Package, abridged to show its shape: ```toml [tool.ruff] -# automatic fix when possible -fix = true -# define the default indent width -indent-width = 4 # define the default line length -line-length = 100 +line-length = 120 # define the default python version target-version = "py314" +[tool.ruff.lint] +# an explicit, reviewed selection instead of Ruff's evolving defaults +select = [ + "B", # flake8-bugbear + "D", # pydocstyle (documented public API) + "E", # pycodestyle errors + "F", # pyflakes + "I", # isort (import sorting) + "N", # pep8-naming + "PTH", # flake8-use-pathlib + "RUF", # ruff-specific rules + "S", # flake8-bandit (security; replaces bandit) + "SIM", # flake8-simplify + "T20", # flake8-print + "UP", # pyupgrade + "W", # pycodestyle warnings +] +ignore = [ + "E501", # line length handled by the formatter +] + +[tool.ruff.lint.pydocstyle] +# set the expected docstring style +convention = "google" + [tool.ruff.lint.per-file-ignores] -# exceptions for docstrings in tests -"tests/*.py" = ["D100", "D103"] +# exceptions for docstrings and asserts in tests +"tests/**" = ["D100", "D103", "S101"] ``` If you need to ignore a specific rule for a single line, you can use an inline `noqa` (no quality assurance) comment: @@ -68,6 +89,39 @@ If you need to ignore a specific rule for a single line, you can use an inline ` from project.module import specific_import # noqa: F401 ``` +## Why should you write an explicit rule selection? + +Ruff ships with a default rule set, and that default set is not stable across releases. [Ruff 0.16.0](https://github.com/astral-sh/ruff/releases) (released 2026-07-23) expanded it from **59 rules to 413**, pulling in whole families that had previously been opt-in. You can verify the current number yourself: + +```bash +# Print the settings Ruff would use with no configuration at all +uv run ruff check --show-settings --isolated . +``` + +The consequence depends entirely on how your project is configured: + +- **With an explicit `select` list** (the reference repositories), nothing changed on upgrade. You opted into thirty-one named families, Ruff enforces exactly those, and a release that broadens the defaults is invisible to you. New rules arrive when you decide to add a family, not when a dependency resolves. +- **Without a `select` list**, the upgrade silently multiplied your enforced rules by seven. A `uv sync` that pulled Ruff 0.16 could turn a green repository red with hundreds of violations in code nobody had touched, and the diff that "caused" it would be a lockfile bump. + +This is the practical argument for writing `select` explicitly, even if your initial selection matches today's defaults: the list becomes a reviewed decision recorded in your repository, and upgrading the linter stops being a source of surprise failures. + +## Why does your Ruff version floor matter? + +Linting and formatting must agree across every machine that runs them: your editor, your teammates' checkouts, your git hooks, and CI. Ruff 0.16 changed formatter behavior too (it now formats Python inside Markdown by default, see [4.5. Formatting](./4.5.%20Formatting.md)), which means an **older Ruff disagrees with a repository formatted by 0.16**. A contributor whose environment resolved 0.15 would see `ruff format --check` fail on files they never opened. + +The fix is to raise the dependency floor whenever you adopt a behavior change, and to say why in a comment: + +```toml +[dependency-groups] +dev = [ + # Ruff 0.16 formats Python inside Markdown and rewrote the default rule set: + # an older Ruff would disagree with this repository's formatting, so floor it. + "ruff>=0.16.2", +] +``` + +Pair that floor with a committed `uv.lock` (see [1.3. uv (project)](../1.%20Initializing/1.3.%20uv%20(project).md)) so everyone resolves the same version, and with `uv lock --check` in your `check:format` task so a stale lockfile fails the gate. + ## How does linting differ from formatting? While often used together, linting and formatting have distinct purposes: diff --git a/docs/4. Validating/4.2. Testing.md b/docs/4. Validating/4.2. Testing.md index ea29f94..9eaa523 100644 --- a/docs/4. Validating/4.2. Testing.md +++ b/docs/4. Validating/4.2. Testing.md @@ -46,8 +46,8 @@ def test_answer(): To execute `pytest` across your project: ```bash -# Install pytest (one-time) -uv add --dev pytest +# Install pytest into your "dev" dependency group +uv add --group dev pytest # Run pytest on the `tests` directory uv run pytest tests/ @@ -62,6 +62,7 @@ You can extend `pytest`'s functionality with plugins: uv run pytest --cov=src/ tests/ ``` +- **[`pytest-mock`](https://pypi.org/project/pytest-mock/)**: Exposes a `mocker` fixture that patches objects for the duration of one test and restores them afterwards. - **[`pytest-xdist`](https://pypi.org/project/pytest-xdist/)**: Executes tests in parallel, significantly reducing runtime by utilizing all available CPU cores. ```bash @@ -69,6 +70,65 @@ uv run pytest --cov=src/ tests/ uv run pytest -n auto tests/ ``` +## How do you enforce a minimum coverage? + +Producing a coverage report is not the same as enforcing one. A report is a number a human may or may not read; a **gate** is a threshold that fails the build. `pytest-cov` provides the gate with `--cov-fail-under=`, and both reference repositories ship one, at deliberately different values: + +```toml +# MLOps Python Package — the suite genuinely covers every line and branch +[tool.pytest.ini_options] +addopts = [ + "--cov=src", + "--cov-report=term-missing", + # The suite actually covers every line and branch, so the gate says so: a drop is a + # regression to fix, not a threshold to lower. + "--cov-fail-under=100", +] +``` + +```toml +# Cookiecutter MLOps Package — a scaffold must not fail on the owner's first commit +[tool.pytest.ini_options] +addopts = [ + "--cov=src", + "--cov-report=term-missing", + "--cov-fail-under=80", +] +``` + +That difference is the whole lesson about choosing a threshold: + +- **Set it to what your suite achieves today, not to an aspiration.** A gate above your real number fails immediately and gets removed; a gate far below it protects nothing. Measure first, then write the number down. +- **A generated project starts at 80.** A freshly scaffolded package contains code the owner has not written tests for yet. Demanding 100 from a template means every new project is born red. +- **A mature package can afford 100.** Once a suite really covers every line and branch, the gate turns "coverage dropped" into a build failure instead of a slow, unnoticed erosion. The rule that makes it work: when the gate fails, you add the missing test, you do not lower the number. + +Ratchet the value up as your suite improves, and treat lowering it as a change that needs a justification in the commit message. + +## Can you run tests in parallel and measure coverage at the same time? + +Not always, and the reference package is honest about it. It declares `pytest-xdist` and exposes parallel execution as a **separate task**, not as part of the gate: + +```toml +# mise.toml +[tasks.test] +alias = "t" +description = "Run the test suite with coverage (pytest)" +run = "uv run pytest" + +[tasks."test:parallel"] +description = "Run the test suite across CPU cores, without the coverage gate (pytest-xdist)" +# Fast local feedback only. pytest-cov and pytest-xdist deadlock on this suite, so the +# coverage gate stays on the serial `test` task that hooks and CI run. +run = "uv run pytest -n auto --no-cov" +``` + +`pytest-cov` and `pytest-xdist` combine badly on this suite: the two plugins deadlock, so `mise run test:parallel` explicitly disables coverage with `--no-cov`. The split that results is a good pattern in general: + +- **`mise run test`** is the authority. It runs serially, with coverage and the `--cov-fail-under` gate. This is what the `pre-push` git hook runs and what CI runs, so the number that blocks a merge is always measured the same way. +- **`mise run test:parallel`** is the fast local loop. You use it while iterating, when you want the answer "did anything break" in a fraction of the time and do not care about coverage yet. + +Do not try to make the fast task the authoritative one. A gate you cannot reproduce deterministically is worse than no gate. + ## How should you configure your project for testing? First, prevent `pytest` cache files from being committed to Git by adding `.pytest_cache/` to your `.gitignore` file. @@ -90,16 +150,30 @@ Finally, define global `pytest` configurations in your `pyproject.toml` file to ```toml [tool.pytest.ini_options] -# Run tests with high verbosity and set the source path -addopts = "--verbosity=2 --cov=src" +addopts = [ + "-ra", # summarize every non-passing outcome at the end of the run + "--strict-config", # a typo in this very section is an error, not a silent no-op + "--strict-markers", # an unregistered @pytest.mark is an error, not a silent skip + "--cov=src", # measure coverage of the source tree + "--cov-report=term-missing", # list the uncovered lines in the terminal + "--cov-fail-under=100", # fail the run below the threshold (see the section above) +] # Add the `src` directory to the Python path for imports pythonpath = ["src"] +# Only collect from the tests directory +testpaths = ["tests"] +# An `xfail` test that unexpectedly passes is a failure, not a pass +xfail_strict = true [tool.coverage.run] # Measure branch coverage to check if `if/else` statements are tested branch = true source = ["src"] -omit = ["**/__main__.py"] # Exclude non-testable files +omit = ["__main__.py"] # Exclude non-testable files + +[tool.coverage.report] +show_missing = true +skip_covered = true # keep the report focused on what still needs tests ``` ## How should you structure your tests? @@ -158,6 +232,52 @@ def tmp_outputs_path(tmp_path: str) -> str: The `scope` parameter controls the fixture's lifecycle. A `session`-scoped fixture is created once for the entire test run, while a `function`-scoped fixture is recreated for every test. +## How do you keep expensive setup from dominating the run? + +Choosing a scope is usually a trade-off between **isolation** (a fresh object per test) and **speed** (one object for the whole session). Sometimes you can have both, by paying the expensive part once and cheaply cloning it. + +The MLOps Python Package hit exactly this case when its MLflow tracking store moved from a directory of files to a SQLite database (see [5.5. AI/ML Experiments](../5.%20Refining/5.5.%20AI-ML%20Experiments.md)). Creating an MLflow SQLite store runs its Alembic schema migrations, which costs several seconds. Doing that per test took the suite from 34 seconds to 339 seconds. + +The fix keeps per-test isolation and pays the migration once: a `session`-scoped fixture builds one migrated, empty database, and a `function`-scoped fixture copies that file for each test. + +```python +@pytest.fixture(scope="session") +def mlflow_db_template(tmp_path_factory: pytest.TempPathFactory) -> str: + """Return a migrated but empty MLflow database used as a template by every test. + + Creating an MLflow SQLite store runs its Alembic migrations, which costs seconds. + Paying that once per session and copying the file per test keeps the isolation of a + fresh database at the cost of a file copy. + """ + path = tmp_path_factory.mktemp("mlflow") / "template.db" + services.MlflowService( + tracking_uri=f"sqlite:///{path}", + registry_uri=f"sqlite:///{path}", + experiment_name="Experiment-Template", + registry_name="Registry-Template", + ).start() + return str(path) + + +@pytest.fixture(scope="function", autouse=True) +def mlflow_service(tmp_path: str, mlflow_db_template: str) -> T.Generator[services.MlflowService]: + """Return and start the mlflow service.""" + # Each test gets its own SQLite file under tmp_path, so runs stay isolated. + database = os.path.join(tmp_path, "mlflow.db") + shutil.copyfile(mlflow_db_template, database) + service = services.MlflowService( + tracking_uri=f"sqlite:///{database}", + registry_uri=f"sqlite:///{database}", + experiment_name="Experiment-Testing", + registry_name="Registry-Testing", + ) + service.start() + yield service + service.stop() +``` + +Copying the file costs about 0.07 seconds instead of seconds of migrations, bringing the suite back to 63 seconds with the same isolation guarantee. The generalizable rule: when setup is expensive but its *result* is a cheap, copyable value (a database file, a fitted model, a rendered dataset), build it once at session scope and hand out copies at function scope. + ## How can you avoid repetitive test scenarios? To test a function against multiple input scenarios without writing duplicate code, use the **[`@pytest.mark.parametrize`](https://docs.pytest.org/en/latest/how-to/parametrize.html)** decorator. This feature allows you to run the same test function with different argument sets. @@ -244,8 +364,8 @@ def test_data_split_is_reproducible(): 2. **Keep Tests Independent and Isolated**: Each test should be able to run on its own, without depending on the state left by other tests. 3. **Use Fixtures for Setup and Teardown**: Leverage fixtures to manage setup and cleanup logic, keeping tests clean and focused. 4. **Test for Edge Cases**: Go beyond the "happy path." Test for invalid inputs, empty data, and other edge conditions that could cause failures. -5. **Aim for High Test Coverage**: Strive for at least **80% code coverage**. This ensures most of your codebase is validated and encourages a culture of quality. -6. **Keep Tests Fast**: Slow tests slow down development. Optimize test performance to ensure the suite can be run quickly and frequently. +5. **Enforce Coverage, Don't Just Measure It**: Start at **80%** with `--cov-fail-under=80`, then ratchet the threshold up as the suite improves. A number nobody enforces erodes silently. +6. **Keep Tests Fast**: Slow tests slow down development. Move expensive, reusable setup to a `session`-scoped fixture, and keep a fast parallel task (`mise run test:parallel`) for the local loop. 7. **Run Tests Automatically**: Integrate your test suite into a CI/CD workflow to catch issues early and often. 8. **Review and Refactor Tests**: Just like production code, tests should be reviewed and updated as the codebase evolves. diff --git a/docs/4. Validating/4.4. Security.md b/docs/4. Validating/4.4. Security.md index 9cc50d0..6bcc5ad 100644 --- a/docs/4. Validating/4.4. Security.md +++ b/docs/4. Validating/4.4. Security.md @@ -1,5 +1,5 @@ --- -description: Understand the main security risks in Python and MLOps, including vulnerable dependencies, input validation, and configuration management. Learn how to mitigate these risks with tools like Ruff's security rules, pip-audit, Trivy, and GitHub Dependabot to ensure the security of your applications. +description: Understand the main security risks in Python and MLOps, including vulnerable dependencies, leaked secrets, and risky configuration. Learn how to mitigate them with Ruff's security rules, pip-audit, gitleaks, Trivy, actionlint, zizmor, hadolint, and GitHub Dependabot. --- # 4.4. Security @@ -55,35 +55,250 @@ Using automated tools to scan for security problems is a highly effective strate uv add --group dev pip-audit # Audit the project's dependencies (mise run check:vuln) - uv run pip-audit + uv run pip-audit --skip-editable --cache-dir .cache/pip-audit ``` -- **Leaked secrets**: [`gitleaks`](https://github.com/gitleaks/gitleaks) scans your code and git history for accidentally committed credentials (`mise run check:leaks`). + Both flags matter. `--skip-editable` excludes your own project, which is installed in editable mode by `uv sync` and has no advisory database entry to look up. `--cache-dir .cache/pip-audit` moves the advisory cache from the user's home directory into the repository, so a CI runner caches it with the rest of the checkout and every developer sees the same behavior; add `.cache/` to your `.gitignore`. -- **Misconfigurations**: [`trivy`](https://trivy.dev/) scans configuration and infrastructure files (Dockerfiles, workflows, manifests) for insecure settings with `trivy config .` (`mise run check:scan`). +- **Leaked secrets**: [`gitleaks`](https://github.com/gitleaks/gitleaks) scans your working tree and git history for accidentally committed credentials (`mise run check:leaks`). + +- **Misconfigurations, secrets, licenses, and vulnerabilities in the checkout**: [`trivy`](https://trivy.dev/) scans the repository against a policy you commit alongside it (`mise run check:scan`). Together these give you defense in depth: Ruff `S` rules catch insecure code patterns, `pip-audit` catches vulnerable dependencies, `gitleaks` catches exposed secrets, and `trivy` catches risky configuration. +## How should you invoke Trivy? + +This is a place where an obvious-looking command quietly does far less than you think. The command the reference repositories run is: + +```bash +# mise run check:scan +trivy --config trivy.yaml fs . +``` + +Two details in that line each fix a real bug. + +### Why `--config trivy.yaml` is explicit + +Trivy discovers its configuration through a precedence chain, and an environment variable sits above the file in your repository. If a developer (or a base image, or a shell profile) exports `TRIVY_CONFIG` pointing at another policy, that policy **silently wins** over the `trivy.yaml` you committed. The scan still succeeds, still prints a clean result, and never applies a single one of your rules. + +Passing `--config trivy.yaml` on the command line puts your repository's policy at the top of the chain. The scan now enforces what the repository says it enforces, on every machine, regardless of the surrounding environment. + +### Why `fs` and not `config` + +`trivy config .` reads naturally as "scan my configuration", and that is exactly the trap. The `config` subcommand enables **only the misconfiguration scanner**. It does not honor the `scan.scanners` list in your configuration file — it has already decided what to run. + +So a repository whose `trivy.yaml` declares four scanners, invoked as `trivy config .`, actually runs one of them. Three quarters of the policy never executed, and nothing in the output said so. + +`trivy fs .` scans the filesystem with the scanners the configuration file declares. Same target, same policy file, four scanners instead of one. + +### What the policy file contains + +```yaml +# trivy.yaml +severity: + - HIGH + - CRITICAL +scan: + scanners: + - license + - misconfig + - secret + - vuln + # Caches and virtualenvs are not source. `.cache` in particular is written by + # `check:vuln` (pip-audit) while this scan runs in parallel, and trivy aborts when a + # temporary file disappears mid-walk. + skip-dirs: + - .cache + - .venv + - .git +vulnerability: + ignore-unfixed: true +``` + +Three choices are worth explaining: + +- **`severity: [HIGH, CRITICAL]`** keeps the gate actionable. A scanner that reports every LOW finding trains people to ignore it. +- **`ignore-unfixed: true`** hides vulnerabilities with no released fix. You cannot act on them today, and a permanently red build is a build nobody reads. Revisit this if you ship to a regulated environment that requires tracking them. +- **`skip-dirs`** is not cosmetic. `mise run check` runs its subtasks in parallel, so `check:vuln` (pip-audit) is writing into `.cache/pip-audit` at the same moment `check:scan` walks the tree. Trivy aborts with a fatal error when a file it has listed disappears mid-walk, which makes the whole gate fail intermittently for a reason that has nothing to do with your code. Excluding caches, the virtualenv, and `.git` removes the race and cuts the scan time, because none of those directories are your source anyway. + +## How do you lint your CI/CD workflows and container image? + +Two more scanners belong in `mise run check`, because workflow files and Dockerfiles are code with production consequences: + +```toml +# mise.toml +[tasks."check:actions"] +description = "Lint and audit GitHub Actions workflows (actionlint + zizmor)" +run = ["actionlint", "zizmor --offline .github/workflows/"] + +[tasks."check:dockerfile"] +description = "Lint the container image definition (hadolint)" +run = "hadolint Dockerfile" +``` + +[`actionlint`](https://github.com/rhysd/actionlint) catches syntax and expression errors in workflow files; [`zizmor`](https://docs.zizmor.sh/) audits them for security weaknesses such as script injection through untrusted `github.event` values or over-broad `permissions`. [5.3. CI/CD Workflows](../5.%20Refining/5.3.%20CI-CD%20Workflows.md) covers what these two find and how the workflows are written to satisfy them. + +One zizmor default deserves a note here, because it is a genuine policy disagreement rather than a bug. By default zizmor requires actions to be pinned to a commit hash. The reference repositories pin to major-version tags instead, so that security patches within a major arrive automatically. Relaxing the rule records that decision explicitly rather than leaving the audit permanently red: + +```yaml +# .github/zizmor.yml +# Actions are deliberately pinned to major-version tags: tags track security patches +# within a major, at the cost of trusting the tag. zizmor's default policy demands +# hash-pins; relax it to ref-pins so the audit enforces the actual policy. +rules: + unpinned-uses: + config: + policies: + "*": ref-pin +``` + +[`hadolint`](https://github.com/hadolint/hadolint) lints the Dockerfile. It is the reason the reference image declares a **numeric** user instead of a name: rule `DL3066` flags a non-numeric user id, because a host cannot resolve a username that only exists inside the image's `/etc/passwd`. + +```dockerfile +# Fixed numeric uid/gid: stable file ownership across rebuilds and bind mounts, and +# resolvable by a host that does not share this image's /etc/passwd. +RUN groupadd -r -g 10001 app && useradd -r -u 10001 -g app -m app +USER 10001:10001 +COPY --from=build --chown=10001:10001 /app/.venv /app/.venv +``` + +## Where should secret scanning run? + +Once, in one place, is not enough. A committed secret has three distinct lifetimes, and each needs a different scan: + +1. **Before it exists in history** — the `pre-commit` hook runs `mise run check:leaks --staged`, which scans only the staged diff. This is the scan that actually saves you: a secret caught here never enters a commit object, so there is nothing to rotate and no history to rewrite. +1. **In the recent history** — the `check:leaks` task itself is deliberately bounded so it stays fast enough to run on every commit and every CI job: + + ```toml + # mise.toml + [tasks."check:leaks"] + description = "Audit codebase for leaked secrets (gitleaks)" + run = 'gitleaks git --log-opts="--max-count=100" --verbose' + ``` + +1. **Anywhere in the full history** — a secret that was committed and later deleted is invisible to a bounded scan forever after. That case needs a scheduled, full-depth job: + + ```yaml + # .github/workflows/security.yml + on: + schedule: + - cron: "17 3 * * 1" + workflow_dispatch: + jobs: + scan: + runs-on: ubuntu-latest + steps: + - name: Checkout complete history + uses: actions/checkout@v7 + with: + fetch-depth: 0 + persist-credentials: false # scanners only read the checkout + - name: Install mise system + uses: jdx/mise-action@v4 + - name: Scan complete Git history + run: gitleaks git --redact=100 --verbose + - name: Scan full checkout + run: trivy fs . + ``` + + `fetch-depth: 0` is the whole point: the default shallow checkout cannot see the commit where the secret was introduced. `--redact=100` keeps the secret itself out of the public workflow logs. + ## How can GitHub help manage security risks? GitHub provides powerful, integrated tools to automate security monitoring. [Dependabot](https://docs.github.com/en/code-security/dependabot) is a key feature that automatically scans your project's dependencies for known vulnerabilities and opens pull requests to update them to secure versions. -To enable Dependabot, create a configuration file at `.github/dependabot.yml` in your repository. This file tells Dependabot what package ecosystems to monitor and how often to check for updates. +To enable Dependabot, create a configuration file at `.github/dependabot.yml`. This file tells Dependabot which package ecosystems to monitor, how often to check, how to group updates, and how to word its commits. ```yaml # .github/dependabot.yml -# For more options, see: https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file - +# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference +# Minor and patch updates arrive grouped, one pull request per ecosystem; majors stay +# separate so a breaking bump is always reviewed on its own. version: 2 updates: - - package-ecosystem: "uv" # Monitor uv-managed Python dependencies - directory: "/" # Check for dependencies in the root directory + - package-ecosystem: "uv" + directory: "/" schedule: - interval: "weekly" # Scan for vulnerabilities weekly + interval: "weekly" + day: "monday" + commit-message: + prefix: "chore(deps)" + groups: + python: + patterns: ["*"] + update-types: ["minor", "patch"] + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + commit-message: + prefix: "chore(deps)" + groups: + actions: + patterns: ["*"] + update-types: ["minor", "patch"] + - package-ecosystem: "docker" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + commit-message: + prefix: "chore(deps)" + groups: + docker: + patterns: ["*"] + update-types: ["minor", "patch"] ``` -By combining automated tools like Ruff's `S` rules, `pip-audit`, `gitleaks`, and `trivy` with GitHub's native security features, you can build a robust defense against common security threats. +Four choices here are worth the extra lines: + +- **Three ecosystems, not one.** `uv` covers your Python dependencies, `github-actions` covers the actions your workflows call, and `docker` covers base images in your `Dockerfile`. A project that monitors only Python is still running last year's `actions/checkout` and last year's base image. +- **`day: "monday"`** turns dependency review into a predictable Monday-morning task instead of an unscheduled interruption. +- **`groups`** collapses every minor and patch bump in one ecosystem into a single pull request. Because `update-types` lists only `minor` and `patch`, major bumps stay on their own pull request, where a breaking change gets reviewed in isolation. +- **`commit-message.prefix: "chore(deps)"`** aligns Dependabot with your changelog configuration, and this one bit us. The `cliff.toml` used to generate the changelog (see [6.3. Releases](../6.%20Sharing/6.3.%20Releases.md)) already excluded dependency noise: + + ```toml + # cliff.toml, inside [git] commit_parsers + { message = "^chore\\(deps\\)", skip = true }, + ``` + + But Dependabot's *default* prefix is `build`, so its commits arrived as `build(deps): ...` (or `build(deps-dev): ...` for development dependencies). The filter never matched anything, and ten dependency bumps ended up in a published changelog written for humans. Nothing was broken enough to notice; the two configuration files simply disagreed about a string. Setting the prefix explicitly makes the filter do the job it was written for. + +## When may you override an upstream version constraint? + +Sooner or later `pip-audit` reports a vulnerability whose fix you cannot install, because one of your dependencies declares an upper bound that excludes the patched release. You then have two options that look superficially similar and are fundamentally opposite. + +**Suppressing the scanner** means telling `pip-audit` to ignore the advisory (or lowering `severity` in `trivy.yaml` until the finding disappears). The vulnerable code is still installed and still running. You have removed the warning, not the risk. + +**Overriding the constraint** means installing the patched library anyway, against the stale upper bound, and then proving with your test suite that the dependency still works. The vulnerable code is gone. + +The MLOps Python Package needed the second one: + +```toml +[tool.uv] +# MLflow 3.15 still declares `cryptography<50`, but the fix for PYSEC-2026-3552 +# (PKCS#7 Bleichenbacher oracle) only ships in 50.0.0. Overriding the stale upper +# bound installs the patched library; the test suite is what proves MLflow still +# works with it. Drop this override once MLflow relaxes the constraint upstream. +override-dependencies = ["cryptography>=50"] +``` + +Remove that override and `uv` re-locks `cryptography` to 49.0.0, and `mise run check:vuln` fails on the advisory again. The override is what keeps the gate green *and* the installation safe. + +An override is legitimate only when all four of these hold: + +1. **The upper bound is stale, not protective.** MLflow does not depend on anything `cryptography` 50 removed; the bound was written before 50 existed. +1. **Your test suite exercises the affected dependency.** This is the load-bearing condition. The tests are the evidence that overriding the bound did not break MLflow, and without them you are simply guessing. +1. **The reason is written down where the override lives.** The comment names the advisory, the version that fixes it, and the condition for removal. +1. **The override has an exit.** "Drop this once MLflow relaxes the constraint upstream" turns a permanent hack into a temporary one with a defined end. + +If you cannot satisfy those, do not reach for a suppression instead — the honest outcome is to record the risk, pin to the last safe version, or replace the dependency. + +By combining automated tools like Ruff's `S` rules, `pip-audit`, `gitleaks`, `trivy`, `actionlint`, `zizmor`, and `hadolint` with GitHub's native security features, you can build a robust defense against common security threats. ## Additional Resources - **[Security configuration from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/pyproject.toml)** +- **[Trivy policy from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/trivy.yaml)** +- **[Dependabot configuration from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/.github/dependabot.yml)** diff --git a/docs/4. Validating/4.5. Formatting.md b/docs/4. Validating/4.5. Formatting.md index d2e861e..b871a43 100644 --- a/docs/4. Validating/4.5. Formatting.md +++ b/docs/4. Validating/4.5. Formatting.md @@ -40,8 +40,8 @@ While [`black`](https://black.readthedocs.io/en/stable/) (a code formatter) and You can install and run Ruff to format your entire codebase with these commands: ```bash -# Install Ruff into your project's development dependencies -uv add --group check ruff +# Install Ruff into your "dev" dependency group +uv add --group dev ruff # Sort and organize all import statements uv run ruff check --select I --fix src/ tests/ @@ -50,9 +50,49 @@ uv run ruff check --select I --fix src/ tests/ uv run ruff format src/ tests/ ``` +The MLOps Python Package wires exactly these two steps to a single `mise run format:python` task, adding `--force-exclude` so the formatter honors its exclusions even when a git hook hands it explicit file paths: + +```bash +uv run ruff check --select=I --fix --force-exclude . && uv run ruff format --force-exclude . +``` + +## What does Ruff 0.16 format that earlier versions did not? + +Since [Ruff 0.16.0](https://github.com/astral-sh/ruff/releases) (released 2026-07-23), `ruff format` also formats **Python code blocks inside Markdown files** by default. A fenced block tagged `python` in your `README.md` or your documentation is now reformatted like any `.py` file: + +````markdown +```python +x = 1 +``` +```` + +becomes: + +````markdown +```python +x = 1 +``` +```` + +Two practical consequences follow: + +1. **Your documentation examples stay correct.** Code snippets in READMEs drift stylistically from the code they illustrate, because nothing used to check them. Now they are held to the same standard as the source. +1. **Your Ruff version floor has to move with it.** A repository formatted by 0.16 fails `ruff format --check` under 0.15, because the older binary leaves those Markdown blocks alone and reports the newer output as unformatted. Declare the floor explicitly, as the reference package does, and explain it in the same place: + + ```toml + [dependency-groups] + dev = [ + # Ruff 0.16 formats Python inside Markdown and rewrote the default rule set: + # an older Ruff would disagree with this repository's formatting, so floor it. + "ruff>=0.16.2", + ] + ``` + +The same release rewrote Ruff's default *lint* rule set, from 59 rules to 413. That change is covered in [4.1. Linting](./4.1.%20Linting.md). + ## How can you format configuration and documentation files? -Ruff only formats Python. A real project also contains JSON, Markdown, TOML, and YAML files (`pyproject.toml`, `mise.toml`, GitHub workflows, this very documentation), and those deserve the same consistency. [`dprint`](https://dprint.dev/) is a fast, pluggable formatter that handles exactly these config and markup formats, complementing Ruff on the Python side. +Ruff only formats Python (including, now, the Python it finds inside Markdown). A real project also contains JSON, Markdown, TOML, and YAML files (`pyproject.toml`, `mise.toml`, GitHub workflows, this very documentation), and those deserve the same consistency. [`dprint`](https://dprint.dev/) is a fast, pluggable formatter that handles exactly these config and markup formats, complementing Ruff on the Python side. ```bash # Install dprint (for example with mise, or see https://dprint.dev/install/) @@ -71,19 +111,43 @@ You enable formats and tune behavior through a `dprint.jsonc` file at the projec { "$schema": "https://dprint.dev/schemas/v0.json", "lineWidth": 120, - "excludes": ["**/*-lock.json", "**/node_modules", ".git", ".venv"], + "excludes": [ + "**/*-lock.json", + "**/node_modules", + "**/references/**", + ".git", + ".venv" + ], "markdown": { "textWrap": "never" }, "plugins": [ "https://plugins.dprint.dev/g-plane/pretty_yaml-v0.6.0.wasm", - "https://plugins.dprint.dev/json-0.22.0.wasm", + "https://plugins.dprint.dev/json-0.23.0.wasm", "https://plugins.dprint.dev/markdown-0.22.1.wasm", "https://plugins.dprint.dev/toml-0.7.0.wasm" ] } ``` +## Do Ruff and dprint fight over Markdown? + +No, and the reason is worth understanding, because both tools now touch `.md` files. + +dprint's Markdown plugin formats the *document*: headings, list markers, tables, emphasis. It can also format the contents of a fenced code block, but only when a plugin for that block's language is loaded. The configuration above declares four plugins — YAML, JSON, Markdown, TOML — and **no Python plugin exists in that list**, so dprint reads a `python`-tagged fence, finds no formatter for it, and copies the block through untouched. + +Ruff does the mirror image: it formats the Python inside those fences and ignores the prose around them. + +The two therefore partition the file cleanly, with no rule in common and no last-writer-wins race. That is why `mise run format` can run them in sequence without either undoing the other: + +```bash +# format:python — Ruff sorts imports, then formats .py files and Python-in-Markdown +uv run ruff check --select=I --fix --force-exclude . && uv run ruff format --force-exclude . + +# format:dprint — dprint formats JSON, Markdown prose, TOML, YAML +dprint fmt +``` + With Ruff and dprint, a single task can format the entire repository. This is exactly why the canonical `mise` vocabulary splits `mise run format` into `format:python` (Ruff) and `format:dprint` (dprint), while `mise run check:format` runs `ruff format --check` and `dprint check` so continuous integration fails on any unformatted file. ## How can you automate formatting? @@ -125,11 +189,15 @@ However, if your project requires specific adjustments, you can configure Ruff i ```toml [tool.ruff] # Set the maximum line length -line-length = 100 +line-length = 120 [tool.ruff.format] # Enable formatting of code snippets within docstrings docstring-code-format = true +# Always write LF endings, so Windows checkouts do not churn the diff +line-ending = "lf" +# Prefer double quotes (the default, stated explicitly for readers of the config) +quote-style = "double" [tool.ruff.lint.pydocstyle] # Set the expected docstring style (e.g., google, numpy) diff --git a/docs/4. Validating/index.md b/docs/4. Validating/index.md index d517755..1667c5a 100644 --- a/docs/4. Validating/index.md +++ b/docs/4. Validating/index.md @@ -2,10 +2,17 @@ description: This chapter emphasizes the importance of code validation for creating robust MLOps pipelines. Learn how to implement typing, linting, and testing to ensure the quality and reliability of your code. --- -# 4.0 Validating +# 4. Validating Code validation is the bedrock of robust MLOps. In this chapter, you'll master the essential techniques to ensure your ML pipelines are scalable, efficient, and reliable. From static analysis to dynamic debugging, these practices are critical for elevating code quality and operational excellence. +Every check in this chapter is exposed as a `mise run check:*` subtask, so the same command runs in your terminal, in your git hooks, and in continuous integration. The canonical gate that ties them together is a single task: + +```bash +# Format, check, test, and build the project +mise run all +``` + - **[4.0. Typing](./4.0. Typing.md):** Implement static type checking to catch errors early and enhance code clarity. - **[4.1. Linting](./4.1. Linting.md):** Use linting to enforce coding standards, eliminate errors, and improve code maintainability. - **[4.2. Testing](./4.2. Testing.md):** Master testing methodologies to verify code behavior and guarantee your models perform as intended. diff --git a/docs/5. Refining/5.1. Task Automation.md b/docs/5. Refining/5.1. Task Automation.md index 97ac541..f02016e 100644 --- a/docs/5. Refining/5.1. Task Automation.md +++ b/docs/5. Refining/5.1. Task Automation.md @@ -35,7 +35,7 @@ While [`Make`](https://en.wikipedia.org/wiki/Make_(software)) is powerful and wi A modern, more intuitive alternative is [`mise`](https://mise.jdx.dev/) (pronounced "meez"), a fast tool written in Rust that combines two jobs in a single file: 1. **A task runner**: It defines project tasks with a clean, readable syntax—replacing tools like `Make`, `Just`, or `PyInvoke`. -1. **A tool-version manager**: It pins the exact versions of your command-line tools (`uv`, `dprint`, `gitleaks`, `trivy`, and even Python itself), so every contributor and every CI machine runs an identical toolchain. +1. **A tool-version manager**: It pins the exact versions of your command-line tools (`uv`, `dprint`, `gitleaks`, `trivy`, `hadolint`, and even Python itself), so every contributor and every CI machine runs an identical toolchain. Consider this example from the [MLOps Python Package template](https://github.com/fmind/mlops-python-package/blob/main/mise.toml) for building a Python distribution: @@ -79,17 +79,20 @@ run_auto_install = false # TOOLS: pin the exact command-line tools every contributor and CI run should use. [tools] -python = "3.14" -uv = "latest" +actionlint = "latest" dprint = "latest" +git-cliff = "latest" gitleaks = "latest" +hadolint = "latest" trivy = "latest" +uv = "latest" +zizmor = "latest" # TASKS: define the project's automation entry points. [tasks.install] alias = "i" description = "Install dependencies and git hooks" -depends = ["install:python", "install:hooks"] +depends = ["install:hooks"] [tasks."install:python"] description = "Sync Python dependencies (uv)" @@ -97,9 +100,16 @@ run = "uv sync --all-groups" [tasks."install:hooks"] description = "Install git hooks (lefthook)" +depends = ["install:python"] run = "uv run lefthook install" ``` +Two details in this file are worth reading twice. + +First, the dependency chain is deliberate: `install` depends on `install:hooks`, which itself depends on `install:python`. `mise` resolves the chain, so a single `mise run install` syncs the environment *before* it tries to run `lefthook` from that environment. Declaring `depends = ["install:python", "install:hooks"]` on `install` instead would let `mise` start both in parallel and fail on a fresh clone. + +Second, the Python interpreter is deliberately absent from `[tools]` in the MLOps Python Package. A `.python-version` file at the repository root holds `3.14`, and `uv` reads it to provision the interpreter—so `mise` pins `uv`, and `uv` pins Python. One version, one file, no chance of the two disagreeing. Note that `mise` does *not* read `.python-version` on its own: support for these "idiomatic" version files is off by default (`idiomatic_version_file_enable_tools` is empty), so listing `python = "3.14"` under `[tools]` as well would be a second source of truth to keep in sync, not a safety net. + `mise` requires you to explicitly trust a project's configuration before it will run any of its tasks or install its tools—a safety measure that protects you from executing untrusted code when you clone a new repository: ```bash @@ -118,18 +128,92 @@ mise run install For more details, refer to the official [mise documentation](https://mise.jdx.dev/). +### Why set `run_auto_install = false`? + +By default, `mise` installs a missing tool on demand the first time a task needs it. That convenience becomes a liability the moment tasks are run from a git hook or a CI job: a pre-commit hook that silently downloads `trivy` mid-commit turns a two-second check into a two-minute one, and a CI job that installs a tool outside of `mise install` can quietly run a version nobody pinned. + +Setting `run_auto_install = false` under `[settings.task]` turns a missing tool into an immediate, explicit error. Installation becomes a deliberate step (`mise install`, or the `jdx/mise-action` step in CI), and every task run afterwards uses exactly the toolchain the repository declared. + +### How do you lock the toolchain with `mise.lock`? + +Pinning `trivy = "latest"` under `[tools]` says *which* tool you want, not *which build*. Two contributors who ran `mise install` a month apart can easily end up on different releases, and a new scanner release can turn a green repository red for reasons that have nothing to do with your change. + +`mise` solves this the same way `uv` does, with a lockfile. Turn the `lockfile` setting on—in your project's `mise.toml` or in your global `~/.config/mise/config.toml`—and declare the platforms your team and CI actually run on: + +```toml +[settings] +lockfile = true +lockfile_platforms = ["linux-x64", "macos-arm64"] +``` + +Then generate or refresh `mise.lock`: + +```bash +# Resolve every tool in [tools] and record versions, URLs, and checksums +mise lock + +# Target specific platforms explicitly, or a single tool +mise lock --platform linux-x64,macos-arm64 +mise lock trivy +``` + +The generated `mise.lock` records, for every pinned tool, the exact resolved version, the download URL, and a SHA-256 checksum per platform: + +```toml +# @generated - this file is auto-generated by `mise lock` + +[[tools.actionlint]] +version = "1.7.12" +backend = "aqua:rhysd/actionlint" + +[tools.actionlint."platforms.linux-x64"] +checksum = "sha256:8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8" +url = "https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz" +``` + +**Commit `mise.lock` next to `uv.lock`.** The two files are the same idea applied to two layers: `uv.lock` pins the Python dependencies your code imports, `mise.lock` pins the command-line tools your tasks execute. Together they make a clone reproducible from the interpreter up. The checksums also matter: they turn a tool download into a verified one, so a compromised release artifact fails the install instead of running. + +The payoff reaches CI as well. `jdx/mise-action` keys its cache on `mise.lock`, so a run that changes no tool reuses the previously downloaded binaries instead of fetching them again—see the [CI/CD Workflows](./5.3. CI-CD Workflows.md) chapter. + ### How should you organize your tasks in an MLOps project? The key to a maintainable project is a **shared task vocabulary**: a small, predictable set of top-level tasks that means the same thing in every repository. Agents, new contributors, git hooks, and CI can all rely on it without reading the implementation. The MLOps Python Package template standardizes on these core tasks (each with a short alias): -| Task | Alias | Purpose | -| --------- | ----- | ---------------------------------------------------------- | -| `install` | `i` | Sync dependencies and install git hooks. | -| `format` | `f` | Auto-format all sources and documents. | -| `check` | `c` | Run all static checks (lint, types, format, security). | -| `test` | `t` | Run the test suite with coverage. | -| `build` | `b` | Build the distribution artifacts (wheel, image). | -| `watch` | `w` | Run the app with live reload (for web services). | +| Task | Alias | Purpose | +| --------- | ----- | --------------------------------------------------------------------------------------------------- | +| `all` | `a` | Run the whole gate in order: `format`, `check`, `test`, `build`. | +| `install` | `i` | Sync dependencies and install git hooks. | +| `format` | `f` | Auto-format all sources and documents. | +| `check` | `c` | Run every static check (format, lint, types, secrets, vulnerabilities, misconfigurations, workflows, Dockerfile). | +| `test` | `t` | Run the test suite with coverage. | +| `build` | `b` | Build the distribution artifacts (wheel + sdist). | + +Beyond that vocabulary, a project adds whatever top-level tasks it genuinely needs. The MLOps Python Package adds `docs` (`d`) to generate the API documentation, `coverage` (`v`) to open the HTML coverage report, `upgrade` (`u`), `clean` (`n`), `project` to run every MLflow job in sequence, and namespaced helpers such as `mlflow:serve`, `build:image`, and `docker:compose`. The point of the vocabulary is not to forbid extra tasks—it is that the six above always exist and always mean the same thing. + +### Why does the `all` task matter? + +`all` is the smallest task in `mise.toml` and the most important one: + +```toml +[tasks.all] +alias = "a" +description = "Format, check, test, and build the project (the canonical gate)" +run = ["mise run format", "mise run check", "mise run test", "mise run build"] +``` + +Note that `run` is a list, not `depends`. `depends` would let `mise` run the four in parallel; a list runs them **in order**, which is what you want here—formatting must happen before checking, and building an artifact from unchecked, untested code is wasted work. + +The reason to name this sequence is that **a list of steps can omit one, but a named gate cannot**. Before this task existed, the CI workflow spelled out its own steps—`mise run format`, then `mise run check`, then `mise run test`—and simply forgot `mise run build`. Nothing was broken and nothing complained; the packaging step was just never exercised on any pull request, and a broken `pyproject.toml` build section would only have surfaced at release time. That is the classic failure mode of duplicated pipelines: they do not diverge loudly, they diverge silently, by omission. + +With `all` in place there is exactly one definition of "the project passes". CI runs a single step: + +```bash +mise run all +``` + +So do you, before opening a pull request. Your [git hooks](./5.2. Pre-Commit Hooks.md) run the same tasks, split across the moments where each one is cheap: the formatters and `check` on `pre-commit`, `test` on `pre-push`. Adding a new gate—say, a Dockerfile linter—means adding one line to `check`, and every hook, every terminal, and every CI run picks it up at once. Nobody has to remember to update a YAML file. + +### How do you organize the subtasks? Each top-level task fans out to **namespaced subtasks** written as `task:subtask`. This keeps `mise.toml` modular and lets you run a single piece in isolation. For example, `check` runs every static check in parallel by depending on its subtasks: @@ -138,12 +222,14 @@ Each top-level task fans out to **namespaced subtasks** written as `task:subtask alias = "c" description = "Run all static checks in parallel" depends = [ + "check:actions", + "check:dockerfile", "check:format", "check:leaks", "check:lint", + "check:scan", "check:types", "check:vuln", - "check:scan", ] # Lint Python sources (Ruff) @@ -159,13 +245,25 @@ run = "uv run ty check" # Scan dependencies for known vulnerabilities (pip-audit) [tasks."check:vuln"] description = "Scan dependencies for vulnerabilities (pip-audit)" -run = "uv run pip-audit" +run = "uv run pip-audit --skip-editable --cache-dir .cache/pip-audit" + +# Lint and audit the GitHub Actions workflows themselves +[tasks."check:actions"] +description = "Lint and audit GitHub Actions workflows (actionlint + zizmor)" +run = ["actionlint", "zizmor --offline .github/workflows/"] + +# Lint the container image definition +[tasks."check:dockerfile"] +description = "Lint the container image definition (hadolint)" +run = "hadolint Dockerfile" ``` The subtask names follow a simple convention: - **`format:`** keys off the *source you format*: `format:python` (Ruff) for `.py` files, and `format:dprint` (dprint) for JSON, Markdown, TOML, and YAML. -- **`check:`** keys off the *property verified*, so the name means the same thing in any language: `check:format`, `check:lint`, `check:types`, `check:vuln` (dependency vulnerabilities), `check:leaks` (leaked secrets), and `check:scan` (configuration misconfigurations). +- **`check:`** keys off the *property verified*, so the name means the same thing in any language: `check:format`, `check:lint`, `check:types`, `check:vuln` (dependency vulnerabilities), `check:leaks` (leaked secrets), `check:scan` (misconfigurations, licenses, and secrets across the checkout), `check:actions` ([CI workflows](./5.3. CI-CD Workflows.md)), and `check:dockerfile` (the [container image definition](./5.4. Software Containers.md)). + +Because `check` fans out with `depends`, `mise` runs all eight subtasks **in parallel**. That parallelism is worth knowing about when a subtask writes to disk: `check:vuln` uses `--cache-dir .cache/pip-audit`, and `check:scan` is configured to skip `.cache`, precisely so that Trivy does not walk a directory pip-audit is writing into at the same moment. This structure lets you run an individual subtask, the whole group, or the entire suite with equally simple commands: @@ -178,6 +276,9 @@ mise run check:types # Run every static check (fans out to all check:* subtasks in parallel) mise run check + +# Run the full gate: format, check, test, build +mise run all ``` ### What are some best practices for writing automation tasks? @@ -186,7 +287,9 @@ To maximize the benefits of task automation, follow these best practices: - **Keep Tasks Atomic**: Each task should have a single, well-defined purpose (e.g., `check:types` instead of a combined check-and-format). Atomic tasks are easier to debug, reuse, and compose. - **Create Meta-Tasks with `depends`**: Combine smaller subtasks into larger workflows using the `depends` array. The `check` task, which fans out to every `check:*` subtask, is a perfect example—and `mise` runs the dependencies in parallel for you. -- **Pin Your Toolchain in `[tools]`**: Declare the exact tools your tasks need so every contributor and CI runner uses identical versions. This eliminates "it works on my machine" surprises without a separate installation step. +- **Name the Gate**: Define a single `all` task that runs `format`, `check`, `test`, and `build` in order, and make it the one thing CI executes. A named gate cannot silently lose a step the way a hand-written list of CI steps can. +- **Pin Your Toolchain in `[tools]`, then Lock It**: Declare the exact tools your tasks need so every contributor and CI runner uses identical versions, and commit the `mise.lock` produced by `mise lock` alongside `uv.lock`. This eliminates "it works on my machine" surprises without a separate installation step. +- **Fail Loudly on a Missing Tool**: Keep `run_auto_install = false` so a task never silently downloads a tool mid-hook. Installation is its own explicit step. - **Use `[env]` and `.env` Files**: Load environment-specific configuration through `[env]` and a local `.env` file (via `_.source = ".env"`) instead of hardcoding values inside tasks. - **Document Every Task**: Give each task a `description`. `mise` surfaces these in `mise tasks` and interactive selectors, turning your `mise.toml` into self-documenting project onboarding. - **Make `mise` the Single Source of Truth**: Have your [git hooks](./5.2. Pre-Commit Hooks.md) and [CI/CD workflows](./5.3. CI-CD Workflows.md) call `mise run ` rather than re-implementing commands. When the definition lives in one place, local checks and remote pipelines can never disagree. @@ -196,3 +299,4 @@ To maximize the benefits of task automation, follow these best practices: - **[Task automation example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/mise.toml)** - **[Official mise Documentation](https://mise.jdx.dev/)** - **[mise Tasks Guide](https://mise.jdx.dev/tasks/)** +- **[mise Lockfile Documentation](https://mise.jdx.dev/dev-tools/mise-lock.html)** diff --git a/docs/5. Refining/5.2. Pre-Commit Hooks.md b/docs/5. Refining/5.2. Pre-Commit Hooks.md index 64a3cb4..c39c12c 100644 --- a/docs/5. Refining/5.2. Pre-Commit Hooks.md +++ b/docs/5. Refining/5.2. Pre-Commit Hooks.md @@ -29,7 +29,7 @@ First, add `lefthook` to your project. Pin it as a development dependency and ex ```toml # dependency-groups in pyproject.toml [dependency-groups] -dev = ["lefthook>=2.1.9"] +dev = ["lefthook>=2.1.10"] ``` Next, create a `lefthook.yml` file in your project's root directory. Every command delegates to a `mise run` task—there are no inline tool invocations to keep in sync: @@ -37,25 +37,27 @@ Next, create a `lefthook.yml` file in your project's root directory. Every comma ```yaml # https://lefthook.dev # Thin hooks: every command delegates to a `mise run` task so hooks and CI stay identical. +# Lefthook orders commands alphabetically, so priorities are explicit: formatters (10) +# restage before the staged secret scan (20) and the whole-tree checks (30) read from disk. pre-commit: parallel: false commands: format:dprint: - priority: 1 + priority: 10 glob: "*.{json,md,toml,yaml,yml}" run: mise run format:dprint {staged_files} stage_fixed: true format:python: - priority: 1 + priority: 10 glob: "*.py" run: mise run format:python {staged_files} stage_fixed: true check:leaks: - priority: 2 + priority: 20 run: mise run check:leaks --staged check: - priority: 3 + priority: 30 run: mise run check pre-push: @@ -82,13 +84,42 @@ uv run lefthook run pre-commit This small `lefthook.yml` encodes a few important design decisions: -- **Thin, delegated commands**: Every `run` is just `mise run `. The tool versions (Ruff, gitleaks, trivy, ...) live in `pyproject.toml` and `mise.toml`, not scattered across hook definitions—so there is only one place to update them. +- **Thin, delegated commands**: Every `run` is just `mise run `. The tool versions (Ruff, gitleaks, trivy, ...) live in `pyproject.toml`, `mise.toml`, and `mise.lock`, not scattered across hook definitions—so there is only one place to update them. - **Format staged files, check the whole tree**: The formatters receive `{staged_files}` and restage their fixes automatically with `stage_fixed: true`, keeping commits fast. The `check` and `test` tasks take no file list, so they always validate the whole project and correctness stays global. -- **Ordered, sequential execution**: Lefthook runs a hook's commands *alphabetically by name* by default, which would let `check` run before the formatters. Setting an explicit `priority` (formatters first, `check` last) together with `parallel: false` ensures the formatters restage their changes *before* `check` reads the files from disk. -- **Fast local secret scanning**: `check:leaks --staged` scans only the incoming change for credentials, complementing the deeper history scan that runs in CI. +- **Ordered, sequential execution**: The `priority` numbers and `parallel: false` are what make the hook correct rather than merely convenient—see below. +- **Layered secret scanning**: `check:leaks --staged` scans only the incoming change for credentials, and it is one of three passes at different depths—see below. You only need to pin one tool—`lefthook` itself (2.x)—because everything the hooks execute is defined by your `mise` tasks. This is a major simplification over legacy setups that duplicated every linter's version inside the hook config. +## Why do the priorities jump by ten? + +Lefthook runs a hook's commands **alphabetically by name** when you do not tell it otherwise. Read the command names in the file above in alphabetical order and the problem is obvious: `check` comes before `check:leaks`, which comes before `format:dprint` and `format:python`. Without priorities, the hook lints and type-checks your files *first*, then formats them—so an unformatted file fails `check:format` even though the very next command was about to fix it. You would fix nothing, re-run `git commit`, and it would pass the second time. A hook that only works on the retry is a hook people learn to bypass. + +The `priority` key overrides that ordering explicitly. Lower numbers run first, so the three tiers are: + +1. **10 — formatters.** They rewrite the staged files and restage the result via `stage_fixed: true`. +1. **20 — the staged secret scan.** It reads the index *after* formatting, so it inspects exactly the bytes that are about to be committed. +1. **30 — the whole-tree checks.** `mise run check` reads the files from disk, which now match what is staged. + +Two supporting details make the ordering real: + +- **`parallel: false`** is mandatory. Priorities decide the order of *starting*, but only sequential execution guarantees that the formatters have finished writing and restaging before `check` reads from disk. With `parallel: true`, the three tiers would race. +- **The gaps of ten** are a convention, not a requirement. Numbering 10/20/30 instead of 1/2/3 leaves room to slot a new command *between* two tiers later—a schema validator at 15, say—without renumbering the whole file and re-reviewing every line of the diff. + +## How should you layer secret scanning? + +Secret scanning is the clearest example of why a single check is not enough. The same tool, `gitleaks`, runs three times at three different depths, and each pass catches something the others structurally cannot: + +| Where | Command | What it can catch | +| ----- | ------- | ----------------- | +| `pre-commit` hook | `mise run check:leaks --staged` | The secret **before it exists in history**—the only moment a fix is free. | +| `mise run check` (local and CI) | `gitleaks git --log-opts="--max-count=100" --verbose` | A secret in the recent commits, fast enough to run on every pull request. | +| Scheduled workflow | `gitleaks git --redact=100 --verbose` over a full-depth checkout | A secret that was committed and later deleted, which no shallow scan will ever see again. | + +The middle pass is the compromise: bounding it with `--max-count=100` keeps the gate quick, but that bound is exactly what makes it blind to older history. That is why the third pass exists—a weekly `.github/workflows/security.yml` that checks out with `fetch-depth: 0` and scans everything. The [CI/CD Workflows](./5.3. CI-CD Workflows.md) chapter covers that workflow in detail. + +The order matters as much as the depth. Once a secret is committed, removing it requires rewriting history *and* rotating the credential, because anyone who pulled the branch already has it. The `--staged` pass is the only one that prevents the incident instead of reporting it. + ## How can you standardize commit messages and changelogs? Clear, consistent commit messages are vital for a healthy project history. The [Conventional Commits](https://www.conventionalcommits.org/) standard gives each commit a structured prefix that explains its intent: @@ -114,10 +145,22 @@ commit_parsers = [ { message = "^refactor", group = "♻️ Refactor" }, { message = "^docs", group = "📚 Documentation" }, { message = "^chore\\(release\\)", skip = true }, + { message = "^chore\\(deps\\)", skip = true }, ] tag_pattern = "v[0-9].*" ``` +The two `skip = true` entries deserve a warning, because a skip rule is a silent contract with whatever writes those commits. Release commits are written by you, so `^chore\(release\)` matches by construction. Dependency bumps are written by **Dependabot**, which by default prefixes its commits `build(deps)` or `build(deps-dev)`—not `chore(deps)`. A repository with the filter above and a default Dependabot configuration therefore filters nothing: the rule never matches, and every routine dependency bump lands in the published changelog next to the features users actually care about. + +The fix is to make the bot speak the convention the filter expects, in `.github/dependabot.yml`: + +```yaml +commit-message: + prefix: "chore(deps)" +``` + +The general lesson is worth more than the specific fix: **a filter that matches nothing looks exactly like a filter that has nothing to filter.** Whenever you add a skip rule, generate the changelog once and confirm that something actually disappeared. + You can then generate or update your changelog and cut releases from the same commit history: ```bash diff --git a/docs/5. Refining/5.3. CI-CD Workflows.md b/docs/5. Refining/5.3. CI-CD Workflows.md index b2d2a14..9a8c1cf 100644 --- a/docs/5. Refining/5.3. CI-CD Workflows.md +++ b/docs/5. Refining/5.3. CI-CD Workflows.md @@ -37,11 +37,11 @@ To create a workflow, you define a YAML file in the `.github/workflows` director ## What are the essential workflows for an MLOps project? -For a typical MLOps project, you should establish two primary workflows: one for verification and another for publication. +For a typical MLOps project, you should establish three workflows: one for verification on every change, one for publication on every release, and one scheduled deep security scan. ### [Continuous Integration Workflow](https://github.com/fmind/mlops-python-package/blob/main/.github/workflows/ci.yml) -This workflow runs on every pull request (and every push to `main`) to ensure that code changes meet quality standards before being merged. Notice that it runs the *exact same* `mise run` tasks you use locally and in your [git hooks](./5.2. Pre-Commit Hooks.md)—there is a single source of truth for what "format", "check", and "test" mean. +This workflow runs on every pull request (and every push to `main`) to ensure that code changes meet quality standards before being merged. Notice how little of it is CI-specific: the entire verification logic is one call to the `all` task defined in [`mise.toml`](./5.1. Task Automation.md), the same task you run in your terminal. ```yaml name: CI @@ -55,34 +55,57 @@ concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} jobs: - check: - runs-on: ubuntu-latest + checks: # name kept as "checks" to satisfy the repository's required-status-check ruleset + runs-on: ubuntu-24.04 + timeout-minutes: 20 steps: - name: Checkout repository uses: actions/checkout@v7 + with: + persist-credentials: false # no step pushes back to the repository - name: Install mise toolchain uses: jdx/mise-action@v4 - - name: Format sources - run: mise run format - - name: Run checks - run: mise run check - - name: Run tests - run: mise run test + with: + cache: true + - name: Run canonical gate + run: mise run all - name: Verify no changes - run: git diff --exit-code + run: test -z "$(git status --porcelain)" ``` **Workflow Breakdown:** - **`name`**: The workflow's name, "CI," as it appears in the GitHub UI. - **`on`**: Triggers the workflow on any pull request and on pushes to `main`. -- **`permissions`**: Grants the job read-only access by default, following the principle of least privilege. +- **`permissions`**: Grants the workflow read-only access by default, following the principle of least privilege. - **`concurrency`**: Ensures that only one run of this workflow per branch is active at a time. If a new commit is pushed to a branch, the previous run is canceled (but runs on `main` are never interrupted). -- **`jobs.check.steps`**: Defines the sequence of steps to execute. - - `actions/checkout@v7`: Checks out the repository code. - - `jdx/mise-action@v4`: Installs the toolchain pinned in `mise.toml` (Python, uv, dprint, and the security scanners) and syncs the project. This single step replaces separate "setup Python" and "install uv" actions. - - `mise run format` / `mise run check` / `mise run test`: Runs the canonical tasks for formatting, static analysis (lint, types, security), and the test suite—identical to what runs locally. - - `git diff --exit-code`: Fails the build if `mise run format` produced any changes, guaranteeing that all committed code is already formatted. +- **`jobs.checks`**: The single verification job. Its *name* is load-bearing—see below. + - `runs-on: ubuntu-24.04`: A pinned runner image rather than `ubuntu-latest`. `latest` moves under you: when GitHub promotes a new image, a build that passed yesterday can fail today for reasons no commit explains. Pinning makes the upgrade a deliberate, reviewable one-line change. + - `timeout-minutes: 20`: A hung job—a test waiting on a socket, a scanner stuck on a network call—otherwise burns runner minutes until GitHub's six-hour default kills it. A tight timeout turns a hang into a fast, obvious failure. + - `actions/checkout@v7` with `persist-credentials: false`: By default, `checkout` leaves a credential in `.git/config` for the rest of the job. No step here pushes back to the repository, so leaving that token available only widens the blast radius if any tool the job runs is compromised. + - `jdx/mise-action@v4` with `cache: true`: Installs the toolchain pinned in `mise.toml` and `mise.lock`, and caches the downloaded binaries keyed on the lockfile. This single step replaces separate "setup Python" and "install uv" actions—`uv` is pinned here, and `uv` in turn provisions the interpreter named in `.python-version`. + - `mise run all`: The whole gate—`format`, `check`, `test`, `build`—in one step. + - `test -z "$(git status --porcelain)"`: Fails the build if the gate modified or created *anything*. + +#### Why one `mise run all` step instead of four? + +An earlier version of this workflow spelled out its own sequence: `mise run format`, then `mise run check`, then `mise run test`. It looked complete. It was missing `mise run build`, and had been for a long time. Nothing failed, nothing warned—the packaging step simply never ran on any pull request. + +That is the failure mode of a duplicated pipeline. It does not break loudly when it drifts from the project's real definition of "passing"; it just quietly does less. Calling a single named task removes the possibility: adding a step to `all` adds it to CI, to your terminal, and to every contributor's pre-push run at the same instant, with no YAML to remember. + +#### Why `test -z "$(git status --porcelain)"` and not `git diff --exit-code`? + +Both commands are meant to answer the same question—*did running the gate change the working tree?*—and if it did, the change belongs in the commit, not in CI. + +But `git diff` only reports modifications to **tracked** files. A task that *creates* a new file leaves `git diff` perfectly empty. This is not hypothetical: moving the vulnerability scanner to `uv run pip-audit --skip-editable --cache-dir .cache/pip-audit` made it write a cache directory nobody had added to `.gitignore`, and `git diff --exit-code` passed happily on a checkout that had grown an untracked directory. + +`git status --porcelain` lists tracked modifications *and* untracked files, so `test -z` on its output fails on both. It is the stricter, more honest question: "is this checkout still exactly what was committed?" + +#### Why is the job named `checks`? + +Branch protection—whether through the classic branch protection rules or a [repository ruleset](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets)—requires a status check *by name*. That name is a plain string matched against the job name GitHub reports, and nothing validates that the string corresponds to a job that exists. + +Get it wrong and the failure is spectacularly quiet: the ruleset waits forever for a status check named `checks` while the workflow dutifully reports one named `check`, and every pull request is blocked with "expected — waiting for status to be reported". No error, no log, no hint. If you codify your ruleset in a JSON file (`.github/rulesets/main.json`) and install it with a task, keep the job name and the required context in sync, and verify on a throwaway pull request that the check actually appears. ### [Continuous Deployment Workflow](https://github.com/fmind/mlops-python-package/blob/main/.github/workflows/cd.yml) @@ -97,8 +120,10 @@ permissions: contents: read jobs: pages: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 + timeout-minutes: 20 permissions: + contents: read pages: write id-token: write environment: @@ -107,8 +132,12 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@v7 + with: + persist-credentials: false # no step pushes back to the repository - name: Install mise toolchain uses: jdx/mise-action@v4 + with: + cache: false # a poisoned tool cache must never reach published artifacts - name: Build API documentation run: mise run docs - name: Configure Pages @@ -121,14 +150,22 @@ jobs: id: deployment uses: actions/deploy-pages@v5 image: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 + timeout-minutes: 30 permissions: + contents: read packages: write steps: - name: Checkout repository uses: actions/checkout@v7 + with: + persist-credentials: false # the registry login below carries its own token - name: Set lower-case image path - run: echo "IMAGE=$(echo "ghcr.io/${{ github.repository }}" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV" + # GitHub expands `${{ }}` before the shell runs, so repository references reach + # the script through `env:` rather than being interpolated into the command. + env: + REPOSITORY: ${{ github.repository }} + run: echo "IMAGE=$(echo "ghcr.io/$REPOSITORY" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV" - name: Log in to GitHub Container Registry uses: docker/login-action@v4 with: @@ -142,8 +179,6 @@ jobs: with: context: . push: true - cache-to: type=gha - cache-from: type=gha tags: | ${{ env.IMAGE }}:latest ${{ env.IMAGE }}:${{ github.ref_name }} @@ -153,14 +188,160 @@ jobs: - **`on`**: Triggers the workflow when a release is `published`. - **`jobs.pages`**: Builds the documentation with `mise run docs` and deploys it with the official Pages pipeline. - - `permissions`: Grants `pages: write` and `id-token: write`, which the Pages deployment requires. + - `permissions`: Grants `contents: read`, `pages: write`, and `id-token: write`, which the Pages deployment requires. - `environment: github-pages`: Deploys into the protected `github-pages` environment and exposes the published URL. - `actions/configure-pages` → `actions/upload-pages-artifact` → `actions/deploy-pages`: The three official steps that configure Pages, package the `docs/` folder as an artifact, and deploy it—no `gh-pages` branch involved. - **`jobs.image`**: Builds and publishes the Docker image. - - `permissions`: Grants `packages: write`, allowing it to publish to the GitHub Container Registry (`ghcr.io`). + - `permissions`: Grants `contents: read` and `packages: write`, allowing it to publish to the GitHub Container Registry (`ghcr.io`). + - `env: REPOSITORY`: The repository name reaches the shell through an environment variable instead of being interpolated into the command string—see the script-injection note below. - `docker/login-action`: Logs into the container registry using the automatically provided `GITHUB_TOKEN`. - `docker/build-push-action`: Builds the image, tags it with `latest` and the release version, and pushes it. Using a container ensures a consistent, portable environment for running the ML model. +#### A job's `permissions` block replaces the workflow's—it does not merge + +This is the single most common way a hardened workflow accidentally breaks itself. The workflow above declares `permissions: contents: read` at the top level, and the `image` job needs to push a package, so it adds `packages: write`. + +If the job had written only: + +```yaml +permissions: + packages: write +``` + +it would have `packages: write` **and nothing else**—no `contents: read` at all. A job-level `permissions` block is a full replacement of the workflow-level one, not an addition to it. The job would then fail at `actions/checkout`, and the error message ("could not read from remote repository") points nowhere near the YAML that caused it. + +The habit that avoids this: whenever you add a job-level `permissions` block, restate *every* scope the job needs, including the read scopes the workflow already granted. Both jobs above spell out `contents: read` for exactly that reason. + +#### Why `cache: false` on release jobs but `cache: true` on CI? + +`jdx/mise-action` can cache the tool binaries it downloads, and on CI that is a straightforward win: the cache is keyed on `mise.lock`, so a run that changes no tool reuses the previous download. + +On a release job, the trade-off inverts. GitHub Actions caches are writable by any workflow run in the repository, including runs triggered from a pull request branch. A cache entry poisoned by one of those runs would be restored here—inside the job that signs, builds, and publishes what your users install. Rebuilding the toolchain from its pinned, checksummed sources costs a minute; letting attacker-controlled bytes into a published container image costs far more. The comment in the YAML records the reasoning, because the next person to read that line will otherwise "optimize" it back. + +This is not a hypothetical worry you have to spot by eye—a workflow auditor flags it, as the next section shows. + +### [Scheduled Security Workflow](https://github.com/fmind/mlops-python-package/blob/main/.github/workflows/security.yml) + +The CI workflow scans what you just changed. It cannot scan what you removed a year ago. + +Push and pull-request CI runs on a shallow checkout, and the secret scanner inside `mise run check` is deliberately bounded (`gitleaks git --log-opts="--max-count=100"`) to keep the gate fast. Both choices are right for a gate and wrong for an audit: a credential that was committed in March and deleted in April still sits in the object database, still works, and will never appear in either view again. + +The answer is a separate, scheduled workflow that trades speed for depth: + +```yaml +name: Security +# Push CI scans only the latest commits, so a secret committed earlier and later +# removed would never be seen again. This is the full-history counterpart: same +# pinned scanners, whole history, no reports to parse — a finding fails the job. +on: + schedule: + - cron: "17 3 * * 1" + workflow_dispatch: +permissions: + contents: read +concurrency: + group: security + cancel-in-progress: true +jobs: + scan: + name: Full-history scan + runs-on: ubuntu-24.04 + timeout-minutes: 30 + steps: + - name: Checkout complete history + uses: actions/checkout@v7 + with: + fetch-depth: 0 + persist-credentials: false # scanners only read the checkout + - name: Install mise system + uses: jdx/mise-action@v4 + - name: Scan complete Git history + run: gitleaks git --redact=100 --verbose + - name: Scan full checkout + run: trivy fs . +``` + +**Workflow Breakdown:** + +- **`on.schedule`**: A weekly cron (`17 3 * * 1`—Mondays at 03:17 UTC). The odd minute is intentional: GitHub delays workflows scheduled on the hour, when everyone else's crons fire. +- **`on.workflow_dispatch`**: Lets you run the audit on demand from the Actions tab, which is how you confirm it works without waiting a week. +- **`fetch-depth: 0`**: Clones the **complete** history. This one line is the entire reason the workflow exists—without it, the deep scan sees no more than CI does. +- **`--redact=100`**: Redacts the matched secret in the log output. Findings appear in a public run log; a scanner that prints the credential it found has published it a second time. +- **`trivy fs .`**: Re-runs the filesystem scan against the whole checkout, unbounded. +- **`timeout-minutes: 30`**: Deep scans are slow, but they are not unbounded—a full-history scan that has run for half an hour is stuck, not thorough. +- **`runs-on: ubuntu-24.04`**: The same pinned image as CI. Pinning matters most for a job whose result blocks a merge, and a scheduled audit could tolerate a moving image—but a scanner that changes underneath you turns "a new finding appeared" into a question about the runner rather than about your code. + +The reason this does not belong in push CI is simple arithmetic. A full-history clone plus a full-history scan takes minutes and grows with the age of the repository, and it would run on every commit of every pull request to discover the same thing it discovered yesterday. Findings in old history are not caused by your pull request and cannot be fixed by it. Put deep, slow, whole-repository audits on a schedule; keep the pull-request gate fast enough that nobody wants to bypass it. + +## How do you lint the workflows themselves? + +Everything above is code, and it is code that runs with your repository's credentials—yet it is usually the only code in the project that no linter, type checker, or test ever inspects. A typo in a step name is harmless; a typo in an `if:` expression can silently make a security step never run. + +Two tools cover the two distinct failure classes, and the `check:actions` task runs both: + +```toml +[tasks."check:actions"] +description = "Lint and audit GitHub Actions workflows (actionlint + zizmor)" +run = ["actionlint", "zizmor --offline .github/workflows/"] +``` + +- **[`actionlint`](https://github.com/rhysd/actionlint)** checks *correctness*. It validates the workflow schema, checks that expression syntax and context references are real (`github.evnt.name` is caught, not silently empty), verifies runner labels exist, and even runs `shellcheck` over your `run:` blocks—so an unquoted shell variable in a workflow is flagged just as it would be in a script. +- **[`zizmor`](https://docs.zizmor.sh/)** checks *security*. It audits for the attack patterns specific to CI: template injection, over-broad permissions, credential persistence, cache poisoning, and unpinned actions. + +Both are standalone binaries, so they are pinned in `mise.toml` alongside the rest of the toolchain and run offline as part of `mise run check`—no network, no API token, no separate CI job. + +### What does zizmor actually catch? + +Two findings from the workflows above are worth walking through, because neither is obvious by inspection. + +**Template injection.** The `image` job originally built its lower-case image name like this: + +```yaml +run: echo "IMAGE=$(echo "ghcr.io/${{ github.repository }}" | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_ENV" +``` + +GitHub expands `${{ }}` by **pasting the value into the script text** before the shell ever sees it. With a repository name that is safe; with a context an outsider controls—a branch name, an issue title, a pull request body—the pasted text becomes shell code running with your job's token. Passing the value through `env:` and referring to `$REPOSITORY` fixes it, because the shell then receives the value as data rather than as source code. + +**Cache poisoning.** Running `zizmor` against the CD workflow with `cache: true` on the `mise-action` step produces an error-level finding: + +```text +error[cache-poisoning]: runtime artifacts potentially vulnerable to a cache poisoning attack + --> .github/workflows/cd.yml:24:9 + | + 2 | / on: + 3 | | release: + 4 | | types: [published] + | |______________________- generally used when publishing artifacts generated at runtime +... +24 | uses: jdx/mise-action@v4 + | ^^^^^^^^^^^^^^^^^^^^^^^^ this step +25 | / with: +26 | | cache: true + | |_____________________- enables caching explicitly here +``` + +Read what it reasoned about: it connected the workflow's *trigger* (`release: published`, a workflow that publishes artifacts) to a *step* several jobs down that restores a mutable cache, and concluded that attacker-influenced cache contents could reach a published container image. That is a two-part inference across the file that a human reviewer skims straight past—and exactly why `cache: false` sits on the release jobs. + +### Why relax zizmor's pinning rule? + +Out of the box, `zizmor` requires every third-party action to be pinned to a full commit SHA, and flags `uses: actions/checkout@v7` as unpinned. That is a defensible policy, but it is not the only defensible one, and it is not the one these repositories chose: actions are pinned to **major-version tags** so that security patches within a major version arrive automatically, at the cost of trusting the tag owner not to move it maliciously. + +The wrong response to a rule you disagree with is to disable the tool, or to sprinkle inline suppressions until it goes quiet. The right response is to configure the rule to enforce the policy you actually hold, in `.github/zizmor.yml`: + +```yaml +# Actions are deliberately pinned to major-version tags: tags track security +# patches within a major, at the cost of trusting the tag. zizmor's default policy +# demands hash-pins; relax it to ref-pins so the audit enforces the actual policy +# instead of fighting it. +rules: + unpinned-uses: + config: + policies: + "*": ref-pin +``` + +With this file, `uses: actions/checkout@v7` passes and `uses: actions/checkout@main` still fails—a floating branch reference is caught, a version tag is accepted. The audit now encodes a decision instead of a default, and the comment explains it to whoever revisits the choice. Note the general principle, which applies to every linter in this course: **tune the rule, do not silence the tool.** + ## How can you avoid repeating steps in CI/CD workflows? The best way to follow the [DRY (Don't Repeat Yourself)](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle in MLOps is to push logic *down* into your `mise` tasks rather than duplicating shell commands across workflows. Both the CI and CD examples above start with the same two lines: @@ -170,14 +351,46 @@ The best way to follow the [DRY (Don't Repeat Yourself)](https://en.wikipedia.or - uses: jdx/mise-action@v4 ``` -A single `jdx/mise-action@v4` step reads your `mise.toml`, installs the pinned toolchain (Python, uv, dprint, gitleaks, trivy, ...), and syncs the project. Because every job then calls high-level tasks like `mise run check` or `mise run docs`, there is nothing to re-declare per workflow—the definitions live in one place and are shared with your terminal and [git hooks](./5.2. Pre-Commit Hooks.md). +A single `jdx/mise-action@v4` step reads your `mise.toml` and `mise.lock` and installs the pinned toolchain (uv, dprint, gitleaks, trivy, hadolint, actionlint, zizmor, ...); the tasks it then runs use `uv` to provision the Python interpreter from `.python-version`. Because every job then calls high-level tasks like `mise run all` or `mise run docs`, there is nothing to re-declare per workflow—the definitions live in one place and are shared with your terminal and [git hooks](./5.2. Pre-Commit Hooks.md). For repository-specific step sequences that are *not* tasks (for example, a bespoke sign-and-attest flow), you can still encapsulate them into a reusable **composite action** stored under `.github/actions`, then reference it with `- uses: ./.github/actions/`. You can also find thousands of pre-built actions on the [GitHub Marketplace](https://github.com/marketplace?type=actions) to integrate with third-party services and streamline your workflows. +## How do you keep action versions current? + +Pinning actions to major tags only helps if somebody eventually moves the pin. [Dependabot](https://docs.github.com/en/code-security/dependabot) does that for you, and it deserves an explicit configuration rather than the defaults. Declare one entry per ecosystem in `.github/dependabot.yml`—here, GitHub Actions alongside the Python and Docker dependencies of the same project: + +```yaml +version: 2 +updates: + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + commit-message: + prefix: "chore(deps)" + groups: + actions: + patterns: ["*"] + update-types: ["minor", "patch"] +``` + +Three choices are worth copying: + +- **`groups`** collapses every minor and patch bump into one pull request per ecosystem. Because `update-types` deliberately excludes `major`, a breaking upgrade still arrives on its own, where it gets the review it deserves. +- **`schedule.day: monday`** puts the review in a predictable slot instead of scattering it across the week. +- **`commit-message.prefix: "chore(deps)"`** makes the bot's commits match the convention your changelog filter expects—see the [Pre-Commit Hooks](./5.2. Pre-Commit Hooks.md) chapter for why the default prefix quietly breaks `git-cliff`. + +Note also that the `docker` ecosystem only tracks base images pinned to a concrete tag. A `FROM` or `COPY --from` line referring to `:latest` is invisible to Dependabot, which is one of the reasons the [Dockerfile](./5.4. Software Containers.md) pins its `uv` image. + ## What are some best practices for CI/CD in MLOps? - **Automate Everything**: Automate all manual steps in your ML lifecycle, including data validation, model training, evaluation, and deployment, to reduce human error and increase velocity. -- **Manage Secrets Securely**: Use encrypted secrets to store sensitive information like API keys, passwords, and cloud credentials. GitHub Actions provides a secure way to manage secrets at the repository or organization level. +- **Call One Task, Not a List of Steps**: Let the workflow run `mise run all` and keep the definition of "passing" in `mise.toml`. A hand-written list of CI steps drifts by omission, and nothing tells you when it does. +- **Lint Your Workflows**: Run `actionlint` and `zizmor` on `.github/workflows/` as part of `mise run check`. Workflow YAML is privileged code; treat it like the rest of your source. +- **Pin the Runner and Bound the Job**: Prefer `runs-on: ubuntu-24.04` over `ubuntu-latest`, and give every job a `timeout-minutes`. Upgrades then happen when you choose, and a hang fails fast instead of burning an hour. +- **Grant the Least Privilege, Job by Job**: Default the workflow to `permissions: contents: read`, add `persist-credentials: false` to `checkout` in jobs that never push, and remember that a job-level `permissions` block *replaces* the workflow-level one. +- **Manage Secrets Securely**: Use encrypted secrets to store sensitive information like API keys, passwords, and cloud credentials. GitHub Actions provides a secure way to manage secrets at the repository or organization level. Never interpolate `${{ }}` into a `run:` script—pass values through `env:` so the shell treats them as data. - **[Master GitHub Actions Syntax](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions)**: A deep understanding of the workflow syntax, including contexts, expressions, and triggers, will allow you to build highly dynamic and powerful pipelines. - **[Use Concurrency Strategically](https://docs.github.com/en/actions/using-jobs/using-concurrency)**: The `concurrency` key is essential for managing workflow runs efficiently, preventing race conditions, and saving resources by canceling outdated jobs. - **[Leverage the GitHub CLI](https://cli.github.com/)**: Use the `gh` command-line tool to interact with your workflows, check run status, and trigger them manually (e.g., `gh workflow run ...`), streamlining your development loop. @@ -187,3 +400,6 @@ For repository-specific step sequences that are *not* tasks (for example, a besp - **[CI/CD Workflow example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/tree/main/.github)** - **[Official GitHub Actions Documentation](https://docs.github.com/en/actions)** +- [actionlint: static checker for GitHub Actions workflows](https://github.com/rhysd/actionlint) +- [zizmor: static analysis for GitHub Actions security](https://docs.zizmor.sh/) +- [Dependabot options reference](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference) diff --git a/docs/5. Refining/5.4. Software Containers.md b/docs/5. Refining/5.4. Software Containers.md index f2ea8e7..97b6343 100644 --- a/docs/5. Refining/5.4. Software Containers.md +++ b/docs/5. Refining/5.4. Software Containers.md @@ -82,7 +82,8 @@ ENV UV_COMPILE_BYTECODE=1 ENV UV_LINK_MODE=copy WORKDIR /app # Bring in the uv binary from its official image (no separate install step). -COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ +# Pinned, not `:latest` — a floating tag is invisible to Dependabot's docker ecosystem. +COPY --from=ghcr.io/astral-sh/uv:0.12.3 /uv /uvx /bin/ # Install dependencies first (cached layer), then the project itself. RUN --mount=type=cache,target=/root/.cache/uv \ --mount=type=bind,source=uv.lock,target=uv.lock \ @@ -96,9 +97,11 @@ RUN --mount=type=cache,target=/root/.cache/uv \ FROM python:3.14-slim ENV PYTHONUNBUFFERED=1 WORKDIR /app -RUN groupadd -r app && useradd -r -g app app -USER app -COPY --from=build --chown=app:app /app/.venv /app/.venv +# Fixed numeric uid/gid: stable file ownership across rebuilds and bind mounts, and +# resolvable by a host that does not share this image's /etc/passwd. +RUN groupadd -r -g 10001 app && useradd -r -u 10001 -g app -m app +USER 10001:10001 +COPY --from=build --chown=10001:10001 /app/.venv /app/.venv ENV PATH="/app/.venv/bin:$PATH" ENTRYPOINT ["bikes"] CMD ["--help"] @@ -114,12 +117,75 @@ docker build --tag=bikes:latest . docker run --rm bikes:latest ``` +Both commands are wrapped as [`mise` tasks](./5.1. Task Automation.md) so they are spelled the same way everywhere: + +```bash +mise run build:image # docker build --tag=bikes:latest . +mise run docker:run # docker run --rm bikes:latest +``` + +## How do you lint your Dockerfile? + +A `Dockerfile` is a build script that runs as `root`, and like your [workflow YAML](./5.3. CI-CD Workflows.md), it is easy to leave uninspected. [`hadolint`](https://hadolint.github.io/hadolint/) fixes that: it parses the `Dockerfile` into an AST, applies a rule set of Docker best practices, and runs `shellcheck` over every `RUN` command. It is a standalone binary, so it is pinned in `mise.toml` and wired into the project's static checks: + +```toml +[tasks."check:dockerfile"] +description = "Lint the container image definition (hadolint)" +run = "hadolint Dockerfile" +``` + +Because `check:dockerfile` is one of the subtasks `mise run check` depends on, the linter runs on every commit through the [pre-commit hook](./5.2. Pre-Commit Hooks.md) and on every pull request through CI—the same run, no separate configuration. + +Two lines in the `Dockerfile` above exist specifically because a linter or a bot asked for them. + +### Pin the `uv` image instead of using `:latest` + +The build stage originally copied the `uv` binary from `ghcr.io/astral-sh/uv:latest`. That is convenient and wrong for two reasons. + +The first is reproducibility, which you already know: `:latest` means "whatever was pushed most recently", so two builds of the same commit can embed different tool versions. + +The second is subtler and is the reason it got fixed. [Dependabot](https://docs.github.com/en/code-security/dependabot)'s `docker` ecosystem updates base images by rewriting a **concrete version** into a newer one. A floating tag has no version to rewrite, so Dependabot sees nothing to do and stays silent. The image was not "always up to date"—it was outside the update system entirely, and nobody would ever have been told about a `uv` release or a vulnerability fix. Pinning `ghcr.io/astral-sh/uv:0.12.3` makes the version visible: it now arrives as a reviewable pull request like every other dependency. + +The same reasoning applies to your `FROM python:3.14-slim` line: a concrete tag is what makes updates trackable. + +### Use a numeric `USER` + +The original final stage created a user and switched to it by name: + +```dockerfile +RUN groupadd -r app && useradd -r -g app app +USER app +COPY --from=build --chown=app:app /app/.venv /app/.venv +``` + +Running `hadolint` on that reports a finding on the `USER` line (the number below is whichever line it sits on in your file): + +```text +Dockerfile:5 DL3066 info: Non-numeric user-id may not be resolvable by host system +``` + +The rule is about a boundary the `Dockerfile` cannot see. Inside the image, `app` resolves through `/etc/passwd` to whatever uid `useradd` happened to allocate. Outside the image, the host kernel only ever deals in numbers, and it does not read your image's `/etc/passwd`. So a Kubernetes `runAsNonRoot` check, a `securityContext` comparison, or a bind-mounted volume's file ownership all operate on a uid that the `Dockerfile` never stated and that a rebuild could change. + +Declaring the number fixes the boundary in place: + +```dockerfile +RUN groupadd -r -g 10001 app && useradd -r -u 10001 -g app -m app +USER 10001:10001 +COPY --from=build --chown=10001:10001 /app/.venv /app/.venv +``` + +The uid is now part of the image's contract: stable across rebuilds, meaningful to the host, and unambiguous to an orchestrator. Note that the group is pinned too (`-g 10001`, and `10001:10001` in `USER`), because file ownership on a mounted volume depends on both. + +`10001` is a conventional choice—a high, unprivileged id that will not collide with system accounts on the host. + ## How can you optimize your container images and workflow? - **Use Multi-Platform Builds**: Use `docker buildx` to build images that can run on different CPU architectures (e.g., `amd64` for cloud servers and `arm64` for Apple Silicon Macs). This "build once, run anywhere" approach is highly efficient. - **Leverage Layer Caching**: Docker builds images in layers. Structure your `Dockerfile` to place steps that change less frequently (like installing system dependencies) before steps that change often (like copying your source code). This allows Docker to reuse cached layers, dramatically speeding up subsequent builds. - **Minimize Image Size**: Smaller images are faster to pull and deploy. After installing packages, clean up cache directories and temporary files. For example, in Debian-based images, add `&& rm -rf /var/lib/apt/lists/*` to your `apt-get install` command. -- **Lint Your Dockerfile**: Use a linter like [Hadolint](https://hadolint.github.io/hadolint/) to automatically check your `Dockerfile` for common mistakes, security vulnerabilities, and violations of best practices. +- **Lint Your Dockerfile Automatically**: Do not run [Hadolint](https://hadolint.github.io/hadolint/) by hand when you remember to. Wire it into `check:dockerfile` so it runs with every other static check, locally and in CI. +- **Pin Every Image Reference**: Give every `FROM` and `COPY --from` a concrete version tag. `:latest` breaks reproducibility *and* hides the dependency from Dependabot. +- **Scan the Image, Not Just the Source**: `trivy image ` inspects the built image's operating-system packages and installed libraries—layers your source-level dependency scanner never sees. - **Manage GPU Dependencies**: For deep learning, your image must include the necessary NVIDIA drivers. Instead of installing them manually, use official base images from NVIDIA, such as `nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04`. ## Additional Resources @@ -127,3 +193,5 @@ docker run --rm bikes:latest - **[Dockerfile example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/Dockerfile)** - [Containerize a Python application](https://docs.docker.com/language/python/containerize/) - [Docker in Visual Studio Code](https://code.visualstudio.com/docs/containers/overview) +- [Hadolint rule reference](https://github.com/hadolint/hadolint#rules) +- [Using uv in Docker](https://docs.astral.sh/uv/guides/integration/docker/) diff --git a/docs/5. Refining/5.5. AI-ML Experiments.md b/docs/5. Refining/5.5. AI-ML Experiments.md index 3a0a9ba..50558d6 100644 --- a/docs/5. Refining/5.5. AI-ML Experiments.md +++ b/docs/5. Refining/5.5. AI-ML Experiments.md @@ -36,46 +36,68 @@ To verify the installation and start the MLflow UI server locally: ```bash uv run mlflow doctor -uv run mlflow server +uv run mlflow server --backend-store-uri=sqlite:///mlflow.db --artifacts-destination=./mlruns ``` -!!! note "MLflow 3 and the local file store" - MLflow 3 puts the filesystem store (the default `./mlruns` directory) in maintenance mode and refuses it unless you explicitly opt in. For local development, set `MLFLOW_ALLOW_FILE_STORE=true` in your environment (the MLOps Python Package sets it in `mise.toml`, so `mise run mlflow:serve` just works). For anything beyond local experimentation, prefer a **database backend** such as `--backend-store-uri sqlite:///mlflow.db`. +In the MLOps Python Package, that command is wrapped as a task, so you never have to retype the flags: -For a more permanent setup using Docker, you can use a `docker-compose.yml` file to launch the MLflow server: +```bash +mise run mlflow:serve +``` + +For a more permanent setup using Docker, you can use a `docker-compose.yml` file to launch the MLflow server on the same store: ```yaml services: mlflow: - image: ghcr.io/mlflow/mlflow:v3.14.0 + image: ghcr.io/mlflow/mlflow:v3.15.1 ports: - 5000:5000 - environment: - # MLflow 3 puts the filesystem store in maintenance mode; opt in for local development. - - MLFLOW_ALLOW_FILE_STORE=true - command: mlflow server --host 0.0.0.0 --port 5000 + volumes: + - ./mlflow.db:/mlflow/mlflow.db + - ./mlruns:/mlflow/mlruns + working_dir: /mlflow + # SQLAlchemy backend, the same store the package writes to locally. + command: mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:///mlflow.db --artifacts-destination ./mlruns ``` Run `docker compose up` to start the service. For information on production-grade deployments, refer to the [MLflow documentation](https://mlflow.org/docs/latest/tracking.html#set-up-the-mlflow-tracking-environment). +## Where should MLflow store your experiments? + +MLflow splits storage into two halves, and they do not have to live in the same place: + +- **The backend store** holds the *metadata*: experiments, runs, parameters, metrics, tags, and every entry in the model registry. It can be a local directory (the *file store*) or any SQLAlchemy-compatible database (SQLite, PostgreSQL, MySQL). +- **The artifact store** holds the *files*: serialized models, plots, data samples, and anything else you log as an artifact. It is a filesystem path or an object store such as S3 or GCS. + +This course uses **SQLite for the backend store and the local `./mlruns` directory for artifacts**: + +```bash +sqlite:///mlflow.db # metadata: experiments, runs, metrics, registered models +./mlruns # artifact files: models, plots, datasets +``` + +Three reasons make this the right default, even on a laptop: + +- **It is the shape you will run in production.** A production MLflow deployment uses a SQLAlchemy backend, almost always PostgreSQL. Developing against SQLite means the same store type, the same schema, and the same query semantics. Moving up to PostgreSQL becomes a change of connection string (`MLFLOW_TRACKING_URI`), not a rewrite of how your code talks to MLflow. +- **The model registry needs a database.** The registry is designed around a relational store; the file store was never built to back it. You will see this the moment you try to register a model, assign an alias, or search versions. The [next section](./5.6. Model Registries.md) builds directly on top of this choice. +- **You are agreeing with upstream, not working around it.** Since MLflow 3.14, `sqlite:///mlflow.db` is MLflow's *own* default tracking URI (`DEFAULT_TRACKING_URI` in `mlflow/store/tracking/__init__.py`). MLflow keeps a backward-compatibility check that falls back to `./mlruns` only when it detects an existing file store on disk. Setting the URI explicitly documents the intent and removes the ambiguity; it does not fight the library. + +!!! warning "The file store is a legacy path" + Earlier versions of this course configured `./mlruns` as the backend store. If you have an existing `./mlruns` directory holding file-store runs, MLflow will silently keep using it instead of SQLite. Delete or move that directory when you migrate, and set the URIs explicitly so the behavior no longer depends on what happens to be on disk. + ## How do you configure MLflow in a project? -To integrate MLflow, you first need to configure it to store experiment data. You can start by setting the tracking and registry URIs to a local directory, such as `./mlruns`. Then, define an experiment name to group related runs. +To integrate MLflow, you first need to point it at the backend store. Set the tracking and registry URIs to your SQLite database, then define an experiment name to group related runs. Enabling [MLflow's autologging](https://mlflow.org/docs/latest/tracking/autolog.html) is highly recommended. It automatically captures metrics, parameters, and models from popular ML libraries without requiring explicit logging statements. ```python -import os - import mlflow -# MLflow 3 requires opt-in for the filesystem store during local development. -# In production, prefer a database backend such as "sqlite:///mlflow.db". -os.environ.setdefault("MLFLOW_ALLOW_FILE_STORE", "true") - -# Configure MLflow to save data to a local directory -mlflow.set_tracking_uri("./mlruns") -mlflow.set_registry_uri("./mlruns") +# Metadata goes to the SQLite database; artifact files land under ./mlruns +mlflow.set_tracking_uri("sqlite:///mlflow.db") +mlflow.set_registry_uri("sqlite:///mlflow.db") # Set a name for the experiment mlflow.set_experiment(experiment_name="Bike Sharing Demand Prediction") @@ -84,6 +106,23 @@ mlflow.set_experiment(experiment_name="Bike Sharing Demand Prediction") mlflow.autolog() ``` +In the MLOps Python Package these URIs are fields of a configuration object rather than loose calls, so every job reads the same defaults and any of them can be overridden from a YAML file or an environment variable: + +```python +class MlflowService(Service): + """Service for Mlflow tracking and registry.""" + + # SQLAlchemy backends are the supported store in MLflow 3: SQLite gives the local + # setup the same shape as a production database (Postgres, MySQL) with no server to + # run, and it is the only local store the model registry is actually designed for. + tracking_uri: str = "sqlite:///mlflow.db" + registry_uri: str = "sqlite:///mlflow.db" + experiment_name: str = "bikes" + registry_name: str = "bikes" +``` + +Remember to add both `mlflow.db` and `mlruns/` to your `.gitignore`: experiment metadata and model artifacts are outputs, not source code. + To start a new run, wrap your training code within an [MLflow run context](https://mlflow.org/docs/latest/tracking.html#tracking-runs). This allows you to add descriptive metadata and enable system metric logging. ```python @@ -114,7 +153,7 @@ Comparing experiments is essential for model selection. MLflow provides two powe The MLflow UI offers an intuitive, visual way to compare runs. -1. **Launch the MLflow Server**: If it's not running, start it with `uv run mlflow server`. +1. **Launch the MLflow Server**: If it's not running, start it with `mise run mlflow:serve`. 2. **Select Runs**: Navigate to the experiment page, where all runs are listed. Use the checkboxes to select the runs you want to compare. 3. **Click Compare**: A "Compare" button will appear. Clicking it opens a detailed view that places the selected runs side-by-side. 4. **Analyze Results**: This view provides a comprehensive summary of parameters, metrics, and artifacts for each run. You can use it to identify which configurations yielded the best performance. @@ -180,6 +219,53 @@ To maximize the value of your experiments, adopt these practices: ``` - **Register Promising Models**: When a run produces a high-quality model, log it to the [MLflow Model Registry](https://mlflow.org/docs/latest/model-registry.html) to version it and prepare it for deployment. +## How do you test code that tracks experiments? + +A real backend store is not free, and your test suite is where you feel it. Every time MLflow opens a SQLite database that does not exist yet, it runs its [Alembic](https://alembic.sqlalchemy.org/) schema migrations before the first row is written. That costs a few seconds — around seven in the MLOps Python Package — and it happens *per database*. + +The naive fixture creates a fresh database for every test, so the suite pays that price on every single test. When the reference package moved from the file store to SQLite with a per-test database, its suite went from **34 seconds to 339 seconds**: a tenfold regression caused entirely by re-running the same migrations hundreds of times. + +The fix keeps the isolation and drops the cost. Build **one** migrated database in a session-scoped fixture, then **copy the file** for each test. A file copy takes about 0.07 seconds instead of seven, and each test still gets a private, empty store it can write to freely. With this fixture, the reference suite settles at **63 seconds** — the honest price of testing against the store you actually ship. + +```python +@pytest.fixture(scope="session") +def mlflow_db_template(tmp_path_factory: pytest.TempPathFactory) -> str: + """Return a migrated but empty MLflow database used as a template by every test. + + Creating an MLflow SQLite store runs its Alembic migrations, which costs seconds. + Paying that once per session and copying the file per test keeps the isolation of a + fresh database at the cost of a file copy. + """ + path = tmp_path_factory.mktemp("mlflow") / "template.db" + services.MlflowService( + tracking_uri=f"sqlite:///{path}", + registry_uri=f"sqlite:///{path}", + experiment_name="Experiment-Template", + registry_name="Registry-Template", + ).start() + return str(path) + + +@pytest.fixture(scope="function", autouse=True) +def mlflow_service(tmp_path: str, mlflow_db_template: str) -> T.Generator[services.MlflowService]: + """Return and start the mlflow service.""" + # Each test gets its own SQLite file under tmp_path, so runs stay isolated the same + # way the old per-test directories were, with the store the package actually ships. + database = os.path.join(tmp_path, "mlflow.db") + shutil.copyfile(mlflow_db_template, database) + service = services.MlflowService( + tracking_uri=f"sqlite:///{database}", + registry_uri=f"sqlite:///{database}", + experiment_name="Experiment-Testing", + registry_name="Registry-Testing", + ) + service.start() + yield service + service.stop() +``` + +The general lesson outlives MLflow: when a test fixture is slow, look for the expensive, *deterministic* setup step hiding inside it and pay for it once. A migrated schema, a compiled model, or a seeded dataset can almost always be built at session scope and cloned per test. + ## How does experiment tracking fit into the MLOps lifecycle? Experiment tracking is a cornerstone of the MLOps lifecycle, bridging the gap between development and production. diff --git a/docs/5. Refining/5.6. Model Registries.md b/docs/5. Refining/5.6. Model Registries.md index fad910a..f4cb843 100644 --- a/docs/5. Refining/5.6. Model Registries.md +++ b/docs/5. Refining/5.6. Model Registries.md @@ -33,13 +33,18 @@ To begin with MLflow, install it in your project: uv add mlflow ``` -Then, verify the installation and start the tracking server: +Then, verify the installation and start the tracking server on a database backend: ```bash uv run mlflow doctor -uv run mlflow server +uv run mlflow server --backend-store-uri=sqlite:///mlflow.db --artifacts-destination=./mlruns ``` +In the MLOps Python Package, that command is wrapped as `mise run mlflow:serve`. + +!!! warning "A registry needs a database, not a directory" + Unlike plain experiment tracking, the MLflow Model Registry is **designed around a relational store**. Registered models, versions, aliases, and tags are rows with foreign keys between them, and the file store was never built to hold that. Point your backend store at a SQLAlchemy URI — `sqlite:///mlflow.db` locally, PostgreSQL in production — before you register anything. Since MLflow 3.14, SQLite is already MLflow's own default tracking URI, so this is agreeing with upstream rather than working around it. + ## What is the difference between an MLflow model and a registered model? An [MLflow Model](https://mlflow.org/docs/latest/models.html) is the output of a training run, logged during an [MLflow experiment](https://mlflow.org/docs/latest/tracking.html) using a command like `mlflow.sklearn.log_model()`. Think of it as a saved artifact. @@ -52,21 +57,16 @@ Integrating the MLflow Model Registry involves four main steps: initializing, sa ### 1. Initializing -First, configure MLflow to know where to store its data. For local development, you can point both the tracking and registry URIs to a local directory. MLflow 3 puts the filesystem store in maintenance mode, so opt in with `MLFLOW_ALLOW_FILE_STORE=true`; in production, prefer a database backend such as `sqlite:///mlflow.db`. +First, configure MLflow to know where to store its data. Point both the tracking and the registry URI at the same SQLite database: the metadata (runs, registered models, versions, aliases) lives in `mlflow.db`, while the artifact files land on disk under `./mlruns`. ```python -import os - import mlflow -# MLflow 3 requires opt-in for the filesystem store during local development. -os.environ.setdefault("MLFLOW_ALLOW_FILE_STORE", "true") - -# Set the location for MLflow to store experiment runs and artifacts -mlflow.set_tracking_uri("./mlruns") +# Set the location for MLflow to record experiment runs +mlflow.set_tracking_uri("sqlite:///mlflow.db") -# Set the location for the model registry -mlflow.set_registry_uri("./mlruns") +# Set the location for the model registry (the same database) +mlflow.set_registry_uri("sqlite:///mlflow.db") # Create a registered model name (only needs to be done once) client = mlflow.tracking.MlflowClient() @@ -76,6 +76,8 @@ except mlflow.exceptions.MlflowException: pass # Model already exists ``` +Keeping the tracking and registry URIs identical is deliberate: a registered model version points back to the run that produced it, so splitting the two stores breaks the lineage link the registry exists to preserve. When you move to a shared environment, both URIs become the same PostgreSQL connection string, and nothing else in the code changes. + ### 2. Saving Next, log your model during a training run. You can do this manually or use [autologging](https://mlflow.org/docs/latest/tracking/autolog.html) for convenience. @@ -238,6 +240,23 @@ mlflow.pyfunc.save_model( ) ``` +## How do you move the registry from your laptop to your team? + +A local SQLite registry is a rehearsal for the real thing, not a different thing. Promoting it is mostly a matter of swapping URIs: + +- **Backend store**: replace `sqlite:///mlflow.db` with a managed PostgreSQL or MySQL connection string. Both are SQLAlchemy backends, so the schema, the migrations, and your code stay the same. +- **Artifact store**: replace the local `./mlruns` directory with an object store (`s3://`, `gs://`, `azure://`) so every teammate and every CI runner can read the same model files. +- **Access**: run `mlflow server` in front of both, and set `MLFLOW_TRACKING_URI` to its HTTP address. Your jobs stop talking to a file and start talking to a service, without touching a single `mlflow.*` call. + +Because the URIs are configuration rather than hardcoded strings, this is an environment change, not a code change: + +```bash +export MLFLOW_TRACKING_URI="https://mlflow.internal.example.com" +``` + +!!! note "Testing against a real backend" + A database-backed registry costs more in your test suite than a directory did, because MLflow runs its schema migrations the first time it opens a new database. Do not answer that by mocking the registry away — you would stop testing the part most likely to break. Build one migrated database per test session and copy the file per test instead; the technique, and the measured numbers behind it, are covered in [5.5. AI/ML Experiments](./5.5. AI-ML Experiments.md#how-do-you-test-code-that-tracks-experiments). + ## Additional Resources - **[Model Registry integration from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/src/bikes/io/registries.py)** diff --git a/docs/5. Refining/index.md b/docs/5. Refining/index.md index d74d361..1d4ee89 100644 --- a/docs/5. Refining/index.md +++ b/docs/5. Refining/index.md @@ -7,9 +7,9 @@ description: This chapter covers advanced techniques to refine your MLOps workfl Refining your MLOps practices is crucial for moving from experimental models to production-grade systems. This chapter focuses on enhancing the efficiency, reliability, and scalability of your projects. By mastering these techniques, you will streamline the entire development-to-deployment pipeline, improve code quality, automate repetitive work, and ensure consistency across all environments. - **[5.0. Design Patterns](./5.0. Design Patterns.md)**: Master architectural blueprints to solve common MLOps challenges, creating scalable and maintainable systems. -- **[5.1. Task Automation](./5.1. Task Automation.md)**: Automate routine development tasks to boost efficiency and minimize human error. -- **[5.2. Pre-Commit Hooks](./5.2. Pre-Commit Hooks.md)**: Enforce code quality standards automatically before commits, ensuring a clean and stable codebase. -- **[5.3. CI/CD Workflows](./5.3. CI-CD Workflows.md)**: Implement CI/CD pipelines to automate model testing and deployment for rapid, reliable delivery. -- **[5.4. Software Containers](./5.4. Software Containers.md)**: Use containers to create consistent, portable environments for development, testing, and deployment. +- **[5.1. Task Automation](./5.1. Task Automation.md)**: Define one canonical task vocabulary with `mise`—including the `all` gate and a locked toolchain—so your terminal, hooks, and CI can never disagree. +- **[5.2. Pre-Commit Hooks](./5.2. Pre-Commit Hooks.md)**: Enforce code quality standards automatically before commits, ordering thin `lefthook` commands so formatting, secret scanning, and checks each run at the right moment. +- **[5.3. CI/CD Workflows](./5.3. CI-CD Workflows.md)**: Implement hardened CI/CD pipelines that call the same gate you run locally, and lint the workflows themselves with `actionlint` and `zizmor`. +- **[5.4. Software Containers](./5.4. Software Containers.md)**: Use containers to create consistent, portable environments, with a linted, pinned, non-root `Dockerfile`. - **[5.5. AI/ML Experiments](./5.5. AI-ML Experiments.md)**: Effectively manage, track, and reproduce experiments to accelerate model development and innovation. - **[5.6. Model Registries](./5.6. Model Registries.md)**: Leverage model registries to version, share, and manage your machine learning models systematically. \ No newline at end of file diff --git a/docs/6. Sharing/6.2. Readme.md b/docs/6. Sharing/6.2. Readme.md index b3f1175..485c1a1 100644 --- a/docs/6. Sharing/6.2. Readme.md +++ b/docs/6. Sharing/6.2. Readme.md @@ -1,5 +1,5 @@ --- -description: Craft an effective README.md file for your repository that provides a clear overview of your project, its purpose, features, and instructions for use. Learn how to make your README engaging and informative for potential users and contributors. +description: Craft an effective README.md for your human readers and an AGENTS.md for the AI coding agents working in your repository. Learn what belongs in each file, why the split matters, and how to keep both from drifting away from the code. --- # 6.2. Readme @@ -68,6 +68,175 @@ Popular tools for creating documentation sites include: - **Sphinx:** A powerful tool that can generate documentation in various formats. - **GitHub Pages:** A platform to host your documentation site directly from your repository. +## Who else reads your repository? + +Your README is written for a person: it sells the project, sets expectations, and walks a newcomer to their first successful command. But a growing share of the traffic through your repository is not human. AI coding agents open your project, look for orientation, and start editing. + +They need different things than a person does. A human skims the README, then learns the conventions by reading code and by getting review comments. An agent has no review history, no colleague to ask, and a strong tendency to reach for whatever is most common on the internet rather than what is correct *here*. Left without guidance, it will invent a plausible command, pick the wrong tool, and produce a change that fails your gate. + +The answer is a second front door: **`AGENTS.md`**. + +## What is `AGENTS.md`? + +[`AGENTS.md`](https://agents.md/) is an open format for the instructions an AI coding agent needs to work in your repository. It is a plain Markdown file at the root of your project. There is no schema and no required section list: the format's only real convention is the filename and the location. + +It matters because it is **shared**. The format emerged from a collaboration across several AI development companies (including OpenAI, Google, Cursor, and Factory) and is now stewarded by the **Agentic AI Foundation under the Linux Foundation**. A broad and growing set of tools reads it — OpenAI Codex, Cursor, GitHub Copilot's coding agent, goose, Gemini CLI, Zed, Aider, Jules, Devin, Windsurf, and many others. + +Before it existed, every tool invented its own file: `.cursorrules`, `.github/copilot-instructions.md`, `CLAUDE.md`, `.aider.conf.yml`, and a dozen more. Repositories accumulated near-duplicate instruction files that were all subtly different because nobody could keep them in sync. One `AGENTS.md` replaces that pile. Tools that still want their own path can symlink to it. + +All four repositories behind this course ship one: the [course](https://github.com/MLOps-Courses/mlops-coding-course), the [MLOps Python Package](https://github.com/fmind/mlops-python-package), the [Cookiecutter MLOps Package](https://github.com/fmind/cookiecutter-mlops-package), and the [MLOps Coding Skills](https://github.com/MLOps-Courses/mlops-coding-skills). + +## Why not simply put this in the README? + +Because the two documents have different readers and different failure modes, and mixing them degrades both. + +- **The README sells and onboards.** It has to be readable, ordered by what a newcomer needs first, and tolerant of prose. Bury your commit-message convention and your typing rules in the middle of it and a human will scroll past them. +- **`AGENTS.md` instructs.** It has to be exact, exhaustive on the points that matter, and utterly boring. Every sentence exists to prevent one specific class of wrong output. + +So keep the split clean: + +| Belongs in `README.md` | Belongs in `AGENTS.md` | +| ---------------------- | ---------------------- | +| What the project is and who it is for | The exact task vocabulary and what each task runs | +| Installation and first-run instructions | The definition of done a change must satisfy | +| Usage examples and configuration | Conventions and idioms an agent would otherwise infer wrongly | +| Badges, screenshots, license, credits | A map of the repository layout | + +Do not copy installation instructions into `AGENTS.md`, and do not move your conventions out of the README into it and nowhere else — a human contributor still needs to find them. Link between the two instead. The MLOps Python Package puts one line in its README: + +```markdown +> **AI agents**: read [`AGENTS.md`](AGENTS.md) for the full stack, task vocabulary, and conventions before contributing. +``` + +and one line at the top of `AGENTS.md`: + +```markdown +Context and rules for AI agents working in this repository. Humans should start with `README.md`. +``` + +## What belongs in an `AGENTS.md`? + +Four things earn their place. Everything else is optional. + +### 1. The exact commands + +Not a description of your workflow — the literal command line. This is the single highest-value content in the file, because a guessed command is the most common way an agent wastes a cycle. + +```markdown +- Everything: `mise run all` — format, check, test, build. This is the gate; CI runs this exact task and nothing else. +- Check: `mise run check` — `ruff` lint, `ty` types, `pip-audit` deps, `dprint`/`validate-pyproject`/`uv lock` format, `gitleaks` secrets, `trivy` filesystem scan, `hadolint` Dockerfile, `actionlint` + `zizmor` workflows. +- Test: `mise run test` — `pytest` with coverage (fails under 100%). +``` + +Note what that does: it names one canonical task, states that CI runs *that exact task*, and explains what each one covers. An agent now knows both what to run and why running something else is not equivalent. + +### 2. A definition of done + +An agent will stop the moment it believes it has finished. Tell it what finished means, and close the escape hatches explicitly: + +```markdown +A change is complete only when, locally, `mise run format` is clean, `mise run check` reports no findings, and `mise run test` is green with new/changed behavior covered by a test. Fix root causes — never weaken an assertion, add a skip/`xfail`, loosen a type, or suppress a lint error to force a green result. +``` + +The second sentence is the important one. Every way of faking a green result is cheaper than fixing the cause, so name them. + +### 3. Conventions and idioms + +These are the decisions an agent cannot recover from the code alone, or would recover *wrongly* because the internet's default differs from yours: + +```markdown +- **Errors with context**: raise specific exceptions and chain with `raise ... from err`; never use a bare `except`. +- **Config over hardcoding**: jobs and objects are Pydantic models parsed from OmegaConf YAML in `confs/`. +- **Typing**: modern annotations (`list[str]`, `X | Y`); keep `ty check` clean. `import typing as T` is the project convention. +- **MLflow**: tracking and registry run on a SQLite backend (`sqlite:///mlflow.db`); artifact files stay on disk under `./mlruns`. +- **Commits**: Conventional Commits (`feat:`, `fix:`, `refactor:`, `chore:`); no attribution in commit messages. +``` + +State the rule *and* the reason where the reason is not obvious. An agent that understands why a rule exists applies it correctly to cases you did not enumerate. + +### 4. A layout map + +A short tour of the directory structure saves an agent from reading the whole tree to find where a change belongs: + +```markdown +- `src/bikes/` — package: `core/` (metrics, models, schemas), `io/` (configs, datasets, registries, services), `jobs/` (tuning, training, promotion, inference), `utils/`. +- `confs/` — one OmegaConf YAML per MLflow job; `tests/` — `pytest` suite mirroring `src/`. +- `.github/` — `workflows/`, `dependabot.yml`, `zizmor.yml`, `rulesets/main.json`. +``` + +## What does a complete `AGENTS.md` look like? + +The [MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/AGENTS.md) uses five sections and fits on roughly one screen and a half: + +```markdown +# AGENTS.md + +Context and rules for AI agents working in this repository. Humans should start with `README.md`. + +## Project overview + + +## Setup & core commands + + +## Definition of done + + +## Conventions & idioms + + +## Repository layout + +``` + +Length is a real constraint: this file is read on every task, so it competes for the agent's attention with the code it is about to change. Aim for the shortest file that answers the questions an agent actually gets wrong. If a section is never violated, delete it. + +## What `AGENTS.md` cannot do + +Here is the boundary that keeps this honest: + +> **`AGENTS.md` guides. It does not enforce.** + +Nothing executes prose. An agent may misread it, ignore it, run out of context before reaching the relevant line, or simply be a different agent that never loaded it. A rule written only in `AGENTS.md` is a rule that holds most of the time — which, for anything that matters, is not enough. + +Enforcement lives where it always has: + +- **Types** make an invalid state unrepresentable (`ty`, Pydantic, Pandera). +- **Tests** fail when behavior regresses. +- **Linters and formatters** reject the style before it is committed (`ruff`, `dprint`). +- **Git hooks and CI** run those gates whether anyone remembers to or not. +- **Branch protection** stops unreviewed work from reaching `main`. +- **Human review** catches what none of the above can express. + +Understood this way, `AGENTS.md` has a modest and useful job: **make the first attempt land close**, so your gates have less to catch and your reviewers spend their attention on design instead of on conventions. It is a productivity tool, not a control. If you find yourself writing "the agent must never..." for something genuinely dangerous, stop writing and go add a check. + +## How do you keep guidance files from drifting? + +Guidance files rot faster than code, because nothing fails when they are wrong. + +The MLOps Python Package proved it the expensive way. Alongside its `AGENTS.md`, it also carried a **vendored copy** of the seven Agent Skills under `.gemini/skills/`, duplicated from the [MLOps Coding Skills](https://github.com/MLOps-Courses/mlops-coding-skills) repository they came from. Within a month, the copies had diverged from their sources by well over a hundred lines each — the worst by more than 170. By then those files were confidently instructing agents to set up task automation with `just` and git hooks with `pre-commit`: precisely the stack that release of the package had removed, in favor of `mise` and `lefthook`. Every agent that read them started from a stale picture of the project and had to be corrected by hand. + +The copies were deleted, and the package's `AGENTS.md` now points at the source instead: + +```markdown +- **Skills**: the reusable practices behind this package are published as Agent Skills in [mlops-coding-skills](https://github.com/MLOps-Courses/mlops-coding-skills). Install them from there rather than vendoring a copy here — a copy drifts, and this repository already lost a month to proving it. +``` + +Four practices keep guidance honest: + +- **Keep exactly one copy** of any guidance file, and reference it from everywhere else. If a tool needs the content at its own path, use a symlink, never a duplicate. +- **Ground every statement in something real.** Every command, path, flag, and version in your `AGENTS.md` must exist in the repository. A plausible-sounding instruction that does not work is worse than no instruction, because the agent will trust it. +- **Update it in the same commit that changes the tooling.** If a pull request renames a task, it is incomplete until `AGENTS.md` says the new name. +- **Treat a correction as a bug report.** Every time you tell an agent something it should have known, that is a line missing from `AGENTS.md` — or a line already there that is wrong. + +## What are Agent Skills, and how do they relate? + +`AGENTS.md` covers *this repository*: its commands, its conventions, its layout. It is not the place for a reusable methodology, which would bloat the file and be copy-pasted into the next project anyway. + +That is what **[Agent Skills](https://agentskills.io/home)** are for. A skill is a folder containing a `SKILL.md` — YAML frontmatter plus Markdown instructions — that an agent loads *on demand* when a task matches its description. Because the agent only pulls a skill in when it is relevant, skills can be long and detailed without competing for attention on every task. + +The practices taught in this course are published this way, one skill per chapter, in **[MLOps Coding Skills](https://github.com/MLOps-Courses/mlops-coding-skills)**: `mlops-initialization`, `mlops-prototyping`, `mlops-industrialization`, `mlops-validation`, `mlops-automation`, `mlops-collaboration`, and `mlops-observability`. Install them from that repository — symlinked into `.agents/skills/`, or through your tool's own installer — and let each project's `AGENTS.md` stay short and specific to that project. + ## Where can you find inspiration and tools? To create a high-quality README, leverage these excellent resources and tools: @@ -79,6 +248,13 @@ To create a high-quality README, leverage these excellent resources and tools: - **[Awesome README](https://github.com/matiassingers/awesome-readme):** A curated list of inspiring README files. - **[Standard README Template](https://github.com/RichardLitt/standard-readme/blob/main/spec.md):** A specification for a standard README layout. +**AGENTS.md and Agent Skills:** + +- **[agents.md](https://agents.md/):** The open format, its governance, and the list of tools that read it. +- **[AGENTS.md from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/AGENTS.md):** The complete example dissected above. +- **[MLOps Coding Skills](https://github.com/MLOps-Courses/mlops-coding-skills):** The seven Agent Skills that accompany this course, one per chapter. +- **[Agent Skills](https://agentskills.io/home):** What a `SKILL.md` is and how agents discover one. + **VS Code Extensions for Markdown:** - **[Markdown All in One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one):** Provides shortcuts, a table of contents generator, and live previews. diff --git a/docs/6. Sharing/6.3. Releases.md b/docs/6. Sharing/6.3. Releases.md index 5988786..917b85f 100644 --- a/docs/6. Sharing/6.3. Releases.md +++ b/docs/6. Sharing/6.3. Releases.md @@ -170,7 +170,7 @@ Because Conventional Commits carry the semantics of each change (`feat` bumps th Once your `main` branch is green, releasing is a short, repeatable sequence. The [MLOps Python Package](https://github.com/fmind/mlops-python-package) follows these steps: -1. **Validate**: Start from a clean working tree on `main` and run the full suite with `mise run check` and `mise run test`. +1. **Validate**: Start from a clean working tree on `main` and run the canonical gate, `mise run all` (format, check, test, build) — the same task CI runs, see [5.1. Task Automation](../5. Refining/5.1. Task Automation.md). 1. **Compute the version**: Run `git-cliff --bumped-version` to get the next tag (e.g., `v1.2.3`). 1. **Bump the manifest**: Update `version` in `pyproject.toml` to match (skip this if you use dynamic, tag-based versioning). 1. **Generate the changelog**: Run `git-cliff --bump -o CHANGELOG.md`. @@ -197,7 +197,7 @@ on: types: [published] jobs: pages: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v7 - uses: jdx/mise-action@v4 diff --git a/docs/6. Sharing/6.4. Templates.md b/docs/6. Sharing/6.4. Templates.md index 3e0d0a7..22233f5 100644 --- a/docs/6. Sharing/6.4. Templates.md +++ b/docs/6. Sharing/6.4. Templates.md @@ -68,32 +68,69 @@ project_name = "{{ cookiecutter.project_name }}" **Example `cookiecutter.json` file:** -This file defines the template's variables and their default values. You can even use variables to define other variables. +This file defines the template's variables and their default values. Here is the real one from the [Cookiecutter MLOps Package](https://github.com/fmind/cookiecutter-mlops-package): ```json { - "user": "fmind", - "name": "MLOps Project", - "repository": "{{cookiecutter.name.lower().replace(' ', '-')}}", - "package": "{{cookiecutter.repository.replace('-', '_')}}", - "license": "MIT", - "version": "0.1.0", - "description": "A new MLOps project.", - "python_version": "3.14", - "mlflow_version": "3.14.0" + "user": "fmind", + "name": "MLOps Project", + "repository": "{{cookiecutter.name.lower().replace(' ', '-')}}", + "package": "{{cookiecutter.repository.replace('-', '_')}}", + "version": "0.1.0", + "year": "2026", + "description": "A short description of the project.", + "python_version": "3.14", + "mlflow_version": "3.15.1", + "_copy_without_render": ["cliff.toml"], + "__prompts__": { + "user": "GitHub User", + "name": "Project Name", + "repository": "GitHub Repository", + "package": "Python Package", + "version": "Project Version", + "year": "Copyright Year", + "description": "Project Description", + "python_version": "Python Version", + "mlflow_version": "MLflow Version" + } } ``` +Three details in that file are worth studying: + +- **Derived variables**: `repository` and `package` are computed from `name` with Jinja expressions, so the user answers one question and the template fills three consistent values. Fewer prompts means fewer chances to answer inconsistently. +- **`_copy_without_render`**: some files legitimately contain `{{ ... }}` that is *not* a cookiecutter variable. `cliff.toml` is a [git-cliff](https://git-cliff.org/) configuration full of Tera templating; rendering it through Jinja would mangle it or fail outright. Listing it here copies the file verbatim. Any private key (a name starting with `_`) is excluded from prompting but kept in the rendered context. +- **`__prompts__`**: this maps each variable to the human-readable question shown on the command line, so the user sees `Copyright Year` instead of the bare identifier `year`. It is pure ergonomics, and it costs one line per variable. + +### Every prompt must have a consumer + +The template used to ask for a `license` and no longer does; it now asks for a `year` instead. That swap encodes a rule worth stealing: + +> **A prompt that nothing consumes is worse than no prompt at all.** + +The old `license` variable was collected on every single generation and then referenced by exactly nothing — the generated `LICENSE.txt` was a fixed MIT text. Users answered a question that changed no output, which quietly teaches them that the answers do not matter. Meanwhile the copyright line in `LICENSE.txt` had no year at all, because no variable supplied one. + +So when you add or review a variable, grep for it: + +```bash +# Every prompt in cookiecutter.json must appear somewhere under the template directory +grep -r "cookiecutter.year" "{{cookiecutter.repository}}/" +``` + +If it returns nothing, either wire the variable up or delete the prompt. + ## How should you structure a Cookiecutter template? A well-structured [Cookiecutter template repository](https://github.com/fmind/cookiecutter-mlops-package) has two main components: 1. **The Template Directory:** A single directory whose name contains a variable, like `{{cookiecutter.repository}}`. Everything inside this directory—files, subdirectories, and their content—will be rendered into the new project. -2. **Configuration and Hooks:** Files that control the generation process but are not part of the final project. These include: - - `cookiecutter.json`: Defines the variables for the template. - - `hooks/`: A directory for scripts that run before or after generation. +1. **The Harness:** Everything at the repository root that controls or validates generation but never ships to the generated project. In the [cookiecutter-mlops-package template](https://github.com/fmind/cookiecutter-mlops-package) this includes: + - `cookiecutter.json`: Defines the variables, their defaults, and their prompts. + - `tests/`: A `pytest-cookies` suite that bakes the template and runs the generated project's own gate. + - `mise.toml`, `lefthook.yml`, `dprint.jsonc`, `trivy.yaml`: The harness's own tooling, which mirrors the tooling it generates. + - `hooks/`: Optional Python scripts that run before or after generation. This template does not need them; reach for a hook only when a value cannot be expressed as a variable. -For a complete, real-world example, explore the [cookiecutter-mlops-package template](https://github.com/fmind/cookiecutter-mlops-package) created by this course's authors. +A template repository therefore has **two layers**, and both need maintaining: the project you generate, and the harness that generates it. Keeping their configuration files identical apart from the cookiecutter variables is the cheapest way to stop the two from drifting apart. **Initialize this template package:** @@ -153,36 +190,65 @@ This feedback loop ensures that your template remains practical, robust, and ali ## How can you automatically test a code template? -Automated testing is critical to ensure a template doesn't break as it evolves. With [pytest-cookies](https://github.com/hackebrot/pytest-cookies), you can write tests that automatically generate a project and verify the output. +Automated testing is critical to ensure a template doesn't break as it evolves. With [pytest-cookies](https://github.com/hackebrot/pytest-cookies), you can write a test that generates a project and verifies the output. ```python # Test that the project generates successfully def test_bake_project(cookies): - result = cookies.bake(extra_context={"project_name": "helloworld"}) + result = cookies.bake(extra_context={"name": "MLOps 123"}) assert result.exit_code == 0 assert result.exception is None - assert result.project_path.name == "helloworld" + assert result.project_path.name == "mlops-123" assert result.project_path.is_dir() ``` -You can also use a library like [pytest-shell-utilities](https://github.com/saltstack/pytest-shell-utilities) to run shell commands and validate that setup tasks in the generated project work as expected. +Generating without an exception only proves that Jinja rendered. What you actually want to know is whether the *generated project works*, so pair `pytest-cookies` with [pytest-shell-utilities](https://github.com/saltstack/pytest-shell-utilities) and run the generated project's own gate inside it: ```python -def test_assert_good_exitcode(shell): - ret = shell.run("exit", "0") - assert ret.returncode == 0 +COMMANDS = [ + "mise trust -y", + "mise install -y", + "git init", + "mise run clean", + "mise run install", + # The generated project's own gate: format, check, test, and build in one task. + "mise run all", + "mise run docs", + "mise run project", + "mise run build:image", + "mise run mlflow:doctor", +] + +shell = Subprocess(cwd=result.project_path) +for command in COMMANDS: + result = shell.run(*command.split()) + assert result.returncode == 0, f"Command failed: {command}" +``` + +Two failure modes made this list what it is, and both are easy to reproduce in your own template: -def test_assert_bad_exitcode(shell): - ret = shell.run("exit", "1") - assert ret.returncode == 1 +- **A gate that skips the expensive task proves nothing.** The bake test used to run every task *except* `mise run test`, so the template's own suite and coverage threshold had never actually executed in CI. Run the single canonical task (`mise run all`) rather than a hand-picked subset, and you cannot forget one. +- **The outer shell's `PATH` leaks into the subprocess.** The generated project sets `run_auto_install = false`, so without an explicit `mise install -y` the tool-dependent checks silently reuse whatever binaries the developer's own machine happens to have. It looks green locally and fails on a clean runner. Install the toolchain inside the generated project, explicitly. + +### Test the contract, not just the code + +Some template bugs are invisible to any test that only runs commands. The MLOps template once shipped a CI workflow whose job was named `check`, alongside a branch ruleset that required the status-check context `checks`. Both files were individually valid, the generated project's tests all passed—and every pull request in every generated project was blocked forever on a check that no workflow could ever report. + +The lesson generalizes: whenever two generated files refer to each other by a **string** (a job name and a required status check, a package name and an entrypoint, a service name and a hostname), that coupling is a contract your template owns. Assert it, or at minimum record it in a comment on both sides, as the template now does: + +```yaml +jobs: + checks: # name kept as "checks" to satisfy the repository's required-status-check ruleset ``` +Also make any setup task the template tells users to run **idempotent**. `mise run install:rulesets` originally `POST`ed the ruleset, creating a duplicate on every run; it now looks the ruleset up by name and `PUT`s over it, so re-running is a no-op. + ## How do you run automated tasks after generation? [Cookiecutter hooks](https://cookiecutter.readthedocs.io/en/stable/advanced/hooks.html) are Python or shell scripts that execute automatically before or after project generation. They are perfect for cleanup tasks or conditional logic. -A common use case is removing files that are not needed based on the user's choices during setup. +A common use case is removing files that are not needed based on the user's choices during setup. Reach for a hook only when the behavior cannot be expressed as a variable — the MLOps template needs none. **Example `post_gen_project.py` hook script:**