-
Notifications
You must be signed in to change notification settings - Fork 76
Introduce support for agentic autofix #1136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
mbaluda
wants to merge
18
commits into
main
Choose a base branch
from
mbaluda-autofix
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
4089866
Introduce support for agentic autofix
mbaluda 13c43e2
Update copilot-instructions.md
mbaluda 0d46c8c
Update docs/copilot-instructions.md
mbaluda cd53b5c
Merge branch 'main' into mbaluda-autofix
mbaluda 04c5cab
Improved instructions
mbaluda f494dde
Fix typos and enhance clarity in instructions
mbaluda 69fbf21
Clarify autofix pull request restrictions
mbaluda feb2f18
Refactor Copilot instructions for clarity and consistency
mbaluda 32b4ede
Rename copilot-instructions.md to autofix-instructions.md
mbaluda 2cd06d8
Update autofix instructions for clickable links
mbaluda 5d49de1
Update autofix instructions by removing link guidance
mbaluda 321e3f1
Update autofix-instructions.md
mbaluda acc3850
Update autofix-instructions.md
mbaluda d8cedd9
Update autofix-instructions.md
mbaluda 84c34f9
Update autofix-instructions.md
mbaluda 8089adf
Add compliance tests for division and modulus operations in unsigned …
mbaluda fb7bd2e
Update autofix-instructions.md
mbaluda 1cca8c3
Update autofix-instructions.md
mbaluda File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| - `INT30-C` - `UnsignedIntegerOperationsWrapAround.ql`: | ||
| - Fixed false positives for `/=` and `%=` assignments. | ||
| - `INT30-C` - `UnsignedOperationWithConstantOperandsWraps.ql`: | ||
| - Fixed false positives for `/=` and `%=` assignments. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| [//]: # "Include this file in the repository to provide instructions to GitHub Copilot Autofix. For more information, see https://docs.github.com/copilot/copilot-for-business/copilot-instructions." | ||
|
|
||
|
|
||
| # Agentic autofix instructions for CodeQL Coding Standards | ||
|
|
||
| This document configures **Agentic Autofix** when generating pull requests to | ||
| remediate alerts produced by [CodeQL Coding Standards](https://github.com/github/codeql-coding-standards/). | ||
| It applies to alerts for any of the supported standards (MISRA C, MISRA C++, | ||
| AUTOSAR C++, CERT C, CERT C++). | ||
|
|
||
| ## 1. Reference material — where to learn each rule | ||
|
|
||
| Before proposing a fix, consult the rule’s authoritative implementation as well | ||
| as the corresponding compliant and non-compliant code patterns available as | ||
| test cases in the [`github/codeql-coding-standards` repository](https://github.com/github/codeql-coding-standards/). | ||
| That repository is the single source of truth for what the query | ||
| detects and what compliant code looks like. | ||
|
|
||
| Project layout (per language / standard): | ||
|
|
||
| ``` | ||
| <language>/<standard>/src/rules/<rule-id>/ # query source (.ql) and rule help (.md, .qhelp) | ||
| <language>/<standard>/test/rules/<rule-id>/ # test cases, with COMPLIANT / NON_COMPLIANT markers | ||
| ``` | ||
|
|
||
| When generating a fix: | ||
|
|
||
| 1. Locate the rule directory matching the alert’s rule id / query id. | ||
| 2. Read the `.md` / `.qhelp` help file in `src/rules/<rule-id>/` to understand | ||
| the intent and the recommended remediation. | ||
| 3. Read the files in `test/rules/<rule-id>/`. Lines (or blocks) annotated with | ||
| `// COMPLIANT` show patterns that pass the query; lines annotated with | ||
| `// NON_COMPLIANT` show patterns that trigger the query. Use these as the | ||
| ground truth for what the fixed code must look like. | ||
|
|
||
| The full list of supported rules per standard is published as | ||
| `supported_rules_list_<version>.csv` / `.md` in each | ||
| [release](https://github.com/github/codeql-coding-standards/releases). | ||
|
|
||
| ## 2. Fix discipline — keep changes minimal and standards-compliant | ||
|
|
||
| - **Minimum diff.** Modify the smallest possible amount of code that | ||
| eliminates the alert. Do not refactor surrounding code, rename symbols, | ||
| reformat unrelated lines, or change public APIs unless strictly required to | ||
| satisfy the rule. | ||
| - **Multiple alerts.** Check if there are other code scanning alerts | ||
| for the same coding standard reported for the same code location, fix all | ||
| the relevant alerts at once in the same PR. | ||
| - **No drive-by changes.** Do not add features, change build flags, update | ||
| dependencies, or “improve” code that the alert does not point at. | ||
| - **Do not attempt at fixing design issues.** A fix should not attempt to | ||
| “improve” the design of the code or address architectural issues. | ||
| Always verify that the code section around the alert is intended to follow | ||
| the standard and add a comment. | ||
| The presence of certain design issues (e.g. dynamic memory allocation) might | ||
| indicate that the code is not intended to be compliant with the standard, and | ||
| that a deviation should be added instead of a code fix. | ||
| - **New code must comply with the same standard.** Any code modified by the | ||
| fix must itself satisfy every rule of the coding standard being verified. | ||
| Cross-check the changed code against the COMPLIANT examples in the | ||
| corresponding `test/rules/<rule-id>/` directory and against every other | ||
| relevant rules (e.g. don’t fix an integer-conversion rule by introducing a | ||
| cast that violates a different MISRA rule). | ||
| - **Preserve safe and desired functional behavior.** ensure the resulting code | ||
| handles all reasonable real-world scenarios as the code originally intended. | ||
| This may involve precisely maintaining the existing code behavior, or it may | ||
| involve fixing subtle or rare bugs, for instance dangerous overflows, | ||
| that the existing code does not handle and the rule is designed to detect. | ||
| - **Fix dangerous bugs** If the alert is flagging unsafe or undefined behavior, | ||
| critically examine how that safety issue in the code should be properly fixed. | ||
| Add detections and error handling if necessary to make the code safe under | ||
| all conditions without introducing unnecessary complexity. Follow existing | ||
| project guidelines on how to handle rare, dangerous, or unexpected scenarios | ||
| that occur at runtime. | ||
| - **Thoroughly explain analysis and functional changes.** If the alert does not | ||
| introduce any unwanted behavior and the change is functionally equivalent, | ||
| explain your thinking, and clearly show that the code before was safe, and | ||
| that the new code is exactly equivalent in behavior. | ||
| If there was a dangerous edge case or condition, explain exactly how that | ||
| scenario would create problems in the code and how the fix will prevent such | ||
| issues and improve the safety and quality of the codebase. | ||
|
|
||
| ## 3. Do not add build output folders, generated files, or `.gitignore` | ||
|
|
||
| Autofix pull requests must only change source files that are part of the | ||
| checked-in project. They must **not** include: | ||
| - Build directories or files generated during compilation (`.build/`, etc.). | ||
| - Editor / IDE state (`.vscode/`, `.idea/`, `.DS_Store`, etc.). | ||
| - **`.gitignore` itself.** Do not add, remove, or reorder entries in | ||
| `.gitignore` as part of an autofix. | ||
| - The CodeQL workflow files under `.github/workflows/` (e.g. `codeql.yml`). | ||
| Suppression or scope changes must use the deviation mechanism (see §4), | ||
| not workflow edits. | ||
|
|
||
| ## 4. Deviations — respect project policy and reference it in fixes | ||
|
|
||
| A project may declare that a rule, file, region, or specific construct is | ||
| intentionally exempt from a coding standard. Such deviations are | ||
| not always expressed through the same mechanism: a project may use the | ||
| **standard CodeQL Coding Standards deviation mechanism**, a | ||
| **custom annotation or attribute** convention, | ||
| **in-source line / block comments**, | ||
| or a **separate documentation file** (for example a `DEVIATIONS.md`, | ||
| `MISRA-deviations.md`, `coding-standards.yml`, compliance matrix, or similar). | ||
|
|
||
| Locate the deviations file and explicitly search for matching deviations | ||
| before proposing code changes. | ||
| The fix proposal must take what is found into account and treat it as an | ||
| existing deviation if it clearly covers the alert location and rule. | ||
|
|
||
| If the alert location is covered by an existing deviation: | ||
| - Look for existing deviations of that rule, and see if any should apply | ||
| - In the pull request description, explicitly state that a matching | ||
| deviation already exists in the project, citing the file path and the | ||
| relevant `rule-id` / `query-id` / `permit-id` / `code-identifier` / scope | ||
| (paths or markers) so reviewers can decide whether to accept the fix or | ||
| keep the deviation. | ||
| - Do not silently delete or weaken an existing deviation, permit, or | ||
| re-categorization entry as part of the fix. | ||
| - Propose a code fix that would make the location compliant by | ||
| default. Authors may have left the deviation in place pragmatically and | ||
| may prefer a real fix. | ||
| - Consider whether an existing code identifier should be used | ||
| - Consider whether a file-wide exception should be used | ||
| - Consider whether a new code identifier should be used | ||
| - If using a code-identifier, look for examples to determine whether | ||
| to use [[attribute]] form | ||
| - If using an [[attribute]], look for project formatting configurations or code | ||
| examples to determine how to format the attribute relative to its declaration | ||
| - When using deviation comments, consider project formatting, the specific | ||
| violation in question, and other example deviation comments in the project to | ||
| determine whether to use same-line, next-line, or begin/end comment deviations | ||
| Project formatting configuration may be .clang-format, etc. | ||
|
|
||
| ## 5. False positives — propose a deviation, do not stay silent | ||
|
|
||
| Precedence: if an alert is judged to be a false positive, the false-positive | ||
| workflow in this section overrides any guidance above about proposing a code | ||
| fix when a deviation exists. | ||
|
|
||
| Copilot autofix normally refrains from opening a pull request when it | ||
| considers an alert to be a false positive. For CodeQL Coding Standards alerts | ||
| this behaviour is **overridden**: a false-positive determination must still | ||
| result in a pull request, and that pull request must record the decision | ||
| using the project’s deviation mechanism. | ||
|
|
||
| The CodeQL Coding Standards project supports an explicit deviation mechanism, | ||
| fully documented in | ||
| [`docs/user_manual.md`](https://github.com/github/codeql-coding-standards/blob/main/docs/user_manual.md) | ||
| (sections “Applying deviations”, “Deviation records”, “Deviation permits”, | ||
| “Guideline re-categorization plan”). | ||
| When an alert is judged to be a false positive, the autofix PR must: | ||
|
|
||
| 1. **Not modify the flagged code** to “work around” the alert. | ||
| 2. **Add or update a deviation record** that scopes precisely to the alert. | ||
| Prefer the one narrowest scope that is appropriate in the following order: | ||
| 1. a `code-identifier` deviation referenced from the exact line, statement, | ||
| function, or block, via an attribute | ||
| (`[[codeql::<standard>_deviation("...")]]`) or a comment marker | ||
| (`// codeql::<standard>_deviation(...)`, | ||
| `// codeql::<standard>_deviation_next_line(...)`, or a | ||
| `..._deviation_begin` / `..._deviation_end` pair); or | ||
| 2. a `paths:`-scoped deviation in `coding-standards.yml` when the rule should | ||
| not be applied to a whole file or directory or | ||
| 3. a project-wide deviation only when the rule is genuinely inapplicable to | ||
| the project. | ||
| Use `<standard>` ∈ {`misra`, `autosar`, `cert`} as appropriate for the | ||
| alert. | ||
| 3. **Populate the deviation record** for deviation records with at least: | ||
| - `rule-id` matching the alert’s rule identifier; | ||
| - `query-id` matching the alert’s `@id` (when the deviation is meant to | ||
| cover a single sub-query of the rule); | ||
| - a clear `justification` explaining why the alert is a false positive | ||
| (what the query missed, why the code is in fact compliant or safe); | ||
| - `scope`, `background`, and `requirements` when they help a reviewer | ||
| audit the decision; | ||
| - a `raised-by` entry (and leave `approved-by` for a human reviewer). | ||
| 4. **Place the deviation entry** of types 2. and 3. in an existing | ||
| `coding-standards.yml` if one exists in an appropriate directory; | ||
| otherwise create one at the most specific directory whose subtree is | ||
| affected. When using a `permit-id`, reference an existing permit | ||
| if one matches; do not invent new permit IDs unless necessary. | ||
| 6. **In the PR description**, explicitly state that the alert is being | ||
| handled as a false positive via a deviation (not by code change), link to | ||
| the | ||
| [deviation mechanism documentation](https://github.com/github/codeql-coding-standards/blob/main/docs/user_manual.md#applying-deviations), | ||
| and summarise the justification so a reviewer can approve or reject it. | ||
|
|
||
| A false-positive PR should therefore contain **only** the deviation entry | ||
| and/or the in-source deviation marker — no changes to logic, no edits to | ||
| build outputs, and no edits to `.gitignore`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd provide more detailed instructions here for how to install into a project.
And we also need to update the user_manual version with author and date etc