feat: add pipeline specialized in Maven package scan by chinyeungli · Pull Request #1953 · aboutcode-org/scancode.io

chinyeungli · 2025-11-13T10:40:59Z

Issue: maven-heaven: Design and implement Scancode.io pipeline for a single Maven package #1763
Create a pipeline for Maven package

Signed-off-by: Chin Yeung Li <tli@nexb.com>

- Update format Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

…1763 - Update package's license if missing while the same package has license detected in RESOURCES Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez

Create a new maven pipe module in place of use resolve
Opening and loading a large file to make edits multiple times in various steps is not great.
To be discussed: Do we need a dedicated pipeline for just an extra step? Shouldn't the original scan_single_package detect that it's a Maven package and apply the necessary? Any reason to keep this new logic separated?

Signed-off-by: Chin Yeung Li <tli@nexb.com>

- Create a new maven pipe module - Use database queries for update_package_license_from_resource_if_missing() - Add tests Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli · 2025-12-29T03:37:00Z

@tdruez I’ve updated the code to include support for the "D2D" option.

The "deploy_to_devel" option is equivalent to the "map_deploy_to_develop" pipeline, which runs on Java, JavaScript, Kotlin, and Scala as these are the languages commonly found in Maven projects.

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Some projects encountered a unique constraint violation when a resource was already mapped: ``` duplicate key value violates unique constraint "scanpipe_codebaserelation_unique_relation" DETAIL: Key (from_resource_id, to_resource_id, map_type)=(1512780, 1512790, jar_to_source) already exists. ``` Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Require for `"tab_context": {"VULNERABLECODE_URL": settings.VULNERABLECODE_URL},` Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung <tli@nexb.com>

chinyeungli · 2026-07-14T02:51:58Z

I've redesigned the code. It now only takes 1 PURL as input.
I've also removed the manual option for d2d. If both the source and binary are available, d2d will be performed automatically, otherwise, only a basic scan is performed.

Example 1:
PURL: pkg:maven/aero/champ/cargojson@1.0

Without fetching the remote POM and updating the package data, no packages are detected because the fetched binary (https://repo1.maven.org/maven2/aero/champ/cargojson/1.0/cargojson-1.0.jar) doesn't contain a pom.xml.

However, with remote POM fetching and package data updates enabled, we successfully detect 1 package and update 17 dependencies in the summary:

(We can see the POM file was fetched remotely under "Datafile paths".)

Example 2:
PURL: pkg:maven/activemq/activemq-axis@1.2
https://repo1.maven.org/maven2/activemq/activemq-axis/1.2/

Since there is no source available for this package, only a basic scan is performed.

* Require for `"tab_context": {"VULNERABLECODE_URL": settings.VULNERABLECODE_URL},` Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez · 2026-07-16T08:43:42Z

    {% endfor %}
  </ul>
-{% endif %}
+{% endif %}


Unwanted change

This was likely auto‑corrected when I ran make check, but in any case the missing tag has now been added and fixed.

tdruez · 2026-07-16T08:43:47Z

    {% endfor %}
  </tbody>
-</table>
+</table>


Unwanted change

This was likely auto‑corrected when I ran make check, but in any case the missing tag has now been added and fixed.

Co-authored-by: tdruez <489057+tdruez@users.noreply.github.com> Signed-off-by: Chin Yeung <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez · 2026-07-20T04:30:24Z

        self.assertEqual(0, project1.discoveredpackages.count())

+    @skipIf(sys.platform == "darwin", "Not supported on macOS")
+    def test_scanpipe_scan_maven_package_single_file(self):


Still missing

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Co-authored-by: tdruez <489057+tdruez@users.noreply.github.com> Signed-off-by: Chin Yeung <tli@nexb.com>

tdruez

@chinyeungli a few issues still need to be addressed. Please also add or update unit tests to cover them, so we catch regressions on those.

Signed-off-by: Chin Yeung Li <tli@nexb.com>

* Better error handling * Fix update_scan_data to append rather than overwrite existing dependencies * Improve perfomance for update_package_license_from_resource_if_missing() Signed-off-by: Chin Yeung Li <tli@nexb.com>

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez

@chinyeungli see one last suggestion before the merge/

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli added 5 commits November 13, 2025 07:52

Add "scan_maven_package" pipeline (Working-in-progress) #1763

9e93317

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Use pom_url as the datafile_path for the dependenies #1763

928f742

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Use empty string for datafile_path in dependencies #1763

d63a1e5

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Removed dup code that's already present in ScanSinglePackage #1763

9812129

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update the matching regex for parse_maven_filename and added test #1763

114eb75

- Update format Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli requested a review from tdruez November 13, 2025 10:40

github-advanced-security AI found potential problems Nov 13, 2025

View reviewed changes

Comment thread scanpipe/pipes/resolve.py Fixed

Refactor code and add tests #1763

13e8e88

Signed-off-by: Chin Yeung Li <tli@nexb.com>

github-advanced-security AI found potential problems Nov 14, 2025

View reviewed changes

Comment thread scanpipe/pipes/resolve.py Fixed

chinyeungli added 5 commits November 17, 2025 18:08

Implement "update_package_license_from_resource_if_missing" function #…

cb623c1

…1763 - Update package's license if missing while the same package has license detected in RESOURCES Signed-off-by: Chin Yeung Li <tli@nexb.com>

Only work with "maven" package type #1763

b937cff

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update docstring description #1763

0bbe979

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Add test for the "scan_maven_package" pipeline #1763

2b3c9bb

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Fix the Incomplete URL substring sanitization #1763

9a11dd8

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli mentioned this pull request Nov 18, 2025

maven-heaven: Design and implement Scancode.io pipeline for a single Maven package #1763

Closed

tdruez requested changes Nov 19, 2025

View reviewed changes

Comment thread scanpipe/pipelines/scan_maven_package.py Outdated

Comment thread scanpipe/pipelines/scan_single_package.py Outdated

Comment thread scanpipe/pipelines/scan_single_package.py Outdated

Comment thread scanpipe/pipes/resolve.py Outdated

Comment thread scanpipe/pipes/resolve.py Outdated

chinyeungli added 7 commits November 19, 2025 13:28

Add "maven.google.com" as a maven_hosts and accept .aar file #1763

f303430

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Reorganize code structure and code enhancement #1763

ab71751

- Create a new maven pipe module - Use database queries for update_package_license_from_resource_if_missing() - Add tests Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update build-in-pipelines.rst #1763

5579db8

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Merge remote-tracking branch 'origin/main' into 1763_pipeline_for_maven

9ed4c2d

Added steps to handle D2D in the maven pipeline #1763

12728e4

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update "exclude_from_diff" list #1763

3242b16

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Ruff reformatted #1763

97a109b

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli added 2 commits December 29, 2025 11:45

Update class description #1763

9992d15

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Error handling if pom url not found #1763

4f1b4d3

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli requested a review from tdruez January 28, 2026 03:31

pombredanne requested a review from ppkarwasz February 20, 2026 10:48

TG1999 self-requested a review May 1, 2026 09:08

chinyeungli added 3 commits July 14, 2026 10:32

Add more comments #1763

c77c169

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Add the missing import settings #1763

52eaf65

* Require for `"tab_context": {"VULNERABLECODE_URL": settings.VULNERABLECODE_URL},` Signed-off-by: Chin Yeung Li <tli@nexb.com>

Merge branch 'main' into 1763_pipeline_for_maven

8514b77

Signed-off-by: Chin Yeung <tli@nexb.com>

chinyeungli added 6 commits July 14, 2026 10:58

Add the missing import settings #1763

4066b85

* Require for `"tab_context": {"VULNERABLECODE_URL": settings.VULNERABLECODE_URL},` Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update tests #1763

a7cadde

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Skip test_scanpipe_scan_maven_package_single_file on macos #1763

9516d88

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Better code organization #1763

f018823

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Remove non-used code #1763

8e83344

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Restore license notice and code enhancement #1763

131f84b

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez requested changes Jul 16, 2026

View reviewed changes

chinyeungli and others added 3 commits July 16, 2026 17:54

Apply suggestion from @tdruez

f3401de

Co-authored-by: tdruez <489057+tdruez@users.noreply.github.com> Signed-off-by: Chin Yeung <tli@nexb.com>

Code enhancement based on the PR review comments #1763

96b9334

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Better code structure and added tests #1763

2631c80

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli requested a review from tdruez July 17, 2026 07:09

tdruez requested changes Jul 20, 2026

View reviewed changes

chinyeungli and others added 2 commits July 20, 2026 15:00

Clean up code and improve formatting #1763

91120ae

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Update scanpipe/pipes/maven.py

f2a7e3b

Co-authored-by: tdruez <489057+tdruez@users.noreply.github.com> Signed-off-by: Chin Yeung <tli@nexb.com>

chinyeungli requested a review from tdruez July 20, 2026 10:24

tdruez requested changes Jul 22, 2026

View reviewed changes

chinyeungli added 3 commits July 23, 2026 07:14

Re-order the pipelines order #1763

f479350

Signed-off-by: Chin Yeung Li <tli@nexb.com>

Refactor Maven pipeline for error handling and performance #1763

90c71dd

* Better error handling * Fix update_scan_data to append rather than overwrite existing dependencies * Improve perfomance for update_package_license_from_resource_if_missing() Signed-off-by: Chin Yeung Li <tli@nexb.com>

Fix code/doc formatting #1763

62a29ac

Signed-off-by: Chin Yeung Li <tli@nexb.com>

chinyeungli force-pushed the 1763_pipeline_for_maven branch from 68dc17b to 62a29ac Compare July 23, 2026 04:25

chinyeungli requested a review from tdruez July 23, 2026 04:31

tdruez requested changes Jul 29, 2026

View reviewed changes

Comment thread scanpipe/pipes/maven.py Outdated

Use PurePosixPath for file checking #1763

895b1ce

Signed-off-by: Chin Yeung Li <tli@nexb.com>

tdruez merged commit 4186863 into main Jul 29, 2026
16 of 24 checks passed

tdruez deleted the 1763_pipeline_for_maven branch July 29, 2026 07:39

tdruez changed the title ~~Pipeline for Maven~~ feat: add pipeline specialized in Maven package scan Jul 29, 2026

Uh oh!

Uh oh!

Conversation

chinyeungli commented Nov 13, 2025 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chinyeungli commented Dec 29, 2025

Uh oh!

chinyeungli commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

chinyeungli Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

tdruez Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

chinyeungli Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tdruez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

chinyeungli commented Nov 13, 2025 •

edited by pombredanne

Loading