Skip to content

Clarify contribution workflow for concurrency-jobs reference implementation #2

Description

@forgottenlab

Hi SCARFBench maintainers,

I am trying to follow CONTRIBUTING.md to understand the expected workflow before contributing a new benchmark application.

While reading the guide, I noticed that benchmark/infrastructure/concurrency-jobs is listed as a reference implementation for basic REST/HTTP applications. However, when I inspected it locally, its current structure seems different from the documented workflow.

What I expected from CONTRIBUTING.md

The guide says that each framework variant should include standard files such as:

  • Dockerfile
  • Makefile
  • smoke.py or a smoke/ folder

It also shows the verification sequence as:

just build
just up
just test
just logs
just down

What I found locally

I checked the three framework variants:

benchmark/infrastructure/concurrency-jobs/spring
benchmark/infrastructure/concurrency-jobs/quarkus
benchmark/infrastructure/concurrency-jobs/jakarta

Each variant contains:

Dockerfile
pom.xml
README.md
test.sh
src/
.mvn/
mvnw
mvnw.cmd

But I could not find:

Makefile
justfile
smoke.py

The Dockerfiles appear to copy and chmod test.sh, then start the application with framework-specific Maven goals:

spring:  CMD ["mvn", "spring-boot:run"]
quarkus: CMD ["mvn", "quarkus:run"]
jakarta: CMD [ "mvn", "liberty:run" ]

For reference, I checked this with commands similar to:

Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter Makefile
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter justfile
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter smoke.py
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter test.sh
Select-String -Path .\benchmark\infrastructure\concurrency-jobs\*\Dockerfile -Pattern "test.sh|CMD|spring-boot|quarkus|liberty"

Question

Is concurrency-jobs an older reference implementation that has not yet been updated to the current Makefile / smoke.py workflow?

Or should new contributors still follow the current CONTRIBUTING.md workflow even if this reference implementation uses test.sh?

I wanted to confirm the expected standard before opening a PR or adding a new benchmark application.

If helpful, I would be happy to submit a small documentation PR to clarify this.

Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions