Hi SCARFBench maintainers,
I am trying to follow CONTRIBUTING.md to understand the expected workflow before contributing a new benchmark application.
While reading the guide, I noticed that benchmark/infrastructure/concurrency-jobs is listed as a reference implementation for basic REST/HTTP applications. However, when I inspected it locally, its current structure seems different from the documented workflow.
What I expected from CONTRIBUTING.md
The guide says that each framework variant should include standard files such as:
Dockerfile
Makefile
smoke.py or a smoke/ folder
It also shows the verification sequence as:
just build
just up
just test
just logs
just down
What I found locally
I checked the three framework variants:
benchmark/infrastructure/concurrency-jobs/spring
benchmark/infrastructure/concurrency-jobs/quarkus
benchmark/infrastructure/concurrency-jobs/jakarta
Each variant contains:
Dockerfile
pom.xml
README.md
test.sh
src/
.mvn/
mvnw
mvnw.cmd
But I could not find:
Makefile
justfile
smoke.py
The Dockerfiles appear to copy and chmod test.sh, then start the application with framework-specific Maven goals:
spring: CMD ["mvn", "spring-boot:run"]
quarkus: CMD ["mvn", "quarkus:run"]
jakarta: CMD [ "mvn", "liberty:run" ]
For reference, I checked this with commands similar to:
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter Makefile
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter justfile
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter smoke.py
Get-ChildItem .\benchmark\infrastructure\concurrency-jobs -Recurse -Filter test.sh
Select-String -Path .\benchmark\infrastructure\concurrency-jobs\*\Dockerfile -Pattern "test.sh|CMD|spring-boot|quarkus|liberty"
Question
Is concurrency-jobs an older reference implementation that has not yet been updated to the current Makefile / smoke.py workflow?
Or should new contributors still follow the current CONTRIBUTING.md workflow even if this reference implementation uses test.sh?
I wanted to confirm the expected standard before opening a PR or adding a new benchmark application.
If helpful, I would be happy to submit a small documentation PR to clarify this.
Thank you!
Hi SCARFBench maintainers,
I am trying to follow
CONTRIBUTING.mdto understand the expected workflow before contributing a new benchmark application.While reading the guide, I noticed that
benchmark/infrastructure/concurrency-jobsis listed as a reference implementation for basic REST/HTTP applications. However, when I inspected it locally, its current structure seems different from the documented workflow.What I expected from
CONTRIBUTING.mdThe guide says that each framework variant should include standard files such as:
DockerfileMakefilesmoke.pyor asmoke/folderIt also shows the verification sequence as:
just build just up just test just logs just downWhat I found locally
I checked the three framework variants:
Each variant contains:
But I could not find:
The Dockerfiles appear to copy and chmod
test.sh, then start the application with framework-specific Maven goals:For reference, I checked this with commands similar to:
Question
Is
concurrency-jobsan older reference implementation that has not yet been updated to the currentMakefile/smoke.pyworkflow?Or should new contributors still follow the current
CONTRIBUTING.mdworkflow even if this reference implementation usestest.sh?I wanted to confirm the expected standard before opening a PR or adding a new benchmark application.
If helpful, I would be happy to submit a small documentation PR to clarify this.
Thank you!