Add/update documentation, Docker compose, and configuration files

.gitignore

@@ -10,9 +10,19 @@ __pycache__/
 *.pyo
 .venv/
 venv/
+#
+sort_json.py
+tmp.py
 
 # Java
 *.class
 target/
 build/
 .gradle/
+
+# JSON
+languages.json
+
+# Folders
+ide/
+judge0-v1.13.1.zip

LANGUAGES.md

@@ -0,0 +1,101 @@
+# Judge0
+
+## Supported/Active Languages
+
+### By Name
+* Assembly (NASM 2.14.02)
+* Bash (5.0.0)
+* Basic (FBC 1.07.1)
+* C (Clang 7.0.1)
+* C++ (Clang 7.0.1)
+* C (GCC 7.4.0)
+* C++ (GCC 7.4.0)
+* C (GCC 8.3.0)
+* C++ (GCC 8.3.0)
+* C (GCC 9.2.0)
+* C++ (GCC 9.2.0)
+* Clojure (1.10.1)
+* C# (Mono 6.6.0.161)
+* COBOL (GnuCOBOL 2.2)
+* Common Lisp (SBCL 2.0.0)
+* D (DMD 2.089.1)
+* Elixir (1.9.4)
+* Erlang (OTP 22.2)
+* Executable
+* F# (.NET Core SDK 3.1.202)
+* Fortran (GFortran 9.2.0)
+* Go (1.13.5)
+* Groovy (3.0.3)
+* Haskell (GHC 8.8.1)
+* Java (OpenJDK 13.0.1)
+* JavaScript (Node.js 12.14.0)
+* Kotlin (1.3.70)
+* Lua (5.3.5)
+* Multi-file program
+* Objective-C (Clang 7.0.1)
+* OCaml (4.09.0)
+* Octave (5.1.0)
+* Pascal (FPC 3.0.4)
+* Perl (5.28.1)
+* PHP (7.4.1)
+* Plain Text
+* Prolog (GNU Prolog 1.4.5)
+* Python (2.7.17)
+* Python (3.8.1)
+* R (4.0.0)
+* Ruby (2.7.0)
+* Rust (1.40.0)
+* Scala (2.13.2)
+* SQL (SQLite 3.27.2)
+* Swift (5.2.3)
+* TypeScript (3.7.4)
+* Visual Basic.Net (vbnc 0.0.0.5943)
+
+### By ID
+43. Plain Text
+44. Executable
+45. Assembly (NASM 2.14.02)
+46. Bash (5.0.0)
+47. Basic (FBC 1.07.1)
+48. C (GCC 7.4.0)
+49. C (GCC 8.3.0)
+50. "C (GCC 9.2.0)
+51. "C# (Mono 6.6.0.161)
+52. "C++ (GCC 7.4.0)
+53. C++ (GCC 8.3.0)
+54. C++ (GCC 9.2.0)
+55. Common Lisp (SBCL 2.0.0)
+56. D (DMD 2.089.1)
+57. Elixir (1.9.4)
+58. Erlang (OTP 22.2)
+59. Fortran (GFortran 9.2.0)
+60. Go (1.13.5)
+61. Haskell (GHC 8.8.1)
+62. Java (OpenJDK 13.0.1)
+63. JavaScript (Node.js 12.14.0)
+64. Lua (5.3.5)
+65. OCaml (4.09.0)
+66. Octave (5.1.0)
+67. Pascal (FPC 3.0.4)
+68. PHP (7.4.1)
+69. Prolog (GNU Prolog 1.4.5)
+70. Python (2.7.17)
+71. Python (3.8.1)
+72. Ruby (2.7.0)
+73. Rust (1.40.0)
+74. TypeScript (3.7.4)
+75. C (Clang 7.0.1)
+76. C++ (Clang 7.0.1)
+77. COBOL (GnuCOBOL 2.2)
+78. Kotlin (1.3.70)
+79. Objective-C (Clang 7.0.1)
+80. R (4.0.0)
+81. Scala (2.13.2)
+82. SQL (SQLite 3.27.2)
+83. Swift (5.2.3)
+84. Visual Basic.Net (vbnc 0.0.0.5943)
+85. Perl (5.28.1)
+86. Clojure (1.10.1)
+87. F# (.NET Core SDK 3.1.202)
+88. Groovy (3.0.3)
+89. Multi-file program

LINKS.md

@@ -0,0 +1,20 @@
+# Judge0
+
+## General
+- [API](https://ce.judge0.com/)
+- [Deployment](https://github.com/judge0/judge0/blob/master/CHANGELOG.md#deployment-procedure)
+
+## Localhost
+
+- [Dummy Client](http://localhost:2358/dummy-client)
+- [API Docs](http://localhost:2358/docs)
+- [Supported Languages](http://localhost:2358/languages)
+- [Status Codes](http://localhost:2358/statuses)
+- [System Information](http://localhost:2358/system_info)
+- [Judge0 Configuration](http://localhost:2358/config_info)
+- [Current Instance Stats](http://localhost:2358/statistics)
+- [Workers Queue](http://localhost:2358/workers)
+- [General Information](http://localhost:2358/about)
+- [Judge0 Version](http://localhost:2358/version)
+- [Judge0 Isolate Version](http://localhost:2358/isolate)
+- [Judge0 License](http://localhost:2358/license)

README.md

@@ -1,46 +1,188 @@
-# Code Execution Question Spike
-
-## What this repo is
-
-This is a time-boxed investigation prototype for adding code-execution question support to [questions.carletoncomputerscience.ca](https://questions.carletoncomputerscience.ca/). It is not production code and will not be deployed.
-
-Three approaches are being evaluated in parallel via spike tickets. Once viable options are identified, the chosen approach is migrated into the main Astro repo and this repo is archived. Nothing here is intended to be maintained long-term.
-
-## Approaches under evaluation
-
-### (a) Pyodide — client-side Python execution
-
-Pyodide runs CPython compiled to WebAssembly, executing Python entirely in the browser with no server involvement. The spike evaluates whether student-submitted Python functions can be loaded, executed against a test suite, and graded without any network round-trip. Key unknowns: bundle size, startup latency, and sandbox adequacy for an exam-adjacent context.
-
-### (b) CheerpJ (or similar) — client-side Java execution
-
-CheerpJ compiles Java bytecode to WebAssembly/JavaScript, allowing Java programs to run in-browser. The spike evaluates whether student Java submissions can be compiled or pre-compiled and executed client-side with comparable grading semantics to the Python approach. Key unknowns: whether student code can be compiled in-browser or requires a pre-compilation step, runtime overhead, and JVM subset limitations.
-Other options may include TeaVM, QEMU-WASM, or something else. 
-
-### (c) Judge0 — server-side execution backend
-
-Judge0 is an open-source code execution API that runs submissions in isolated containers server-side. The spike evaluates the integration complexity of setting up [Judge0](https://judge0.com/) (self-hosted), the latency and infrastructure overhead, the process of configuring questions on the platform, and the viability as a fallback for languages that cannot run client-side.
-
----
-
-The goal is to have either a way to mark both Python and Java questions fully locally in the user's browser, and/or to determine the Judge0 integration process and complexity. Python and Java are the two priority languages as they are used in the intro courses.
+# Judge0
+
+## Goals
+✅ Self-host a Judge0 instance.  
+✅ Submit a student's Python or Java submission alongside a test file.  
+✅ Get back pass/fail results matching the JSON format in `SCHEMA.md`.  
+❔ Understand what it takes to keep this running reliably.  
+
+## Findings
+
+1. **Setup complexity.**  
+Here's some information on what is involved with self-hosting Judge0 based on my own setup experience:
+
+***Steps***:  
+Clone this repository to replicate the self-hosted judge0 instance I used, otherwise you can follow the instructions listed [here].(https://github.com/judge0/judge0/blob/master/CHANGELOG.md#deployment-procedure)
+```
+git clone [URL]
+cd [REPOSITORY]
+```
+Run all services and wait a few seconds until everything is initialized:
+```
+docker-compose up -d db redis
+sleep 10s
+docker-compose up -d
+sleep 5s
+```
+A Judge0 CE v1.13.1 instance is now up and running; visit docs at `http://<IP ADDRESS OF YOUR SERVER, OR LOCALHOST>:2358/docs` to confirm the instance is working as expected.
+
+To test python code execution, you can use the following commands:
+```
+python3 test_judge0.py ./submissions/add-two-numbers/broken.py questions/add-two-numbers/tests.py
+
+python3 test_judge0.py ./submissions/add-two-numbers/correct.py questions/add-two-numbers/tests.py
+
+python3 test_judge0.py ./submissions/add-two-numbers/wrong-answer.py questions/add-two-numbers/tests.py
+
+python3 test_judge0.py ./submissions/reverse-string/broken.py questions/reverse-string/tests.py
+
+python3 test_judge0.py ./submissions/reverse-string/correct.py questions/reverse-string/tests.py
+
+python3 test_judge0.py ./submissions/reverse-string/wrong-answer.py questions/reverse-string/tests.py
+```
+A sample of Java code submitted through the [Dummy Client](http://localhost:2358/dummy-client):
+```
+import java.util.ArrayList;
+import java.util.List;
+
+public class Main {
+    public static void main(String[] args) {
+        Tests.main(args);
+    }
+}
+
+// SOLUTION (user provided)
+
+class Solution {
+    public static int add(int a, int b) {
+        return a + b;
+    }
+}
+
+class Tests {
+    static int passed = 0;
+    static int failed = 0;
+    static List<String> failures = new ArrayList<>();
+
+    static void run(String name, int got, int expected) {
+        if (got == expected) {
+            passed++;
+        } else {
+            failed++;
+            failures.add(
+                "{\"test\": \"" + name + "\", \"message\": \"expected " + expected + " got " + got + "\"}"
+            );
+        }
+    }
+
+    public static void main(String[] args) {
+        run("test_positive",      Solution.add(1, 2),     3);
+        run("test_both_negative", Solution.add(-3, -4),  -7);
+        run("test_zero",          Solution.add(0, 0),     0);
+        run("test_mixed",         Solution.add(-1, 5),    4);
+        run("test_large",         Solution.add(100, 200), 300);
+
+        StringBuilder sb = new StringBuilder();
+        sb.append("{\"passed\": ").append(passed);
+        sb.append(", \"failed\": ").append(failed);
+        sb.append(", \"failures\": [");
+        for (int i = 0; i < failures.size(); i++) {
+            if (i > 0) sb.append(", ");
+            sb.append(failures.get(i));
+        }
+        sb.append("]}");
+        System.out.println(sb.toString());
+        System.exit(failed);
+    }
+}
+```
+
+***Prerequisites***:
+- Judge0 Docker image size: 3.295GB
+- Base storage: 20GB
+- Recommended storage: >=50GB
+- Base RAM: 2GB
+- Recommended RAM: >=4GB
+- Base CPU: 2vCPU
+- Recommended CPU: >=4vCPU
+
+***Time***:
+- Editing judge0 (CONF file) & docker compose (YAML source): ≈15 mins
+- Judge0 works well with minimum technical setup requirements, making deployment fairly quick.
+
+***Potential Roadblocks***:
+- I appended two python scripts together so they are submitted as a single file for code execution. This means that I had to tweak the test file so that it wasn't importing the user submission, it simply used the user-defined functions within the test suite as a single file.
+- Judge0 has only been tested on Linux and might not work well on other systems (if at all)
+- Implementing some kind of UI, or using an external code editor (Monaco, CodeMirror, Ace), will require development as well as regular maintenance.
+
+***Hosted API***:
+- RapidAPI
+  - Basic: $0.0017/use
+  - Pro: $44.99/mo (2000 submissions/day)
+  - Ultra: $89.99/mo (5000 submissions/day)
+  - Mega: $169.99/mo (10000 submissions/day)
+  - \+ Bandwidth Platform fee (10240MB/mo + $0.001/ per 1MB)
+
+2. **Infrastructure requirements.**    
+Further testing is needed to determine any issues that might arise during spikes in traffic.   
+Implementing a pseudo-IDE or code editor may take some time for developers, based on server/hardware restrictions, instance requirements, and potentially refactoring existing website code. Security/software updates will be dependent on the latest Docker image. And an effective method to prevent misuse (ie: input validation, infinite loops, bots) will need to be considered - on top of what can be mitigated through the Judge0 configuration file.
+
+
+***My Docker Statistics***
+- Container CPU Usage: 12%/1200% (12 CPUs available)  
+With 20 runs, 200%/1200%
+- Container memory usage: 2.28GB/15.17GB
+- Compose stack:
+    - server (judge0/judge0:1.13.1)
+    - workers (judge0/judge0:1.13.1)  
+    *\* Additional worker nodes can be deployed to process code execution in parallel during traffic spikes*
+    - db (postgres:16.2)  
+    *\* Stores persistent configuration, submission records, and language metadata.*
+    - redis (redis:7.2.4)
+
+3. **End-to-end latency.**  
+Average Code Execution Time: 0.02 seconds  
+More testing needs to be done for concurrent submissions, queuing, and multi-user wait times.
+
+4. **Question and submission workflow.**  
+A test file and the corresponding submission code can be appended together to execute the test suite using the user provided solution. The 200 OK Response will be JSON formatted to include:
+```
+{
+    "stdout": [Standard output of the program after execution],  
+    "time": [Program’s run time],  
+    "memory": [Memory used by the program after execution],
+    "stderr": [Standard error of the program after execution],
+    "token": [Unique submission token which can be used to get a specific submission],
+    "compile_output:" [Compiler output after compilation],
+    "message": [If submission status is Internal Error then this message comes from Judge0 itself, otherwise this is status message from isolate],
+    "status": [Submission status] 
+    {
+        "id": [The status code as listed in Statuses.md],
+        "description":[A description for the corresponding status code]
+    }
+}
+```
+5. **Time and memory limits.**   
+The `judge0.conf` file allows you to customize server, workers, server access, authorization, redis, postgreSQL, submission, and rails configurations. For submissions, platform-wide settings include CPU time, max memory, max stack, max processes/threads, file size, number of runs, and more. It is also possible to tweak these settings per submission using JSON formatted HTTP requests - these can be added to specific submissions based on how it is implemented within the question repository.  
+Judge0 surfaces timeouts and memory errors back through the API within the 200 OK Response using JSON formatting to list `stdout`, `stderr`, `message`, and `status` fields.
+
+6. **Python and Java support.**   
+A python script has been created for the purposes of testing Python code execution, while I have manually used the [Localhost Dummy Client](http://localhost:2358/dummy-client) to verify Java code execution (similar method of combining the user submission and the test files). Compiler flags and command line arguments can also be added to submissions within the POST request.
+
+\* Make sure to update the variables `REDIS_PASSWORD` and `POSTGRES_PASSWORD` in the `judge0.conf` file before deploying to a production environment.
+
+
+### ✅ Is it usable for our case?
+Depending on available computing resources for CCSS, I believe that this option is feasible (with the caveats listed above), but would need further testing to determine weakpoints during periods of high traffic (ie: during exams).
 
 ## Repo contents
 
 | Path | Description |
 |---|---|
 | [SCHEMA.md](SCHEMA.md) | Question format specification |
+| [LANGUAGES.md](LANGUAGES.md) | Supported/Active Languages |
+| [LINKS.md](LINKS.md) | Documentation & Localhost Links |
+| [STATUSES.md](STATUSES.md) | Status codes returned after code execution |
 | [questions/](questions/) | Example questions in the defined schema |
 | [submissions/](submissions/) | Sample student submissions used to test grading logic |
-
-## What is deliberately NOT here yet
-
-The following are out of scope until spike options have been evaluated and a direction is chosen:
-
-- A code editor (Monaco, CodeMirror, or otherwise)
-- A UI shell or any frontend scaffolding
-- A shared runner interface or abstraction layer
-- Integration with the main Astro site
-- Styling of any kind
-
-Spike authors should ideally not build any of the above. The purpose of each spike is narrow: determine whether the execution approach works and what its constraints are. UI and integration work follows after a direction is picked.
+| [Python Test Script](test_judge0.py) | Reads an example python code submission as well as the corresponding tester file, appending them together and submitting as a JSON payload to the Judge0 instance for code execution |

STATUSES.md

@@ -0,0 +1,18 @@
+# Judge0
+
+## Statuses
+
+01. In Queue
+02. Processing
+03. Accepted
+04. Wrong Answer
+05. Time Limit Exceeded
+06. Compilation Error
+07. Runtime Error (SIGSEGV)
+08. Runtime Error (SIGXFSZ)
+09. Runtime Error (SIGFPE)
+10. Runtime Error (SIGABRT)
+11. Runtime Error (NZEC)
+12. Runtime Error (Other)
+13. Internal Error
+14. Exec Format Error