Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,33 @@ All notable changes to EigenScript are documented here.

## [Unreleased]

### Added
- **`read_line` builtin (#558)** — blocking line read from stdin
(`getline(3)`): returns the next line without its trailing newline
(`\r\n` stripped as one unit), `null` at EOF, `""` for an empty line.
The stream-safe stdin primitive: `read_text of "/dev/stdin"` sizes
with fseek/ftell and silently reads `""` on a pipe, so a CLI in a
shell pipeline (`gen | eigenscript client.eigs`) had no way to
receive lines. Tape-first: recorded/replayed like `read_bytes`
(TAKE/RECORD — replay serves recorded lines, no live stdin read).
Hosted-only (freestanding profile excludes it, like the other IO).
- **`is_dir` builtin (#576)** — `is_dir of path` → 1/0 via `stat(2)`,
replacing the consumer-side `file_exists of f"{path}/."` probe.
Trace-recorded (`TRACE_NONDET_RET`), unlike the older untraced fs
predicates (`file_exists`, `ls`, `mkdir`, `getcwd` — flagged as a
follow-up).

### Fixed
- **`json_decode` accepts RFC 8259 exponent notation (#557).** The
number scanner accepted `e`/`E`/`+` but never the `-` of a negative
exponent, so `1e-06` — emitted by Python `json.dumps`, JS
`JSON.stringify`, serde, and by EigenScript's own `json_encode`
(`%.15g`) for tiny/huge magnitudes — failed to parse: encode→decode
did not round-trip. The scanner now implements the exact RFC §6
grammar (`[minus] int [frac] [exp]`) and raises on malformed tails
(`"1e"`, `"1e+"`, `"1."`) instead of letting `atof()` guess silently.
Leading zeros stay accepted (old laxity, unchanged).

### Added
- **Lint W019 — statement-level interrogative discards its result
(#583).** `why is "..."` where `why` is a question word parses as the
Expand Down
2 changes: 2 additions & 0 deletions docs/BUILTINS.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,9 @@ Boolean keywords that check the most recently observed value:
|------|-----------|-------------|
| `load_file` | `load_file of "path.eigs"` | Load and execute EigenScript file. A missing/unreadable path raises a catchable `io` error (matching `import`); a parse/compile failure in the file raises `parse`. |
| `file_exists` | `file_exists of "path"` | 1 if file exists, 0 otherwise |
| `is_dir` | `is_dir of "path"` | 1 if the path names a directory, 0 for a plain file / missing path (#576 — replaces the `file_exists of "path/."` probe). Trace-recorded, so replay is deterministic |
| `read_text` | `read_text of "path"` | Read file contents as string ("" on failure, 10 MB cap) |
| `read_line` | `read_line of null` | Blocking line read from **stdin**: next line without its trailing newline (`\r\n` stripped as one unit), `null` at EOF; an empty line is `""`. Works on pipes — the stream-safe primitive `read_text of "/dev/stdin"` can't be (fseek fails on unseekable fds, #558). Trace-recorded, so replay is deterministic |
| `read_bytes` | `read_bytes of "path"` | Read a file's raw bytes as a list of integers 0–255 (`null` on failure, 10 MB cap). Trace-recorded, so replay is deterministic |
| `proc_read_buf` | `proc_read_buf of [out_fd, max]` | Single `read(2)` of up to `max` bytes from a child fd, returned as a list of integers 0–255 — the byte-list twin of `proc_read`. `null` on EOF / error, 10 MB cap. Replay-gated |
| `write_text` | `write_text of ["path", text]` | Write string to file (1 on success, 0 on failure) |
Expand Down
2 changes: 1 addition & 1 deletion docs/TRACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ perspective lands on the tape as an `N` record:
- **Random:** `random`, `random_int`, `random_normal`, `random_hex`
- **Time:** `monotonic_ns`, `monotonic_ms`
- **Environment / files:** `env_get`, `read_text`, `read_bytes`,
`read_bytes_buf`
`read_bytes_buf`, `read_line` (stdin, #558), `is_dir` (#576)
- **Process:** `args` (command-line arguments — differ across
invocations, so the recorded list is served on replay regardless of
the live argv; #471)
Expand Down
83 changes: 80 additions & 3 deletions src/builtins.c
Original file line number Diff line number Diff line change
Expand Up @@ -939,14 +939,45 @@ static Value* eigs_json_parse_string(const char *s, int *pos) {
return v;
}

/* #557: RFC 8259 §6 number grammar — [ minus ] int [ frac ] [ exp ].
* The old scanner accepted 'e'/'E'/'+' but never the '-' of a negative
* exponent, so any mainstream producer's scientific notation (Python
* json.dumps, JS JSON.stringify, serde: "1e-06") failed at the sign —
* including json_encode's OWN %.15g output for tiny/huge magnitudes.
* The grammar is scanned exactly; a malformed tail ("1e", "1e+", "1.")
* sets g_json_parse_err instead of letting atof() guess silently.
* (Deliberately lax vs RFC in one spot: leading zeros ("01") stay
* accepted, matching the old scanner.) numbuf caps at 63 chars — beyond
* double's ~24-char maximum the extra digits carry no value anyway, and
* the scan itself continues so *pos stays correct. */
static Value* eigs_json_parse_number(const char *s, int *pos) {
char numbuf[64];
int len = 0;
if (s[*pos] == '-') numbuf[len++] = s[(*pos)++];
while (s[*pos] && (isdigit(s[*pos]) || s[*pos] == '.' || s[*pos] == 'e' || s[*pos] == 'E' || s[*pos] == '+') && len < 63) {
numbuf[len++] = s[(*pos)++];
int p = *pos;
#define JSON_NUM_PUSH() do { if (len < 63) numbuf[len++] = s[p]; p++; } while (0)
if (s[p] == '-') JSON_NUM_PUSH();
if (!isdigit((unsigned char)s[p])) {
g_json_parse_err = 1; *pos = p; return make_num(0);
}
while (isdigit((unsigned char)s[p])) JSON_NUM_PUSH();
if (s[p] == '.') {
JSON_NUM_PUSH();
if (!isdigit((unsigned char)s[p])) { /* "1." — frac needs digits */
g_json_parse_err = 1; *pos = p; return make_num(0);
}
while (isdigit((unsigned char)s[p])) JSON_NUM_PUSH();
}
if (s[p] == 'e' || s[p] == 'E') {
JSON_NUM_PUSH();
if (s[p] == '+' || s[p] == '-') JSON_NUM_PUSH();
if (!isdigit((unsigned char)s[p])) { /* "1e", "1e+" — exp needs digits */
g_json_parse_err = 1; *pos = p; return make_num(0);
}
while (isdigit((unsigned char)s[p])) JSON_NUM_PUSH();
}
#undef JSON_NUM_PUSH
numbuf[len] = '\0';
*pos = p;
return make_num(atof(numbuf));
}

Expand Down Expand Up @@ -2780,6 +2811,21 @@ Value* builtin_file_exists(Value *arg) {
}
#endif /* !EIGENSCRIPT_FREESTANDING */

/* is_dir of path — 1 if path names a directory, 0 for a plain file, a
* missing path, or a non-string arg. Replaces the consumer-side
* `file_exists of f"{path}/."` probe (#576). Nondeterministic fs read →
* trace-recorded per the tape-first rule (NB: the older fs predicates —
* file_exists, ls, mkdir, getcwd — predate the tape and are untraced;
* that inconsistency is flagged on the PR, not silently copied here). */
#if !EIGENSCRIPT_FREESTANDING
Value* builtin_is_dir(Value *arg) {
if (!arg || arg->type != VAL_STR) TRACE_NONDET_RET("is_dir", make_num(0));
struct stat st;
TRACE_NONDET_RET("is_dir",
make_num(stat(arg->data.str, &st) == 0 && S_ISDIR(st.st_mode) ? 1 : 0));
}
#endif /* !EIGENSCRIPT_FREESTANDING */

/* rename of [old_path, new_path] — rename/replace a file. On POSIX rename(2) is
* atomic: a crash leaves either the old file or the new file fully in place,
* never a torn mix — the basis for crash-safe log compaction (write a new log to
Expand Down Expand Up @@ -2891,6 +2937,35 @@ Value* builtin_read_text(Value *arg) {
}
#endif /* !EIGENSCRIPT_FREESTANDING */

/* ==== BUILTIN: read_line ==== */
/* read_line of null — blocking line read from stdin via getline(3):
* returns the next line without its trailing newline (a "\r\n"
* terminator is stripped as one unit), or null at EOF. An empty line is
* "" — distinguishable from EOF. The stream-safe stdin primitive
* (#558): read_text of "/dev/stdin" sizes with fseek/ftell, which fails
* on an unseekable fd, so a PIPE silently reads as "" — any CLI meant to
* sit in a shell pipeline needs this instead. Nondeterministic input →
* tape-first: TAKE/RECORD like read_bytes, so under EIGS_REPLAY the
* recorded lines are served and no live stdin read runs. */
#if !EIGENSCRIPT_FREESTANDING
Value* builtin_read_line(Value *arg) {
(void)arg;
TRACE_NONDET_TAKE("read_line");
char *line = NULL;
size_t cap = 0;
ssize_t n = getline(&line, &cap, stdin);
if (n < 0) {
free(line);
TRACE_NONDET_RECORD("read_line", make_null());
}
if (n > 0 && line[n - 1] == '\n') line[--n] = '\0';
if (n > 0 && line[n - 1] == '\r') line[--n] = '\0';
Value *v = make_str(line);
free(line);
TRACE_NONDET_RECORD("read_line", v);
}
#endif /* !EIGENSCRIPT_FREESTANDING */

/* ==== BUILTIN: write_text ==== */
/* write_text of ["path", text] → 1 on success, 0 on failure. */
#if !EIGENSCRIPT_FREESTANDING
Expand Down Expand Up @@ -6003,6 +6078,7 @@ void register_builtins(Env *env) {
env_set_local_owned(env, "regex_replace", make_builtin(builtin_regex_replace));
env_set_local_owned(env, "load_file", make_builtin(builtin_load_file));
env_set_local_owned(env, "file_exists", make_builtin(builtin_file_exists));
env_set_local_owned(env, "is_dir", make_builtin(builtin_is_dir));
env_set_local_owned(env, "rename", make_builtin(builtin_rename));
env_set_local_owned(env, "remove_file", make_builtin(builtin_remove_file));
#endif /* !EIGENSCRIPT_FREESTANDING */
Expand All @@ -6011,6 +6087,7 @@ void register_builtins(Env *env) {
env_set_local_owned(env, "read_bytes", make_builtin(builtin_read_bytes));
env_set_local_owned(env, "read_bytes_buf", make_builtin(builtin_read_bytes_buf));
env_set_local_owned(env, "read_text", make_builtin(builtin_read_text));
env_set_local_owned(env, "read_line", make_builtin(builtin_read_line));
env_set_local_owned(env, "write_text", make_builtin(builtin_write_text));
env_set_local_owned(env, "write_bytes", make_builtin(builtin_write_bytes));
env_set_local_owned(env, "exec_capture", make_builtin(builtin_exec_capture));
Expand Down
2 changes: 2 additions & 0 deletions src/eigenlsp.c
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,10 @@ static const char *builtin_docs[][2] = {
{"index_of", "index_of of [string, substring] -- first index or -1"},
{"str_replace", "str_replace of [string, old, new] -- replace all occurrences"},
{"read_text", "read_text of path -- read file as string"},
{"read_line", "read_line of null -- read next line from stdin (null at EOF)"},
{"write_text", "write_text of [path, text] -- write string to file"},
{"file_exists", "file_exists of path -- 1 if file exists"},
{"is_dir", "is_dir of path -- 1 if path is a directory"},
{"json_encode", "json_encode of value -- encode as JSON string"},
{"json_decode", "json_decode of string -- parse JSON to value"},
{"spawn", "spawn of fn -- create thread running function, returns handle"},
Expand Down
18 changes: 17 additions & 1 deletion tests/run_all_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1386,6 +1386,22 @@ else
fi
echo ""

# [42a2] read_line (#558): stream-safe stdin line read + record/replay
echo "[42a2] read_line (counted dynamically)"
RL_OUTPUT=$(bash "$TESTS_DIR/test_read_line.sh" 2>&1)
RL_PASS=$(echo "$RL_OUTPUT" | grep -c "PASS:" || true)
RL_FAIL=$(echo "$RL_OUTPUT" | grep -c "FAIL:" || true)
TOTAL=$((TOTAL + RL_PASS + RL_FAIL))
PASS=$((PASS + RL_PASS))
FAIL=$((FAIL + RL_FAIL))
if [ "$RL_FAIL" -gt 0 ]; then
echo " FAIL: $RL_FAIL read_line check(s) failed"
echo "$RL_OUTPUT" | grep "FAIL:" | head -5
else
echo " PASS: all $RL_PASS read_line checks"
fi
echo ""

# [42b] --test --trace-on-fail (#394): every failure is a replayable tape
echo "[42b] Trace-on-fail (7 checks)"
TOF_OUTPUT=$(bash "$TESTS_DIR/test_trace_on_fail.sh" 2>&1)
Expand Down Expand Up @@ -2618,7 +2634,7 @@ check_eigs_suite "error propagation" test_error_propagation.eigs "error propagat
check_eigs_suite "handle forge" test_handle_forge.eigs "PASS: handle table" 1
check_eigs_suite "byte<->value builtins (str_from_bytes / f64 bytes)" test_byte_value_builtins.eigs "All tests passed" 19
check_eigs_suite "write_bytes (binary append/truncate)" test_write_bytes.eigs "All tests passed" 10
check_eigs_suite "rename / remove_file (atomic swap, delete)" test_file_rename.eigs "All tests passed" 12
check_eigs_suite "rename / remove_file / is_dir (atomic swap, delete, dir probe)" test_file_rename.eigs "All tests passed" 17
check_eigs_suite "vm_run_bytecode + sandbox (self-hosting bridge)" test_vm_run_bytecode.eigs "All tests passed" 29
check_eigs_suite "sandbox fail-closed allowlist (no host-global escape)" test_sandbox_allow.eigs "SANDBOX_ALLOW_OK" 1
check_eigs_suite "JIT and/or heap-operand decref (no per-iteration leak)" test_jit_andor_leak.eigs "jit-and-or-ok" 1
Expand Down
12 changes: 12 additions & 0 deletions tests/test_file_rename.eigs
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,16 @@ assert_eq of [file_exists of B, 0, "file gone after remove_file"]
assert_eq of [rename of ["/tmp/eigs_no_such_file_zzz", B], 0, "rename of missing -> 0"]
assert_eq of [remove_file of "/tmp/eigs_no_such_file_zzz", 0, "remove missing -> 0"]

# is_dir (#576): directory detection without the file_exists of "path/." probe
D is "/tmp/eigs_isdir_probe/nested"
F is "/tmp/eigs_isdir_file.bin"
mkdir of D
write_bytes of [F, [1], 0]
assert_eq of [is_dir of "/tmp/eigs_isdir_probe", 1, "is_dir: directory -> 1"]
assert_eq of [is_dir of D, 1, "is_dir: nested directory -> 1"]
assert_eq of [is_dir of F, 0, "is_dir: plain file -> 0"]
assert_eq of [is_dir of "/tmp/eigs_no_such_dir_zzz", 0, "is_dir: missing path -> 0"]
assert_eq of [is_dir of 42, 0, "is_dir: non-string -> 0"]
remove_file of F

test_summary of null
45 changes: 45 additions & 0 deletions tests/test_json_hard.eigs
Original file line number Diff line number Diff line change
Expand Up @@ -115,4 +115,49 @@ assert of [len of three_mid.s == 5, "JH42 3-byte cp surrounded by ASCII"]
unknown_esc is json_decode of "{\"s\": \"a\\qb\"}"
assert of [unknown_esc.s == "aqb", "JH43 unknown escape drops backslash"]

# --- RFC 8259 exponent notation (#557) — decode side ---
e1 is json_decode of "[1e-06]"
assert of [e1[0] == 0.000001, "JH44 1e-06 decodes"]
e2 is json_decode of "[1E+5]"
assert of [e2[0] == 100000, "JH45 1E+5 decodes (capital E, plus sign)"]
e3 is json_decode of "[2.5e3]"
assert of [e3[0] == 2500, "JH46 frac+exp decodes"]
e4 is json_decode of "[-6.179673164297128e-06]"
assert of [e4[0] < 0, "JH47 negative mantissa + negative exponent sign"]
assert of [e4[0] > (0 - 0.00001), "JH48 tiny magnitude in range"]
e5 is json_decode of "{\"v\": 1.5E-3}"
assert of [e5.v == 0.0015, "JH49 exponent inside object value"]

# --- malformed number tails now raise instead of atof-guessing ---
bad_exp_err is 0
try:
x1 is json_decode of "[1e]"
catch err:
bad_exp_err is 1
assert of [bad_exp_err == 1, "JH50 bare 'e' with no digits raises"]

bad_sign_err is 0
try:
x2 is json_decode of "[1e+]"
catch err2:
bad_sign_err is 1
assert of [bad_sign_err == 1, "JH51 exponent sign with no digits raises"]

bad_frac_err is 0
try:
x3 is json_decode of "[1.]"
catch err3:
bad_frac_err is 1
assert of [bad_frac_err == 1, "JH52 trailing '.' with no digits raises"]

# --- encode/decode round-trip for tiny and huge magnitudes (#557) ---
# json_encode emits %.15g, i.e. scientific notation for tiny/huge values;
# json_decode must accept its own output.
tiny is json_decode of (json_encode of ([0.000001]))
assert of [tiny[0] == 0.000001, "JH53 tiny magnitude round-trips"]
huge is json_decode of (json_encode of ([1.5e300]))
assert of [huge[0] == 1.5e300, "JH54 huge magnitude round-trips"]
neg_tiny is json_decode of (json_encode of ([0 - 0.0000000123]))
assert of [neg_tiny[0] == (0 - 0.0000000123), "JH55 negative tiny round-trips"]

print of "json hard: all passed"
109 changes: 109 additions & 0 deletions tests/test_read_line.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/bin/bash
# read_line (#558): stream-safe stdin line read. Covers: lines from a PIPE
# (the case read_text of "/dev/stdin" cannot serve — fseek fails on
# unseekable fds), null at EOF, empty-line vs EOF distinction, a final
# line without its trailing newline, CRLF stripping, a seekable redirect,
# and record/replay (tape-first: replay serves the recorded lines with no
# live stdin read — stdin is /dev/null during replay).
#
# Run directly or from run_all_tests.sh. Summary line: READLINE: N passed, M failed.
set -u
TESTS_DIR="$(cd "$(dirname "$0")" && pwd)"
SRC_DIR="$(cd "$TESTS_DIR/.." && pwd)/src"
EIGS="$SRC_DIR/eigenscript"

PASS=0
FAIL=0
TMPDIR=$(mktemp -d -t eigs_readline.XXXXXX)
trap 'rm -rf "$TMPDIR"' EXIT

ok() { echo " PASS: $1"; PASS=$((PASS+1)); }
fail() { echo " FAIL: $1${2:+ ($2)}"; FAIL=$((FAIL+1)); }

if [ ! -x "$EIGS" ]; then
echo " FAIL: eigenscript binary not found at $EIGS"
echo "READLINE: 0 passed, 1 failed"
exit 1
fi

# Three reads: two lines + EOF.
cat > "$TMPDIR/rl3.eigs" <<'EOF'
a is read_line of null
b is read_line of null
c is read_line of null
print of a
print of b
if c == null:
print of "EOF-NULL"
EOF

# Two reads: one line + EOF.
cat > "$TMPDIR/rl2.eigs" <<'EOF'
a is read_line of null
b is read_line of null
print of a
if b == null:
print of "EOF2"
EOF

# ---- pipe: the case the issue is about ----
OUT=$(printf 'hello\nworld\n' | "$EIGS" "$TMPDIR/rl3.eigs" 2>&1)
EXPECT=$'hello\nworld\nEOF-NULL'
[ "$OUT" = "$EXPECT" ] \
&& ok "pipe: lines delivered, newline stripped, null at EOF" \
|| fail "pipe lines" "out='$OUT'"

# ---- empty line is "" (not null) — distinguishable from EOF ----
cat > "$TMPDIR/rl_empty.eigs" <<'EOF'
a is read_line of null
if a == "":
print of "EMPTY-STR"
if a == null:
print of "WRONG-NULL"
EOF
OUT=$(printf '\n' | "$EIGS" "$TMPDIR/rl_empty.eigs" 2>&1)
[ "$OUT" = "EMPTY-STR" ] \
&& ok "empty line reads as empty string, not null" \
|| fail "empty line" "out='$OUT'"

# ---- final line without a trailing newline still delivered ----
OUT=$(printf 'tail' | "$EIGS" "$TMPDIR/rl2.eigs" 2>&1)
EXPECT=$'tail\nEOF2'
[ "$OUT" = "$EXPECT" ] \
&& ok "unterminated final line delivered before EOF" \
|| fail "unterminated final line" "out='$OUT'"

# ---- CRLF input: \r\n stripped as one terminator ----
OUT=$(printf 'x\r\n' | "$EIGS" "$TMPDIR/rl2.eigs" 2>&1)
EXPECT=$'x\nEOF2'
[ "$OUT" = "$EXPECT" ] \
&& ok "CRLF terminator stripped" \
|| fail "CRLF strip" "out='$OUT'"

# ---- seekable redirect works identically ----
printf 'f1\nf2\n' > "$TMPDIR/in.txt"
OUT=$("$EIGS" "$TMPDIR/rl3.eigs" < "$TMPDIR/in.txt" 2>&1)
EXPECT=$'f1\nf2\nEOF-NULL'
[ "$OUT" = "$EXPECT" ] \
&& ok "regular-file redirect" \
|| fail "file redirect" "out='$OUT'"

# ---- record/replay: tape-first (#558) ----
TAPE="$TMPDIR/rl.tape"
REC=$(printf 'one\ntwo\n' | EIGS_TRACE="$TAPE" "$EIGS" "$TMPDIR/rl3.eigs" 2>&1)
REP=$(EIGS_REPLAY="$TAPE" "$EIGS" "$TMPDIR/rl3.eigs" </dev/null 2>&1)
EXPECT=$'one\ntwo\nEOF-NULL'
if [ "$REC" = "$EXPECT" ] && [ "$REP" = "$EXPECT" ]; then
ok "replay serves recorded lines (stdin is /dev/null — no live read)"
else
fail "record/replay" "rec='$REC' rep='$REP'"
fi

# Exactly one N record per call: 2 lines + the EOF null = 3.
NREC=$(grep -c '^N read_line=' "$TAPE")
[ "$NREC" = "3" ] \
&& ok "tape carries one N record per call (2 lines + EOF null)" \
|| fail "tape N-record count" "got $NREC"

echo "READLINE: $PASS passed, $FAIL failed"
[ "$FAIL" -eq 0 ] && exit 0 || exit 1
Loading
Loading