Extend @gba-kit/debug-info for address→symbol maps

.changeset/declaration-shapes.md

@@ -0,0 +1,30 @@
+---
+'@gba-kit/debug-info': minor
+---
+
+Read what a name is DECLARED as — shapes, signatures and macro names — from either byte order.
+
+- `variableShape(name)` classifies a global/static as `scalar | pointer | array | struct`,
+  resolved through typedef/cv chains: `volatile`/`const`, array `elemSize`/`elemSigned`/`length`,
+  and the pointer's `pointee` (the name `struct()` resolves, its size, its own qualifiers).
+  A `typedef struct {…} T;` is named by its alias; `null` doubles as the "is this name
+  declared in the project headers?" probe.
+- `struct()` members carry the declaration facts layout alone cannot: `signed`, `pointer`,
+  `pointeeSize`/`pointeeSigned`, `volatile`/`const`, and array `elemSize`/`elemSigned`/`length`.
+  Every key is absent when the DWARF does not determine it.
+- `functionSignature(name)` returns a COMPILED function's return and parameter types
+  (`low_pc` is the witness): `null` means "this ELF did not compile it", never "it takes
+  no arguments". gcc's abstract/concrete split at `-O1+` resolves to one definition.
+- `DebugInfo.macros` / `parseDebugMacinfo` read the `-g3` macro table (DWARF 2/3
+  `.debug_macinfo`, the self-contained form) — the only place an address-cast `#define`
+  name survives, since a macro leaves no symbol and no DIE. A truncated stream yields a
+  sound prefix, never a corrupted entry.
+- Big-endian ELF/DWARF end to end — bitfields are allocated from the MSB end and reported
+  that way — and RELA relocations are applied to `.debug_*` in relocatable objects.
+- `.debug_line` is walked by its own `DW_LNE_end_sequence` terminators: agbcc (GCC 2.95)
+  mispredicts `unit_length`, which used to cost every row after the first short unit.
+- Producer-dialect fixes, pinned on committed toolchain output: DWARF 2/3's `DW_FORM_flag`
+  decodes as a boolean (every declaration/prototyped test was inert, so a forward-declared
+  struct could shadow its own definition by link order); GCC 2.95's `0xffffffff` upper
+  bound reads as zero-length, not 2^32 elements; and a DECLARATION's `[1]` array is agbcc's
+  unsized-extern spelling, reported as `length: null`.

.github/workflows/ci.yml

@@ -8,12 +8,21 @@ jobs:
         with:
           submodules: recursive
 
-      # GBA cross toolchain for the @gba-kit/debug-info test ELFs
+      # ARM cross toolchain for the @gba-kit/debug-info little-endian test ELFs
       - name: Install arm-none-eabi GCC 14
         uses: carlosperate/arm-none-eabi-gcc-action@v1
         with:
           release: '14.2.Rel1'
 
+      # Big-endian cross toolchains for the mips-min / ppc-min test projects. The
+      # runner is x86 Linux, so these stock Ubuntu packages run natively — no
+      # Docker or qemu. vitest's globalSetup rebuilds every project when CI is set.
+      - name: Install big-endian cross toolchains
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y gcc-mips-linux-gnu binutils-mips-linux-gnu \
+            gcc-powerpc-linux-gnu binutils-powerpc-linux-gnu
+
       # Build the agbcc (GCC 2.95) compiler from the submodule once
       - name: Resolve agbcc commit
         id: agbcc-rev

packages/debug-info/README.md

@@ -1,17 +1,37 @@
 # @gba-kit/debug-info
 
-Parse ELF symbols and DWARF debug info from a (`-g`-built) GBA ELF, and answer
-the queries a source-level debugger needs:
+Parse ELF symbols and DWARF debug info from a (`-g`-built) ELF32, and answer the
+queries a source-level debugger needs:
 
 - **PC → function** (`pcToFunction`) — from `.symtab`, so it covers every linked
   function, including `INCLUDE_ASM` stubs with no DWARF.
 - **name → address** / **address → symbol** (`symbolToAddress`, `addressToSymbol`).
 - **PC → C `file:line`** (`pcToSource`) — from the DWARF `.debug_line` table.
+- **type layout** (`struct`, `structMember`, `enumValues`) and **declaration shape**
+  (`types.variableShape`) — from `.debug_info`.
+- **function signatures** (`types.functionSignature`) — return and parameter types of
+  every function the ELF compiled from C; `null` means "not compiled here", never
+  "takes no arguments".
+- **the `-g3` macro table** (`macros`, `parseDebugMacinfo`) — the only place an
+  address-cast `#define gCounter (*(u16 *)0x03001234)` name survives: a macro leaves
+  no symbol and no DIE.
 
-It's a small, dependency-free, DOM-free parser meant to be shared by the headless
-runtime, the scripting engine, and the webapp's source debug view. The shipped
-`.gba` ROM carries no debug info (`objcopy -O binary` strips it); load the sidecar
-ELF — its loadable bytes are identical to the ROM, so addresses line up.
+This is the general ELF/DWARF piece of gba-kit, not a GBA-only one:
+
+- **both byte orders** — the order is read from `e_ident` and threaded through the
+  container and the DWARF payload alike. Big-endian bitfields are allocated from
+  the most significant end of the storage unit, and are reported that way.
+- **linked ELFs and relocatable objects** — in a `.o` whose relocations are
+  RELA-style (PowerPC), the raw `.debug_*` fields are zeros and the real values sit
+  in `.rela.<section>` addends; those are applied on read.
+- **DWARF 2 through 5**, as emitted by anything from GCC 2.95 to GCC 14.
+
+It is exercised against real ARM, MIPS and PowerPC toolchain output (see
+[Testing](#testing)). It's a small, dependency-free, DOM-free parser, shared by
+the headless runtime, the scripting engine, and the webapp's source debug view.
+For the GBA case: the shipped `.gba` ROM carries no debug info
+(`objcopy -O binary` strips it); load the sidecar ELF — its loadable bytes are
+identical to the ROM, so addresses line up.
 
 ## Usage
 
@@ -26,6 +46,13 @@ di.pcToSource(0x0801466a);
 di.pcToFunction(0x0801466a)?.name; // 'PlayerRespawnOrDeath'
 di.symbolToAddress('InitLevelGameplay'); // 0x0800ca0c
 di.addressToSymbol(0x0801466a); // { name: 'PlayerRespawnOrDeath', offset: 0x46 }
+
+di.types.variableShape('gSineTable');
+// → { kind: 'array', elemSize: 2, elemSigned: true, length: null, const: true, volatile: false }
+di.types.functionSignature('ReadUnalignedU16');
+// → { returns: { size: 4, signed: false }, params: [{ name: 'ptr', size: 4, pointer: true, ... }], prototyped: true, ... }
+di.macros.find((m) => m.name === 'gGfxStreamBuffer'); // (a -g3 build records the macro table)
+// → { name: 'gGfxStreamBuffer', body: '(*(u32 *)0x030007C8)', line: 191 }
 ```
 
 ## Develop
@@ -37,14 +64,23 @@ pnpm --filter @gba-kit/debug-info test
 
 ## Testing
 
-`@gba-kit/debug-info` is tested against real GBA ELFs that are **committed** to the
-repo (`packages/debug-info/test-projects/*/build/`), so tests run with no cross
-toolchain.
+`@gba-kit/debug-info` is tested against real ELFs from four minimal projects,
+**committed** to the repo (`packages/debug-info/test-projects/*/build/`), so tests
+run with no cross toolchain:
 
-You only need to rebuild those ELFs when you change a test project's sources,
-and that's a per-project step (see[test-projects/README](packages/debug-info/test-projects/README.md)):
+| Project         | Toolchain                       | Target                 |
+| --------------- | ------------------------------- | ---------------------- |
+| `agbcc-min`     | agbcc (GCC 2.95), git submodule | ARM, little-endian     |
+| `devkitarm-min` | `arm-none-eabi-gcc` (GCC 14)    | ARM, little-endian     |
+| `mips-min`      | `mips-linux-gnu-gcc`            | MIPS o32, big-endian   |
+| `ppc-min`       | `powerpc-linux-gnu-gcc`         | PowerPC 32, big-endian |
+
+`ppc-min` vendors a relocatable `main.o` as well as the linked ELF — the artifact
+shape that exercises the RELA path.
 
-- `agbcc-min` — `cd packages/debug-info/test-projects/agbcc-min && ./setup.sh` (builds the agbcc submodule)
-- `devkitarm-min` — `cd packages/debug-info/test-projects/devkitarm-min && ./build.sh` (builds in **Docker**, so no local devkitARM needed)
+You only need to rebuild those ELFs when you change a test project's sources,
+and that's a per-project step (see [test-projects/README](test-projects/README.md)):
+`agbcc-min` builds the agbcc submodule via `./setup.sh`, the other three build in
+**Docker** via `./build.sh`, so no local cross toolchain is needed.
 
-CI rebuilds both from scratch on every run to re-validate the toolchains.
+CI rebuilds all four from scratch on every run to re-validate the toolchains.

packages/debug-info/src/__tests__/debug-line.spec.ts

@@ -0,0 +1,161 @@
+/**
+ * `.debug_line` section-walking contract.
+ *
+ * The line table is a *concatenation* of independent units, so the parser's job is
+ * as much finding the next unit as decoding one. These tests take the real DWARF-2
+ * bytes agbcc (GCC 2.95) emitted for `test-projects/agbcc-min` and perturb the
+ * section the way real producers do, asserting the decoded rows never change and
+ * that no perturbation costs more than the unit it belongs to.
+ *
+ * The load-bearing case is the first one: agbcc sizes a unit by *predicting* the
+ * encoded length of each statement, and mispredicts, so `unit_length` can stop a
+ * few bytes short of the program it describes (in pokeemerald 28 of 303 units, by
+ * 1–4 bytes and one by 51). Clamping to the declared end leaves the cursor
+ * mid-statement, and the next unit's header is then read as line-program bytes —
+ * from there a walk runs off the section and every row after it is lost. The
+ * DW_LNE_end_sequence terminator, not the declared length, is what ends a unit.
+ */
+import { readFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+import { type LineRow, parseDebugLine } from '../debug-line.js';
+import { ElfFile } from '../elf.js';
+
+const here = dirname(fileURLToPath(import.meta.url));
+const elfPath = join(here, '..', '..', 'test-projects', 'agbcc-min', 'build', 'min.elf');
+
+const elf = ElfFile.parse(new Uint8Array(readFileSync(elfPath)));
+/** Real agbcc (GCC 2.95) DWARF-2 line table: two units (main.c, util.c). */
+const section = elf.sectionData('.debug_line')!;
+const pristine = parseDebugLine(section).rows;
+
+const u32At = (bytes: Uint8Array, off: number): number =>
+  new DataView(bytes.buffer, bytes.byteOffset).getUint32(off, true);
+const setU32 = (bytes: Uint8Array, off: number, v: number): void =>
+  new DataView(bytes.buffer, bytes.byteOffset).setUint32(off, v, true);
+
+const concat = (...parts: Uint8Array[]): Uint8Array => {
+  const out = new Uint8Array(parts.reduce((n, p) => n + p.length, 0));
+  let at = 0;
+  for (const p of parts) {
+    out.set(p, at);
+    at += p.length;
+  }
+  return out;
+};
+
+/** A unit header we deliberately cannot decode, with a valid `unit_length`. */
+function unmodellableUnit(version: number, body = 24): Uint8Array {
+  const unit = new Uint8Array(4 + 2 + body);
+  const view = new DataView(unit.buffer);
+  view.setUint32(0, 2 + body, true); // unit_length covers everything after itself
+  view.setUint16(4, version, true);
+  unit.fill(0xaa, 6); // header we never read
+  return unit;
+}
+
+/** A 64-bit DWARF unit: the 0xffffffff escape then a 64-bit unit_length. */
+function dwarf64Unit(body = 24): Uint8Array {
+  const unit = new Uint8Array(4 + 8 + body);
+  const view = new DataView(unit.buffer);
+  view.setUint32(0, 0xffffffff, true);
+  view.setUint32(4, body, true); // low half of the 64-bit length
+  view.setUint32(8, 0, true); // high half
+  unit.fill(0xaa, 12);
+  return unit;
+}
+
+const rowsOf = (bytes: Uint8Array): LineRow[] => parseDebugLine(bytes).rows;
+
+it('the fixture is the shape these tests assume (two units, real rows)', () => {
+  expect(u32At(section, 0) + 4).toBeLessThan(section.length); // a second unit follows
+  expect(pristine.length).toBeGreaterThan(20);
+  expect(new Set(pristine.map((r) => r.file.split('/').pop()))).toEqual(new Set(['main.c', 'util.c']));
+});
+
+describe('a unit_length that undercounts its own line program (agbcc / GCC 2.95)', () => {
+  // The producer bug is a *size misprediction*, so the bytes are correct and only
+  // the length field is short: shortening it must change nothing we decode.
+  it.each([1, 2, 3, 4, 7])('recovers when the first unit is declared %d bytes short', (missing) => {
+    const short = section.slice();
+    setU32(short, 0, u32At(short, 0) - missing);
+
+    // The whole point: the *following* unit is still found, so no rows are lost.
+    expect(rowsOf(short)).toEqual(pristine);
+  });
+
+  it('recovers on the last unit too (nothing follows it)', () => {
+    const lastStart = 4 + u32At(section, 0);
+    const short = section.slice();
+    setU32(short, lastStart, u32At(short, lastStart) - 3);
+
+    expect(rowsOf(short)).toEqual(pristine);
+  });
+});
+
+describe('units we cannot decode are skipped by their own unit_length', () => {
+  it('keeps the rest of the section when a DWARF 5 unit comes first', () => {
+    // DWARF 5 rewrote the header (address_size/segment_selector_size, and typed
+    // directory/file entry formats), so its bytes are not a v2–v4 header.
+    expect(rowsOf(concat(unmodellableUnit(5), section))).toEqual(pristine);
+  });
+
+  it('keeps the rest of the section when a 64-bit DWARF unit comes first', () => {
+    expect(rowsOf(concat(dwarf64Unit(), section))).toEqual(pristine);
+  });
+
+  it('steps over zero-word padding between units', () => {
+    const pad = new Uint8Array(8); // two zero unit_lengths
+    expect(rowsOf(concat(pad, section))).toEqual(pristine);
+  });
+});
+
+it('finds the program by header_length, not by walking the file table', () => {
+  // Insert padding between the end of the file-name table and the program start,
+  // growing header_length (and unit_length) to match — exactly what an unmodelled
+  // header field would look like. A parser that starts the program where the file
+  // table happened to end would run the padding as opcodes.
+  const pad = 6;
+  const programStart = 10 + u32At(section, 6);
+  const grown = concat(section.slice(0, programStart), new Uint8Array(pad).fill(0xaa), section.slice(programStart));
+  setU32(grown, 0, u32At(grown, 0) + pad); // unit_length
+  setU32(grown, 6, u32At(grown, 6) + pad); // header_length
+
+  expect(rowsOf(grown)).toEqual(pristine);
+});
+
+describe('a section that cannot be walked degrades instead of throwing', () => {
+  // parseDebugLine is the only thing standing between a hostile .debug_line and
+  // DebugInfo.fromElf, which must still deliver symbols and types. It reports what
+  // it decoded and stops — it never throws, so callers need no rescue wrapper.
+  it('keeps every row of the complete units when the last unit is truncated', () => {
+    const cut = section.slice(0, section.length - 12);
+    const rows = rowsOf(cut);
+
+    const fromUnitOne = (rs: LineRow[]) => rs.filter((r) => r.file.endsWith('main.c'));
+    expect(fromUnitOne(rows)).toEqual(fromUnitOne(pristine)); // unit 1 is untouched
+    expect(rows.length).toBeLessThan(pristine.length); // unit 2 loses its cut-off tail
+  });
+
+  it('returns no rows for garbage, and terminates', () => {
+    const garbage = new Uint8Array(4096);
+    for (let i = 0; i < garbage.length; i++) {
+      garbage[i] = (i * 37) & 0xff;
+    }
+    expect(() => rowsOf(garbage)).not.toThrow();
+  });
+
+  it('decodes what it can of a unit whose length runs past the section', () => {
+    const overlong = section.slice(0, 64); // unit 1 claims 261 bytes; 64 are here
+    const rows = rowsOf(overlong);
+
+    expect(rows.length).toBeGreaterThan(0);
+    expect(rows).toEqual(pristine.slice(0, rows.length)); // real rows, no invented ones
+  });
+
+  it('handles an empty section', () => {
+    expect(rowsOf(new Uint8Array(0))).toEqual([]);
+  });
+});

packages/debug-info/src/__tests__/debug-macro.spec.ts

@@ -0,0 +1,92 @@
+/**
+ * `.debug_macinfo` — the macro table, against a real artifact.
+ *
+ * `devkitarm-min/build/macinfo.o` is that project's `main.c` compiled the way a decomp's
+ * macro sidecar is (`-gdwarf-2 -g3 -gstrict-dwarf`; see the project Makefile): one
+ * self-contained `.debug_macinfo` with inline strings, in a relocatable `.o` — the same
+ * artifact shape a real project grafts from. The fixture macros live at the END of that
+ * `main.c` and are asserted by exact line number (append there, never insert above).
+ * `readelf --debug-dump=macro` agreed on every define when the artifact was added.
+ *
+ * The macro channel exists for one decomp idiom above all: a fixed RAM cell named by an
+ * address-cast `#define` instead of an extern. Such a name is in no symbol table and has
+ * no DIE — the preprocessor's record is the only place it survives.
+ */
+import { readFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { describe, expect, it } from 'vitest';
+
+import { parseDebugMacinfo } from '../debug-macro.js';
+import { ElfFile } from '../elf.js';
+
+const here = dirname(fileURLToPath(import.meta.url));
+const project = join(here, '..', '..', 'test-projects', 'devkitarm-min', 'build');
+
+const elf = ElfFile.parse(new Uint8Array(readFileSync(join(project, 'macinfo.o'))));
+const section = elf.sectionData('.debug_macinfo')!;
+const macros = parseDebugMacinfo(section);
+const byName = (name: string) => macros.find((m) => m.name === name);
+
+describe('parseDebugMacinfo on a real -gdwarf-2 -g3 object', () => {
+  it('reads the address-cast fixtures verbatim, with their lines', () => {
+    expect(byName('REG_DISPSTAT')).toEqual({
+      name: 'REG_DISPSTAT',
+      body: '(*(volatile unsigned short *)0x04000004)',
+      line: 157,
+    });
+    expect(byName('g_save_slot')).toEqual({
+      name: 'g_save_slot',
+      body: '(*(unsigned char *)0x03007FF0)',
+      line: 158,
+    });
+    expect(byName('EWRAM_BASE')).toEqual({ name: 'EWRAM_BASE', body: '0x02000000', line: 159 });
+  });
+
+  it('keeps a function-like macro as one name, the parameter list as recorded', () => {
+    // DWARF stores the define as written post-lex: params squeezed, body spacing kept.
+    // Splitting "CLAMP(x,lo,hi)" further would invent structure the section lacks.
+    expect(byName('CLAMP(x,lo,hi)')).toEqual({
+      name: 'CLAMP(x,lo,hi)',
+      body: '((x) < (lo) ? (lo) : (x) > (hi) ? (hi) : (x))',
+      line: 160,
+    });
+  });
+
+  it('reports a body-less define with an empty body, not a missing entry', () => {
+    expect(byName('NO_BODY')).toEqual({ name: 'NO_BODY', body: '', line: 161 });
+  });
+
+  it('reports definitions in stream order', () => {
+    const lines = ['REG_DISPSTAT', 'g_save_slot', 'EWRAM_BASE', 'CLAMP(x,lo,hi)', 'NO_BODY'].map((n) =>
+      macros.findIndex((m) => m.name === n),
+    );
+    expect(lines.every((i) => i >= 0)).toBe(true);
+    expect([...lines].sort((a, b) => a - b)).toEqual(lines);
+  });
+
+  it('carries the compiler built-ins at line 0 alongside the user macros', () => {
+    // The exact built-in set is the compiler's business (do not pin a total): assert the
+    // class exists and is large, which is what makes "grep the table" a real capability.
+    const builtins = macros.filter((m) => m.line === 0);
+    expect(builtins.length).toBeGreaterThan(300);
+    expect(byName('__VERSION__')).toBeDefined();
+  });
+
+  it('a truncated stream yields a sound prefix, never a throw', () => {
+    // Sections get grafted between tools; the contract is that every returned entry was
+    // really read. Cutting the stream anywhere must give a prefix of the full parse.
+    for (const cut of [section.length >> 2, section.length >> 1, section.length - 3]) {
+      const partial = parseDebugMacinfo(section.slice(0, cut));
+      expect(partial.length).toBeLessThanOrEqual(macros.length);
+      expect(partial).toEqual(macros.slice(0, partial.length));
+    }
+  });
+});
+
+describe('the -g3 requirement', () => {
+  it('a plain -g ELF has no .debug_macinfo at all', () => {
+    const plain = ElfFile.parse(new Uint8Array(readFileSync(join(project, 'min.elf'))));
+    expect(plain.sectionData('.debug_macinfo')).toBeUndefined();
+  });
+});