PR #137 fiber-scheduler verification: ps2xTest build fixes + BUG-024 (open)#1
Open
TH3BACKLOG wants to merge 25 commits into
Open
PR #137 fiber-scheduler verification: ps2xTest build fixes + BUG-024 (open)#1TH3BACKLOG wants to merge 25 commits into
TH3BACKLOG wants to merge 25 commits into
Conversation
…ecomp process (#128) * feat(recomp): Reduce recompiler output memory usage Stream output generation, add low-memory config controls, and avoid pathological indirect-jump switch expansion in generated C++. Low-memory mode now avoids retaining per-instruction disassembly strings while still emitting asm comments during output generation. Output workers are bounded/configurable, combined output is streamed, and decoded buffers are released after generation. Also document the new output memory settings. * fix(recomp): added tests for unregistered JR/JALR, updated fallback logic to cover JR/JALR, moved Rabbitizer formatting into R5900Decoder
* feat: modularize elf analyzer feat: added experimental sce symbol scanner feat: change analyzer order feat: small optimizations on analyzer * feat: remove example_config.toml because its causing confusion on some people * feat: embed sce symbol but leave optional import path feat: killed skip function on analyzer but leave it so you can skip manual if you want * feat: pin elfio tag * feat: manually create string view with size * feat: update ghidra script
* feat: invert codegen hight to low convertion feat: added copy and GetEntryAddress feat: handle truncated DMAC * feat: always use address on analyzer now * feat: correct pick syscalls ID * feat: added deci2Call * feat: added wip dbcmain IOP * feat: added InitTLB feat: added err logs on thread for debug sus crash * fix: fix SetupHeap for strange cases * feat: fix incorrect SetupHeap test(it use a wrong idea on how heap allocate memory) * feat: added memalign and memalign_r feat: added GetOsdConfigParam2 and SetOsdConfigParam2 but idk if was a good idea * feat: added more memory stuff * feat: back to library functions
* [runtime] Memory swizzling rework Reimplementation of memory swizzling supporting all valid GS texture storage types. An additional optimization was done to improve the speed of lookups. Since page access is linear, we can precompute the non-linear bit (blocks and columns) for an entire page for each format. This reduces the runtime calculation to just calculating the correct page in 2d memory space and then performing a lookup in the table to get the location in memory inside a page. Since all page accesses are the same, this works across all pages. Speed improvements should be compounded by the number of accesses but there might be some reductions as a result of actually implementing formats that were not implemented before. I took care to do this with simple math. GS swizzling is complicated and optimizing it for debug builds is outside the scope of my current work. This is just the standalone code. It will be worked into the current rasterizer in a follow up commit. * [runtime] Reimplement GS host -> local transfers Reimplement the host -> local transfers using new swizzling code * [runtime] Reimplement sampling with new swizzling Reimplements the rasterizer sampling code to use the new swizzling code * [runtime] Replace fb writes with new swizzling Replace the framebuffer write code with the new swizzling code * [runtime] Reduce vram read/write to func lookup Reduces the vram read write switches to a function lookup * [tests] fix GS tests not handling depth test * [runtime] z interpolation and write support Add support for interpolating z from primitives and writing to the z buffer after z testing and masking * [runtime] local to local transfer rework * [runtime] host readout uses new sizzling code plus cleanup some dead code * [runtime] rework local->host transfer update to the new swizzling code * [runtime] update clear function with new swizzling
* fix: semaphore syscalls return sid on success instead of KE_OK The PS2 EE BIOS returns the semaphore ID on success for all semaphore syscalls, not zero. PollSema, WaitSema, SignalSema, and DeleteSema were all returning KE_OK (0) on success, which breaks games that compare the return value against the semaphore ID. DQ8's init code polls a mutex semaphore at 0x12A670 and checks the result against the semaphore ID using bne. With KE_OK returned, the comparison always fails and the game spins forever at PC 0x164978 and never starts. Other games that follow the same EE BIOS convention would hit the same problem. CreateSema already returned the ID correctly; this brings the other four operations in line with the same convention. The fix is five changes in Sync.cpp: the four return sites are updated to return sid, and the count-decrement guard in WaitSema is changed from ret == 0 to ret >= 0. The guard change is necessary because ret is now seeded to sid (a positive value) on the success path, and the old equality check would have silently stopped decrementing the semaphore count. Error paths (KE_WAIT_DELETE, KE_RELEASE_WAIT) are negative and still correctly bypass the decrement. Tests are updated to expect sid instead of KE_OK on success paths, and new test cases cover the blocked-wait-then-signal path (the actual DQ8 scenario), force-release via ReleaseWaitThread, and the count-decrement guard directly. * fix: update stale KE_OK assertions in expansion and SIF RPC tests Two test files were not updated alongside the semaphore return-value change. PollSema and SignalSema now return sid on success; update the four assertions that expected KE_OK on those success paths. * fix: tighten WaitSema count-decrement guard to ret == sid ret is seeded to sid on success and only ever overwritten with negative error codes, so ret == sid precisely expresses "this wait acquired this semaphore" — more explicit than the looser ret >= 0. Suggested by ran-j in PR review.
- missing RGBA5551 to RGBA8888 conversion in lookupClut - round depth up for triangles (Code Veronica text bug)
* feat: added ffmepg as dependency * feat: wip decoder video * feat: some perf and cleanup * feat: added generic MPEG stream notification * feat: CMakeLists.txt in ps2xStudio to configure SDL2 build options for static linking. fix: fix ffmpeg setup for linux fix: now MPEG decoder now identify that movie has ended and can play again anytime feat: better audio stub to not block games * feat: fix expansion test * feat: foo * a * feat: finally added a helper to to prevent thread starvation * feat: added basic vu0 code execution * feat: added yield Guest Execution After Wake to prevent deadlock * feat: added options on cmake for logs feat: better input for keyboard pad * feat: small corrections like top and itop vu branches etc * feat: changes * feat: working feature * feat: fatal frame iop * feat: test fix feat: z buffer fix * fix: gix GsPutIMR IMR * feat: added rl imgui * feat: added helper to get snapshot * feat: added debug panel consuming snapshots * feat: added pad snapshot feat: added RCP debug events * feat: final cleanup from old code * feat: added EE timer counter feat: applyed sound driver for Lotr feat: better check for sound driver compat layout feat: enquee and cosumed DMa cause feat: added Pad execCMd feat: update GS vsync signal flag feat: custom IOPs for LotR * feat: small cleanups * fix: fix wrong import
…tructions (#141) * feat: add sses2 support for PADDSW and PSUBSW instructions * Fixed SSE4.1 PADDSW/PSUBSW implementations
* feat: added guestBranchKind enum to categorize branch types feat: added missingFunctionPolicy enum to define behaviors for missing function scenarios refactor: added handle guest branches and report missing functions feat lookupFunction to utilize new dispatch logic and improve error handling for unregistered functions * fix: fix test conflict * feat: added debug sound driver logs * feat: emmiter for return * feat: added recompiler reporter feat: added strict diagnostics flag for heavy debug calls * feat: staticc table insted of hashmap for runtime * feat: back file to ignore * feat: explode code across helpers and classes * feat: update codegen test feat: better guest nop check * feat: fix link problem on linux * feat: fix Segmentation fault
…(#145) updateGsCsrFieldForVSync runs on the detached vsync worker and updated the FIELD bit with a non-atomic read-modify-write of GSRegisters::csr while guest threads concurrently read/write the same word (MMIO read32/read64 and the W1C handling in write32/write64) and the GIF path sets SIGNAL/FINISH. Even though the writers touch disjoint bits, a whole-word RMW loses the other side's update: a clobbered SIGNAL/FINISH set hangs a game synchronizing on GS completion, a clobbered W1C clear re-asserts a handled interrupt, and a clobbered FIELD toggle stalls interlace field polling. ThreadSanitizer flags the race on main (updateGsCsrFieldForVSync vs PS2Memory::write64 and GS::writeRegister). Make the field std::atomic<uint64_t> and perform every update as a single atomic RMW: - vsync FIELD toggle -> fetch_or / fetch_and - MMIO W1C writes -> compare_exchange loop in shared helpers (a load-then-store pair would still race); a 32-bit store to the CSR's upper dword previously bypassed the W1C special case entirely and went through the plain merge branch - both halves now share the same atomic helper with unchanged guest-visible semantics - GIF SIGNAL/FINISH -> fetch_or - reads -> load() std::atomic<uint64_t> is lock-free on all supported targets (static_assert added), the struct's size/alignment asserts are unchanged, GSRegisters is never copied by value, and default (seq_cst) ordering is used throughout - these operations are rare (vblank ticks, GIF signals, CSR MMIO), so reviewability wins over micro-optimization. New regression test: two racer threads each own one status bit (SIGNAL / FINISH) and loop 80k GIF-set + W1C-clear cycles verifying their own bit after each half-op while the vsync worker toggles FIELD. Fails 20/20 runs against the previous code, passes 50/50 with the fix, ~350ms runtime, no sanitizer needed.
* feat: added guestBranchKind enum to categorize branch types feat: added missingFunctionPolicy enum to define behaviors for missing function scenarios refactor: added handle guest branches and report missing functions feat lookupFunction to utilize new dispatch logic and improve error handling for unregistered functions * fix: fix test conflict * feat: added debug sound driver logs * feat: emmiter for return * feat: added recompiler reporter feat: added strict diagnostics flag for heavy debug calls * feat: staticc table insted of hashmap for runtime * feat: back file to ignore * feat: explode code across helpers and classes * feat: update codegen test feat: better guest nop check * feat: fix link problem on linux * feat: fix Segmentation fault * feat: added recompile replace for DMA and MMIO feat: added a clean memory helpers feat: use memory helpers across the project feat: fix ucrt on msvc * feat: undo messup merge
Replace the per-guest-thread std::thread model (g_hostThreads) with an N=1
cooperative fiber scheduler: exactly one host executor thread runs all guest
EE threads as fibers, highest priority first, FIFO within a priority, with
cooperative yield points sampled every 128 back-edges. This matches the real
EE's single-core execution model that games' kernel-object usage assumes
(semaphores as ownership tokens, priority scheduling, RotateThreadReadyQueue
rotation) and removes the cross-thread races and lock-order deadlocks of the
previous model.
Blocking syscalls park through a publish -> arm_park -> block_current
protocol whose wake_pending handshake cannot lose wakeups; all wait sites
re-check their predicates in Mesa-style loops. Host workers (IRQ, vsync,
alarm, RPC) either wake fibers through gated entry points (with
{generation,tid} tokens rejecting stale wakeups after tid reuse) or borrow
the guest token via AsyncGuestScope. Host threads carrying a guest tid get
full ThreadInfo wait bookkeeping (wait lists, ReleaseWaitThread targeting)
but never park on the fiber scheduler; they poll with bounded backoff.
Four fiber backends: ucontext (POSIX default), Win32 Fibers (_WIN32),
SceFiber (PLATFORM_VITA; compiles against a real VitaSDK, untested on
hardware, no guard page - documented), and pthread (PS2X_FIBER_PTHREAD=ON,
exists so ThreadSanitizer can instrument the scheduler). PS2X_SANITIZE wires
TSan/ASan through every target from the top-level CMakeLists.
Preserves upstream's guest-visible semantics (#136 sid-on-success semaphore
returns throughout, including older tests).
Bugs found and fixed while hardening: lost ReleaseWaitThread during Mesa
retries; alarm-worker use-after-free vs CancelAlarm; non-fiber WaitEventFlag
and WaitForNextVSyncTick returning without waiting; INTC/DMAC enable-mask
races; a terminate or resume wake dropped in the publish->park window; a
tid-recycling ABA in join_fiber; main-thread guest identity (tid 1)
restored to upstream semantics. Dogfooding against a real commercial title found
more: executor/worker token-handoff starvation, fixed across three layers
(the executor's defer-to-parked-worker predicate, yield_point's fiber-side
handoff, and a dispatchLoop back-edge hook so cross-function spins reach
both); sceSifRpcLoop's stub returning instead of parking, which
monopolized the executor; async callback stacks carved from top-of-RAM
inside the guest's own main stack, relocated to kernel-reserved memory
(0x80000-0x100000); requestStop joining workers from guest context
(deadlock - now signal-only); DMAC dispatch borrowing the guest token from
guest context (fatal abort - now dispatches inline); per-OS-thread
thread_local state shared across fibers under N=1 - the dispatch-history
ring, the syscall-override reentrancy guard, and the RPC-invoke/inline-
DMAC/MPEG-callback scratch stacks - all now fiber-owned or per-invocation;
pthread-backend fiber threads not marked as guest context, self-deadlocking
the is_guest_thread gates; and join_fiber flooring the joiner one priority
level below the target (t->priority + 1) instead of at the target's own
level, so an equal-priority sibling of the target kept the joiner off the
run-queue head and TerminateThread never returned (DQ8: main thread
terminating a same-priority worker while a sibling stayed runnable). EE
priority is "lower number = higher"; flooring at the target's own level lets
the joiner tie the siblings and cycle back via FIFO rotation to observe the
target finish and return.
Tests: passing across ucontext and pthread backends (deterministic across
repeated runs), TSan-clean (zero warnings) with the three documented
intentional hardware-modeled vsync/GS-CSR reports suppressed via
ps2xTest/tsan.supp, ASan/LSan clean. Scheduler suites cover the
wakeup-window protocol, suspend/resume gating, tid reuse, terminate windows,
priority rotation and join floors (including a SchedulerJoinStarvation
regression: a higher-priority joiner terminating a target with an
equal-priority runnable sibling, asserting TerminateThread returns within a
bounded number of rounds), shutdown ordering, and borrowed workers, with
regression tests that fail against the unfixed code. The workload dogfood
regression suite (ps2_scheduler_workload_regression_tests.cpp) adds targeted
coverage for the bugs above: token handoff, RPC-loop park, recovery
isolation, guest-context stop, kernel stack pool, DMAC guest dispatch,
stack isolation, and override isolation.
…S/current, drop SceFiber tls_pending_self
…d prologue, self-exit mark helper
…tackPool allocator
… lock in block_current borrowed-worker path
…en-drop, fix log-once backoff warning - Extract withGuestTokenDropped() as the single place that knows the async_guest_end/begin bracketing for a non-fiber wait pause; both NonFiberBackoff::step's sleep and nonFiberBlockBackoff's yield now route through it instead of duplicating the drop/reacquire branch. - Fix a real bug in NonFiberBackoff::step: the cap-reached warning was meant to fire once, but spins froze at kMaxSpins so `spins == kMaxSpins` stayed true and it re-fired every iteration after the cap. Now increments-and-compares so it fires exactly once. Also drop the now-redundant spins < kMaxSpins guard around the delay ramp since delay already self-clamps at the 1ms cap. - Fold arm_park + block_current + the conditional step into NonFiberBackoff::wait(onFiber), and make step() private so no caller can feed it a fiber block result. Route WaitSema, WaitEventFlag, SleepThread, and WaitForNextVSyncTick's two block_current sites through it, preserving each site's exact lock handling and the arm-before-block ordering.
…iberToken, context-carrying stop callback
…elpers, markDeletedAndWake
…he workload regression suite
…ers in the expansion suite
After hard-resetting to pr-137-latest, the new tree's root sits one level higher than the old layout, surfacing ~72 untracked paths that are local-only tools, dumps, and asset directories rather than repo content.
classified.csv/gaps.csv/suspects.csv/PNG are 2026-06-22 gap-scan tool output; the PSD is an unrelated UI mockup. Neither belongs in the tracked source tree.
TH3BACKLOG
force-pushed
the
work/laptop-session-0519
branch
from
July 12, 2026 00:37
5586c13 to
73d4e49
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Note: this branch also carries the prior in-progress PR #137 integration work (commits 6a23062/9656ab8 etc.) already pushed to main on this fork in earlier sessions -- main here has been significantly behind for a while, so this PR's diff spans the full session history, not just this fix.
Test plan