linbofs: NVMe controller resets on Lenovo L16 Gen1 (Intel Core Ultra 5 / Meteor Lake)

[HappyBasher]

Problem

On a Lenovo L16 Gen1 (Intel Core Ultra 5 125U, Meteor Lake), LINBO boots and starts loading/writing the image, but then hangs at 0% with repeated dmesg output:

nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x11
nvme0n1: I/O Cmd(0x2) @ LBA ..., I/O Error (sct 0x3 / sc 0x71)
I/O error, dev nvme0n1, sector ..., op 0x0:(READ) ...

The resets recur at a very regular ~30s interval (matching the default NVMe io_timeout), hundreds of times per session.

Diagnosis

• Adding nvme_core.default_ps_max_latency_us=0 and pcie_aspm=off via KernelOptions in the group's start.conf did not fix the issue (confirmed active in the boot Command line, resets persisted at the same cadence).
• Comparing lsmod from a working native Ubuntu 26.04 boot on the same machine against linbofs's harvested kernel modules showed linbofs is missing an entire class of Meteor Lake platform power-management drivers that Ubuntu's generic kernel loads:
  • pinctrl_meteorlake (and the whole drivers/pinctrl/intel/* family)
  • intel_pmc_core, intel_pmc_core_pltdrv, intel_pmc_ssram_telemetry
  • intel_vsec, pmt_telemetry, pmt_discovery, pmt_class
  • idma64

These drivers manage GPIO-controlled ACPI power resources and PMC handshakes used on Ice/Tiger/Alder/Meteor Lake platforms to bring devices behind the PCH (including NVMe) back up cleanly after D3/APST power transitions. Without the matching pinctrl driver, the platform can't correctly service the relevant ACPI _PS0 power-resource methods, which lines up with the observed reset pattern that plain NVMe-side kernel parameters could not fix.

Fix

Added the missing modules to build/config/modules.d/:

• new kernel.drivers.pinctrl (all Intel per-platform pinctrl drivers, Alder Lake through Meteor Lake, so the whole recent Intel generation is covered, not just this one device)
• idma64.ko.zst added to kernel.drivers.dma
• intel-vsec, intel_pmc_core (+ dependencies), pmt_* added to kernel.drivers.platform

Verification

Package rebuilt and installed; LINBO boot on a KVM test client works fine. Awaiting real-hardware confirmation on the reported Lenovo L16 Gen1.

[HappyBasher]

Co-authored by Claude.

[HappyBasher]

Follow-up analysis on the new hardware logs (linbo 7.4.9 vs 7.4.11 vs native Ubuntu 26.04)

Compared dmesg_linbo_7.4.9.txt (before the #165 fix), dmesg_linbo_7.4.11.txt (after the fix), dmesg_ubuntu26.04.txt, and lsmod_ubuntu26.04.txt from cache/Lenovo_L16_Problem/.

The fix from 8ce8f70 shipped correctly, but did not change the failure.

• intel_vsec, idma64, intel_pmc_core, and intel_pmc_ssram_telemetry all load and print their init messages in the 7.4.11 capture — the added modules are present and active in linbofs.
• Despite that, the reset signature (controller is down; will reset: CSTS=0x3, PCI_STATUS=0x11, ~30s cadence matching the default nvme io_timeout) is identical in 7.4.9 (before the fix) and 7.4.11 (after the fix).

Reproducible detail that changes the diagnosis: in both captures — taken on different days — the first reset occurs at almost the same LBA:

• 7.4.9: LBA ≈ 465,272,320
• 7.4.11: LBA ≈ 465,488,128

Both are ≈238 GB into nvme0n1. A power-management/APST/pinctrl-driver bug would not reliably fail at the same disk offset across independent sessions — a physical bad-sector / firmware-remapped defect region on the drive would. Combined with the PMC/pinctrl drivers being confirmed loaded while the fault persisted unchanged, this strongly suggests the original #165 root-cause theory (missing Meteor Lake power-sequencing drivers) was not the actual (or at least not the dominant) cause.

Additionally, per reset cycle the LBA only advances by a few thousand blocks (~1–15 MB) before hanging again — the controller isn't sustaining real throughput between resets, it's barely completing anything before faulting again. That fits a failing controller/media region better than thermal throttling under sustained load.

Data-risk finding: the 7.4.11 capture spans ~88 hours of continuous retries (347 logged resets before the dmesg ring buffer wrapped — the real count over 88h is likely much higher). This already produced real corruption: ntfs3(nvme0n1p3): mft corrupted, EXT4-fs: warning: mounting fs with errors, and near the end buffer_io_error: ... lost async page write, Failed to configure AEN, reading non-mdts-limits failed: -4. The machine was left retrying the same fault for days.

Recommended next steps

1. Stop further imaging attempts on this specific unit until the drive is checked — continued retries risk more corruption/wear on a possibly-failing device.
2. Boot the L16 from a live Ubuntu/rescue USB and run nvme smart-log /dev/nvme0, nvme error-log /dev/nvme0, and smartctl -a /dev/nvme0 to check for media errors, reallocated sectors, or thermal-throttle counters around the ~238 GB offset.
3. Try to reproduce outside LINBO: read through that offset under native Ubuntu (e.g. dd if=/dev/nvme0n1 of=/dev/null bs=1M skip=... positioned at LBA ~465,000,000). If the same CSTS=0x3 reset reproduces there too, it rules out LINBO/linbofs entirely and points to a hardware/firmware fault on this unit.
4. If confirmed, this looks like an RMA/drive-replacement case for this specific machine rather than a LINBO defect. The module additions from linbofs: NVMe controller resets on Lenovo L16 Gen1 (Intel Core Ultra 5 / Meteor Lake) #165 remain good general hygiene for Meteor Lake platforms, but shouldn't be expected to resolve this particular report.

Co-authored by Claude.

[HappyBasher]

Correction to my previous comment: "bad drive" was likely the wrong conclusion

Thomas asked whether this class of failure is known to happen on this hardware under native Ubuntu 26.04. That prompted a closer look, and it changes the read on this.

The LBA-locality argument I made previously is weak. Both 7.4.9 and 7.4.11 run the identical deterministic boot sequence (same partition scan → same hwinfo collection → same start of the image/checksum read), so hitting the same disk offset at nearly the same elapsed time is expected from the boot sequence itself, not evidence of a bad sector. I overweighted that.

Closer precedent found: a live LKML thread from November 2025, "controller is down; will reset" on SK Hynix NVMe drive in Lenovo IdeaPad Pro 5 (see also the reply thread). Same symptom class on a recent Lenovo laptop:

• Resets after idle periods, same ~30s io_timeout cadence, same kernel hint ("does your device have a faulty power saving mode enabled?"), same risk of filesystem corruption (btrfs forced read-only there; ntfs3/ext4 corruption here).
• They also tried nvme_core.default_ps_max_latency_us=0 pcie_aspm=off first — the same two parameters already tested on the L16 — and it was not sufficient on its own.
• Full stability only came after adding a third parameter: pcie_port_pm=off (disables PCIe root-port power management, distinct from link-level ASPM).
• The NVMe maintainer (Keith Busch) subsequently added a device-specific NVME_QUIRK_NO_APST kernel quirk for that drive, since blanket-disabling port PM is a blunt workaround.

No L16/Meteor-Lake-specific report was found by name, but the pattern match (recent Lenovo laptop + recent NVMe + reset-after-idle + 30s cadence + the 2-parameter workaround being insufficient by itself) lines up closely enough that this looks like the same class of bug, not a failing drive on this specific unit.

Also worth noting: the native Ubuntu 26.04 capture we have is only ~313s from what looks like a live/installer session — not long enough to include an idle period, so it doesn't actually establish that the hardware is fine under Ubuntu. That comparison is inconclusive rather than exculpatory, and shouldn't be used to argue the fault is LINBO-side either.

Revised next steps

1. Retest with pcie_port_pm=off added to the existing KernelOptions (nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off) before drawing any hardware-fault conclusion.
2. If that still doesn't fully stabilize it, check whether a newer/HWE kernel already carries an NVME_QUIRK_NO_APST-style quirk entry for this drive's vendor/model — that's where this class of bug tends to get fixed upstream.
3. Hold off on the RMA/drive-replacement recommendation from my previous comment until the above is tried; it was premature given this precedent.

Co-authored by Claude.

[HappyBasher]

Update: pcie_port_pm=off does not fix it — power management is now likely ruled out

Tested with all three parameters active: ... nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off. No change — same ~30s reset cadence as before.

This is a meaningful result: those three parameters are exactly what fully stabilized the Lenovo IdeaPad Pro 5 case referenced in my last comment. Since disabling APST, link-level ASPM, and PCIe root-port PM together still doesn't help here, this L16 case is very likely not the same host-controlled-power-management bug, or at least not one the host can work around the same way. Host-driven power management now looks largely ruled out as the cause.

What's left as candidates:

• The drive's own autonomous/internal behavior (proprietary idle power-saving or background GC/thermal-throttle pauses not gated by APST/ASPM at all).
• A genuine media/firmware defect in this drive.
• Lost/misdelivered interrupts (MSI-X) on this platform, unrelated to power state.

Next diagnostic step: ask the NVMe controller directly

Rather than continuing to guess from host-side dmesg, the next step is the controller's own error log, which is authoritative about whether the drive itself saw a failure:

nvme error-log /dev/nvme0
nvme smart-log /dev/nvme0
nvme id-ctrl /dev/nvme0 | grep -i apsta   # confirm the drive actually advertises APST at all

• If error-log shows real entries correlating with the reset timestamps → genuine drive-side fault, supports pursuing an RMA.
• If the error log is clean despite the resets happening → the drive itself doesn't think anything failed, which shifts suspicion back to the host/platform side (lost interrupts, PCIe signal integrity) rather than the SSD.

Also useful: check whether the reset still onsets at the same ~450–550s mark / same LBA region as in the earlier captures. If so, it's worth identifying what LINBO is actually doing at that point in the boot sequence (likely the partition checksum/verify pass) to see if it's reproducibly touching the same data region — that would narrow things down further regardless of which side turns out to be at fault.

Could you run the three nvme-cli commands above (ideally from a live/rescue boot right after reproducing the resets, before power-cycling) and post the output here?

Co-authored by Claude.

[HappyBasher]

NVMe error-log/smart-log came back clean — and two more targeted tests

Thanks for the nvme-cli output in cache/Lenovo_L16_Problem/nvme-log/. This is decisive:

• id-ctrl: apsta: 0x1 — drive does support/advertise APST.
• smart-log: media_errors: 0, critical_warning: 0, available_spare: 100%, percentage_used: 0%, normal temperature (38°C), zero thermal-throttle transitions, num_err_log_entries: 0.
• error-log: all 64 entries are empty (status_field: 0 — Successful Completion) — the drive has never once logged an internal error, despite the host seeing hundreds of "controller is down" resets.

This is a healthy drive from its own point of view. If the SSD's firmware never recorded a fault, this doesn't look like degrading media or a drive-side firmware defect — it points back to the host side: the Linux nvme driver is declaring the controller dead and force-resetting it without the drive itself ever seeing anything wrong. That's the classic signature of a missed/lost command completion (e.g. an interrupt that never reached the host), not a real device hang.

A confound found while checking this: kernel version mismatch

The native Ubuntu capture (dmesg_ubuntu26.04.txt) is running kernel 7.0.0-14-generic, while linbofs harvests and ships 7.0.0-28-generic (confirmed via cache/kversion and each file's Linux version line) — two different kernel builds. I pulled the actual Ubuntu kernel changelog between those versions and didn't find an obvious matching NVMe/interrupt/IOMMU fix, but the version mismatch itself means the earlier "does native Ubuntu also fail" comparison was never truly apples-to-apples.

Two complementary follow-up tests

1. Linbo with the full, unfiltered kernel module tree. kernel-harvester.sh only copies modules explicitly listed in build/config/modules.d/*, not the complete /lib/modules/<kernelver> tree (kernel-harvester.sh). That's a manually curated subset — the same mechanism that under-shipped modules for the original #165 fix. If any currently-listed module silently depends on something we didn't include, modprobe/udev's kmod builtin can fail quietly without showing up in dmesg. A one-off diagnostic build that skips the modules.d filtering and copies every module Ubuntu ships for this kernel would rule out "missing/broken dependency chain" entirely, regardless of which specific driver anyone might guess at next.

Note this does not touch the kernel binary itself (linbo64/vmlinuz) — IOMMU/VT-d/ACPI-core init happen before the initramfs is even unpacked in the boot log, so those are compiled in (=y), not modules, and are identical either way. This test only targets the "is a loadable driver missing" hypothesis.

2. Native Ubuntu, but specifically on kernel 7.0.0-28-generic (not -14), with a workload that includes a real idle period before sustained I/O (e.g. let it sit idle for a few minutes, then start a large sequential read through the ~238 GB region). This isolates the kernel-version/boot-path question the first test can't reach — GRUB chainload vs. direct EFI boot, ACPI table timing, microcode load timing, real systemd/userspace vs. Linbo's busybox/init.sh.

Running both would narrow this down to either "a specific still-missing linbofs driver" (test 1 positive) or "something in the kernel version or Linbo's boot path itself, unrelated to module completeness" (test 1 clean, test 2 diverges from Linbo's behavior).

Co-authored by Claude.