install --nand leaves 0 UBI user volumes (rootfs panic); raw 'nand write' of rootfs.ubi fixes it

[widgetii]

Summary

defib install --nand (non-V500 path) reports success, but the resulting UBI has zero user volumes — the kernel attaches ubi0 (all PEBs good) yet root=ubi0:rootfs fails with VFS: Cannot open root device "ubi0:rootfs" ... error -19 → kernel panic. Writing rootfs.ubi raw with nand write (OpenIPC's own urnand method) produces a correct, bootable UBI.

Environment

• defib a39e0e2 (master)
• Target: hi3516ev300, 128 MiB SPI-NAND (W25N01GV), board Rostelecom IPC8232SWC-WE
• Firmware: openipc.hi3516ev300-nand-ultimate.tgz (contains uImage.hi3516ev300 + rootfs.ubi.hi3516ev300, a full ubinize image, 16121856 bytes)

Repro

defib install -c hi3516ev300 --firmware openipc.hi3516ev300-nand-ultimate.tgz \
  --nand -p /dev/ttyUSB4 --nic eth0 --host-ip <a> --device-ip <b> --output json

→ {"event": "done", "success": true}

Observed (first boot after install)

ubi0: attaching mtd3
ubi0: attached mtd3 (name "ubi", size 118 MiB)
ubi0: good PEBs: 944, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128   <-- no rootfs volume
...
VFS: Cannot open root device "ubi0:rootfs" or unknown-block(0,0): error -19
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Code path

src/defib/cli/app.py, non-V500 NAND rootfs branch (~L2867–2888):

await _cmd(f"nand erase 0x{r_off:x} 0x{r_sz:x}", ...)
await _cmd("setenv mtdids nand0=hinand", ...)
await _cmd("setenv mtdparts mtdparts=hinand:1024k(boot),1024k(env),8192k(kernel),-(ubi)", ...)
await _cmd("ubi part ubi", ...)
await _cmd("ubi create rootfs", ...)
await _cmd(f"ubi write 0x{ram_addr:x} rootfs 0x{len(ubifs_data):x}", ...)

This formats a fresh UBI (ubi part ubi), creates a rootfs volume and ubi writes the (extracted) UBIFS into it. In practice this left ubi0 with 0 user volumes on reattach by the kernel — the volume table did not persist as expected from the u-boot ubi create/ubi write session.

Fix that works

Write the shipped rootfs.ubi raw to the ubi partition (exactly what OpenIPC's urnand env macro does), instead of ubi part/ubi create/ubi write:

tftpboot 0x42000000 rootfs.ubi.hi3516ev300
nand erase 0xA00000 0x7600000
nand write 0x42000000 0xA00000 0xf60000      # full 16121856-byte image

After this the kernel attaches ubi0 with user volume: 2 (rootfs + rootfs_data) and UBIFS mounts cleanly; the camera boots OpenIPC and streams.

rootfs.ubi from the OpenIPC release is a complete ubinize image (with volume table) designed for a raw nand write on chips with 0 bad blocks — the ubi write-into-a-created-volume approach does not reproduce it here.

Side note

For a fully bootable result on this board I also had to add ubi.mtd=3,2048 back to bootargs (OpenIPC firstboot regenerates bootargs and drops it) and set fw_setenv sensor sp2305 — but those are OpenIPC firmware-side, not defib.

[widgetii]

Follow-up with a real-world data point + a hardening angle that's relevant to the fix.

The UBI-write path skips the post-write read-back verification that tftp_and_flash already does. In src/defib/cli/app.py, tftp_and_flash verifies both stages:

• RAM after TFTP — L2737–2745 (TFTP CRC verified)
• flash after write — L2775–2783 (Flash verified, reads it back and compares CRC)

But the rootfs UBI paths don't get that:

• non-V500 rootfs (ubi part/ubi create/ubi write, ~L2860–2888) does no CRC check at all — neither RAM nor read-back.
• V500 _tftp_ubi_write checks RAM CRC but not the written flash.

So when ubi write silently produces a bad result (the 0-user-volumes case here), nothing catches it and install still reports success: true.

Real-world consequence I hit after working around this issue. I reflashed the rootfs manually via raw nand write (OpenIPC's urnand method) to get a bootable UBI. It booted and ran fine initially — but I had only checked Bytes transferred, not a CRC. A short time later the camera started throwing UBIFS errors on the read-only rootfs volume:

UBIFS error (ubi0:0): ubifs_check_node: bad CRC: calculated 0x72328821, read 0x9135bf5e
UBIFS error (ubi0:0): ubifs_check_node: bad node at LEB 120:101472  (magic 0x06101831 valid)
UBIFS error (ubi0:0): ubifs_read_node: expected node type 9
UBIFS error (ubi0:0): ubifs_iget: failed to read inode 198, error -117
UBIFS ... switched to read-only mode, error -117

UBI/MTD were healthy (944 good PEBs, 0 bad, 0 corrupted, no ECC/bitflip logs), so the bytes on flash were ECC-valid but semantically wrong — i.e., a bad byte in the RAM image had been written faithfully. Re-flashing with verification fixed it permanently:

tftpboot 0x42000000 rootfs.ubi.<soc>
crc32   0x42000000 0xf60000        # == host zlib.crc32 of the file  (verify BEFORE writing)
nand erase 0xA00000 0x7600000
nand write 0x42000000 0xA00000 0xf60000
nand read  0x48000000 0xA00000 0xf60000   # read back into a scratch addr
crc32   0x48000000 0xf60000        # == same CRC  (verify flash is byte-perfect)

Both CRCs matched and the rootfs has mounted error-free since.

Suggestion. The fix already proposed here — write rootfs.ubi via raw nand write instead of ubi write — has a nice side benefit: routing it through tftp_and_flash gives it the existing RAM-CRC and flash read-back CRC for free, which would have caught both the 0-volumes failure and this write-integrity issue. If the ubi write path is kept for any chip, it'd be worth adding the same read-back CRC check (crc32 of a nand read/ubi read of what was just written) so a bad write can't slip through as success: true.