furtka/docs/hardware-test-2026-08.md
Daniel Maksymilian Syrnicki 8b07b57242
All checks were successful
CI / lint (pull_request) Successful in 24s
CI / test (pull_request) Successful in 1m18s
CI / validate-json (pull_request) Successful in 22s
CI / markdown-links (pull_request) Successful in 14s
fix(updater): start newly linked timers and link units via current; bench test report
After the 26.19 → 26.20 self-update the bench had furtka-welcome.timer
linked + enabled but inactive (dead): enable only arms a timer for the
next boot. The link also pointed at /opt/furtka/versions/<ver>/ instead
of /opt/furtka/current/, unlike the installer-created links, so it would
have stopped following later updates and rollbacks. Link via current,
enable, start.

Adds docs/hardware-test-2026-08.md (timeline, five findings with root
causes / fixes / validation status, what worked, open items) and a
'Real hardware' section in iso/README.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MScAinbyMdeNc7H2BZdnnG
2026-08-26 09:46:13 +02:00

6.8 KiB
Raw Blame History

Hardware test report — "junk bench", 2026-08-24/26

First end-to-end run of Furtka on real hardware that was not a Proxmox VM and not the Medion box: a bare old-laptop mainboard lying on the desk, a 1 TB Crucial BX SATA SSD that had previously lived in a Proxmox VE host, a USB hub for keyboard + boot stick, LAN cable, HDMI monitor. Tester: Daniel. Release under test: 26.19-alpha, then self-updated to 26.20-alpha.

Why it mattered: every bug below is invisible on the smoke VM. A VM has a fresh empty disk, a fast virtual NIC, UEFI firmware and no USB stick to pull.

Timeline

When What happened Outcome
08-24 ISO flashed with Fedora Media Writer, bench booted, browser opened proski.local:5000 "Server Not Found" — typo (proski vs proksi). Bench powered off.
08-25 proksi.local:5000 — wizard came up (BIOS/syslinux boot path, stock Arch splash) OK
08-25 13:07 Install run 1 Failed at 12 %: Partition(s) 2, 3 on /dev/sda have been written, but we have been unable to inform the kernel of the change … in use
08-25 Root shell over SSH (passwd + ssh-copy-id on the live ISO): SSD still carried a Proxmox VE pve volume group (swap, root, thin pool); udev had activated it, six device-mapper nodes sat on sda3 Cleared by hand: dmsetup remove ×6, wipefs -a /dev/sda, blockdev --rereadpt
08-25 13:16 Install run 2 Installation completed without any errors; BIOS detected → Grub, MBR table, sda1 1 G vfat /boot, sda2 ext4 /
08-25 "Reboot now" clicked Box booted the USB stick again (BIOS order). Stick pulled, rebooted → furtka.local = 192.168.178.142, Caddy answering
08-25 16:22 First boot of the installed system furtka-catalog-sync.service failed: Network is unreachable. network-online.target had fired on the IPv6 SLAAC address ~1 s before DHCPv4 landed. State file stuck on checking, apps page empty; next timer run would have been 18:26
08-25 ~17:00 Catalog synced via the "Sync apps catalog" button Catalog 26.13-alpha, 7 apps listed
08-25 17:47 26.20-alpha released with the fixes below Box reports Update available: 26.19-alpha → 26.20-alpha
08-26 09:34 Self-update to 26.20-alpha from /settings current → 26.20-alpha, no failed units. New finding: furtka-welcome.timer linked + enabled by the updater but inactive (dead)enable alone only arms it for the next boot

Findings and fixes

# Finding Root cause Fix Shipped in Validated on hardware
1 Console banner showed only proksi.local / furtka.local, never the IP fallback Banner script was a oneshot after network-online.target and dropped the IP line when no address existed yet. On the live ISO releng's root autologin on tty1 also made agetty --reload a no-op furtka-issue.timer (ISO) / furtka-welcome.timer (installed) re-run the script every 5 s; fallback line is always present ("no IP address yet — check the cable / DHCP" until one arrives); ISO no longer auto-logs root in on tty1 26.20-alpha (0e05bef) Installed side: pending reboot or next update. ISO side: pending reinstall from the 26.20 ISO
2 Install failed at partitioning on a disk with a previous OS Live ISO ships lvm2/mdadm/dmraid/cryptsetup; udev auto-activates whatever the disk carries; parted cannot re-read the partition table while dm nodes hold the partitions webinstaller/diskprep.py: release_disk() runs before archinstall — swapoff/umount leaves-first, dmsetup remove / mdadm --stop, wipefs -a, blockdev --rereadpt; steps logged at the top of the install log; progress page names this failure explicitly 26.20-alpha (5ddda53) Pending reinstall from the 26.20 ISO (the disk now carries Furtka itself, which exercises the same path minus LVM)
3 "Remove the USB stick" was easy to miss — the box booted the installer again Hint lived on the rebooting page and in a confirm() popup, i.e. after the click, with the machine restarting 3 s later Done state shows a two-step list (pull the stick → restart) above the button; button reads "USB stick is out — restart now" 26.20-alpha (60404ae) Pending reinstall
4 No apps after first boot Boot-time catalog sync raced DHCPv4 (IPv6 made network-online fire early), no retry, timer jitter pushed the next attempt 2 h out, state file stuck on checking furtka-catalog-sync.service retries on failure (20 s, up to 8× in 15 min); failures recorded as stage error; apps page shows "last sync failed: …" 26.20-alpha (32ba522) Retry config confirmed live on the box after the update; a cold boot with the new unit is still pending
5 New timer dead after self-update _link_new_units() linked + enabled but never started the timer; also linked from the versioned dir instead of current Link via current, enable, start Unreleased (this branch)

Not a bug, but worth knowing: on this board the BIOS boots the USB stick before the SSD, so every reinstall needs the stick pulled at the reboot prompt. The stock Arch splash on the BIOS/syslinux boot path is unbranded (only menu labels are) — cosmetic.

What worked first time

  • Fedora Media Writer → hybrid ISO boots on BIOS firmware.
  • mDNS: proksi.local (live ISO) and furtka.local (installed) resolved from a Fedora client once spelled correctly.
  • BIOS detection → Grub + MBR layout; the installed system booted from the SSD without intervention.
  • Login, /settings, catalog sync via the UI button, furtka update --check, self-update 26.19 → 26.20 with symlink flip and clean unit state.
  • Time sync, Docker 29.7.2, Caddy on :80, sshd with key auth for the wizard user (wheel + docker groups as configured).

Still open from this run

  • Reinstall from the 26.20-alpha ISO on the bench to validate findings 13 on the ISO side (the installed side of #1 validates on the next reboot).
  • Cold boot with the retrying catalog-sync unit (#4) on a dual-stack LAN.
  • Rebrand the syslinux boot splash.
  • smoke-latest.yml is workflow_dispatch only and has not run since April; the smoke VM would not have caught any of the above anyway, but it is currently not running at all.

Reproducing the bench conditions in a VM

Most of this can be provoked on Proxmox if you want a regression check without the hardware:

  • Finding 2: attach a disk that already carries an LVM install (e.g. a spare Proxmox VE or Ubuntu LVM disk image) before booting the ISO.
  • Finding 3: leave the ISO attached as CD-ROM with boot order CD-first.
  • Finding 4: give the VM's bridge IPv6 RA before DHCPv4 — harder; the retry logic is unit-tested instead.
  • Finding 1: use SeaBIOS instead of OVMF (the README notes SeaBIOS fails on ldlinux.c32 from this ISO, so this one really needs hardware).