From 67afb1e5a493a6356cac9b84bf957a27a9980c7d Mon Sep 17 00:00:00 2001 From: BarnacleBoy Date: Fri, 12 Jun 2026 21:33:17 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20NixOS=20install=20plan=20=E2=80=94=20Ni?= =?UTF-8?q?xOS=20Anywhere=20eval,=20DTB=20notes,=20install=20approach,=20b?= =?UTF-8?q?oot=20management?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/nixos-install.md | 248 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 docs/nixos-install.md diff --git a/docs/nixos-install.md b/docs/nixos-install.md new file mode 100644 index 0000000..115bbe0 --- /dev/null +++ b/docs/nixos-install.md @@ -0,0 +1,248 @@ +# NixOS Install — Orange Pi 3B (Cyberdeck Multi-Boot) + +**Goal:** NixOS on eMMC partition p4 (root_nixos), booting via shared extlinux.conf alongside Orange Pi OS Arch and Armbian. + +## Context + +The cyberdeck's eMMC has a four-partition layout with a shared /boot on p1: + +``` +mmcblk0 (233GB eMMC user data area) +├── mmcblk0p1 FAT32 1GB /boot (shared — kernels, DTBs, extlinux.conf) +├── mmcblk0p2 ext4 85GB root_opios (Orange Pi OS Arch — live) +├── mmcblk0p3 ext4 85GB root_armbian (Armbian — live or planned) +└── mmcblk0p4 ext4 ~84GB root_nixos (NixOS — target) +``` + +Boot is Rockchip-standard user-data-area: PARTITION_CONFIG=0x78, idbloader at sector 64, u-boot.itb at sector 16384, SPI flash erased. u-boot reads `extlinux.conf` from the FAT32 /boot partition for the boot menu. + +Full walkthrough of the boot architecture and partition setup: `docs/emmc-multiboot.md`. +Existing second-OS install pattern: `docs/armbian-install.md`. + +## NixOS Anywhere — Evaluated, Not a Fit + +[NixOS Anywhere](https://github.com/nix-community/nixos-anywhere) is an SSH-based tool that installs NixOS onto a remote machine. It uses `kexec` to boot a temporary NixOS installer in RAM, then uses `disko` to partition and format the disk. It's a great tool for Hetzner servers and fresh-install scenarios, but it doesn't work for our multiboot setup for three reasons: + +1. **Disko is destructive by default.** Default mode is `destroy,format,mount` — it wipes the entire disk and repartitions from scratch. There is no supported flow to say "install into partition 4 and leave the rest alone." [GitHub issue #274](https://github.com/nix-community/nixos-anywhere/issues/274) documents this gap; the [disko dual-boot discourse thread](https://discourse.nixos.org/t/disko-dualbooting-windows-and-nixos/61665) shows someone whose existing partitions got eaten. + +2. **Shared /boot is incompatible with NixOS's boot generation model.** NixOS auto-generates boot entries. Every `nixos-rebuild switch` or `nixos-rebuild boot` writes to /boot and would overwrite our extlinux.conf, removing the Arch and Armbian entries. + +3. **kexec on ARM adds complexity without solving anything.** Aarch64 kexec images exist (from `nix-community/nixos-images`) but even once you kexec in, you're back to the disko problem. + +**Verdict:** NixOS Anywhere is built for clean-slate installations on blank disks. Our shared-extlinux.conf multiboot setup needs a manual approach. + +## Device Tree Blob (DTB) — Same One Works + +The DTB describes the Orange Pi 3B's hardware — the RK3566 SoC, its memory map, its peripherals. It's not OS-specific. The same `rk3566-orangepi-3b.dtb` that Orange Pi OS Arch uses will boot NixOS. + +The only caveat: DTBs are compiled from the kernel source tree, and kernel bindings evolve. A DTB from kernel 6.6 might not have everything a kernel 6.12 expects. But in practice, for Rockchip boards, they're backward-compatible within a major version range. + +**Recommendation:** Point the NixOS extlinux.conf `FDT` directive at the DTB file already on the shared /boot partition. If it doesn't work (kernel panic about missing clocks or an undetectable PHY), extract a matching DTB from the NixOS closure instead. + +## NixOS-Specific Challenges + +NixOS differs from Arch and Debian in how it handles kernels and initrds. These three wrinkles make installation more involved than `rsync` and an extlinux.conf entry. + +### 1. Kernel + DTB Are Inside the Nix Store + +Unlike Arch (where `/boot/vmlinuz-linux` is a standalone file) or Debian (where `/boot/vmlinuz-*` is installed by the kernel package), NixOS bundles the kernel and DTB inside the Nix store at paths like: + +``` +/nix/store/-linux-/Image +/nix/store/-linux-/dtbs/rockchip/rk3566-orangepi-3b.dtb +``` + +These are not on a traditional /boot partition. They're symlinked into the NixOS /boot by `systemd-boot` or the extlinux generator, but the actual files live in the Nix store. You can't just copy them — you need to build the closure on a machine with Nix installed, then extract the boot artifacts. + +### 2. Initrd Is Generated From Configuration + +NixOS generates the initrd at build time from the system configuration. It doesn't exist as a standalone file until `nixos-rebuild` produces it. Like the kernel, it lives in the Nix store: + +``` +/nix/store/-initrd-linux-/initrd +``` + +You need to extract it from a pre-built closure and place it on the shared /boot manually. + +### 3. NixOS Wants to Manage /boot + +By default, NixOS installs a bootloader (systemd-boot on aarch64 UEFI, or extlinux on non-UEFI). Once running, `nixos-rebuild switch` attempts to write to /boot — which would overwrite our shared extlinux.conf and break the other OSes. + +**Fix:** Configure NixOS to manage only the kernel and initrd on /boot, not the bootloader menu. Set: + +```nix +boot.loader.grub.enable = false; +boot.loader.generic-extlinux-compatible.enable = true; +# OR, for manual-only: +boot.loader.generic-extlinux-compatible.enable = false; +``` + +With `generic-extlinux-compatible` enabled, NixOS writes a `boot.txt` or `extlinux.conf` to /boot, but will overwrite our shared version. With it disabled, NixOS doesn't touch /boot at all — you manage the extlinux.conf entry yourself on every kernel update. + +## Recommended Install Approach + +The pattern is: **pre-build on a machine with Nix, extract boot artifacts, copy to eMMC.** + +### Prerequisites + +- A machine with Nix installed (Lucy, Mermaid Man, or BarnacleBoy if Nix is available) +- An SD card to transfer files to the Orange Pi 3B +- The Orange Pi 3B booted into Orange Pi OS Arch (or any working OS on eMMC) + +### Step 1: Build a NixOS closure (on Lucy or Mermaid Man) + +Create a Nix flake for the Orange Pi 3B configuration. At minimum: + +```nix +{ + description = "NixOS for Orange Pi 3B"; + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + }; + outputs = { self, nixpkgs }: { + nixosConfigurations.orangepi3b = nixpkgs.lib.nixosSystem { + system = "aarch64-linux"; + modules = [ + ({ pkgs, ... }: { + # Minimal config for first boot + boot.loader.grub.enable = false; + boot.loader.generic-extlinux-compatible.enable = false; + boot.kernelParams = [ "console=tty1" "console=ttyS2,1500000n8" ]; + system.stateVersion = "24.11"; + # Root on p4 — set the UUID or /dev path + fileSystems."/" = { + device = "/dev/mmcblk0p4"; + fsType = "ext4"; + }; + # Do NOT mount /boot — it's managed externally + # fileSystems."/boot" = ...; /* leave this out */ + users.users.root.initialPassword = "changeme"; + services.openssh.enable = true; + }) + ]; + }; + }; +} +``` + +Build it: + +```bash +nix build .#nixosConfigurations.orangepi3b.config.system.build.toplevel +``` + +### Step 2: Extract boot artifacts + +From the built closure, extract the kernel, initrd, and DTB: + +```bash +# Find the closure path +ls -la result/ + +# Kernel +cp result/boot/Image /tmp/nixos-kernel + +# Initrd +cp result/boot/initrd /tmp/nixos-initrd + +# DTB — find it under the kernel package +find result/ -name "rk3566-orangepi-3b.dtb" -exec cp {} /tmp/ \; + +# The entire Nix store closure (for copying to p4) +nix copy --to ./result /tmp/nixos-closure +``` + +Getting the full rootfs onto p4 is the trickiest part. Options: + +- **Option A:** `nix copy` the closure to a path on the SD card, then `nix-store --import` on the OPi3B (if Nix is installed) +- **Option B:** Use `nixos-enter` or manual chroot manipulation +- **Option C:** Build a tarball of the rootfs and extract it on the target + +### Step 3: Copy to the Orange Pi 3B + +Put the kernel, initrd, DTB, and rootfs archive on an SD card. Boot the OPi3B into Orange Pi OS Arch. + +```bash +# Mount the SD card +sudo mount /dev/mmcblk1p1 /mnt/sd + +# Mount the target partitions +sudo mount /dev/mmcblk0p4 /mnt/nixos_root +sudo mount /dev/mmcblk0p1 /mnt/nixos_root/boot + +# Copy kernel, initrd, DTB to shared /boot +sudo cp /mnt/sd/nixos-kernel /mnt/nixos_root/boot/ +sudo cp /mnt/sd/nixos-initrd /mnt/nixos_root/boot/ +sudo cp /mnt/sd/rk3566-orangepi-3b.dtb /mnt/nixos_root/boot/ + +# Extract rootfs to p4 +sudo tar -xzf /mnt/sd/nixos-rootfs.tar.gz -C /mnt/nixos_root/ +``` + +### Step 4: Add extlinux.conf entry + +Edit `/mnt/nixos_root/boot/extlinux/extlinux.conf` (the shared one) and add: + +``` +LABEL NixOS + LINUX /nixos-kernel + INITRD /nixos-initrd + FDT /rk3566-orangepi-3b.dtb + APPEND root=UUID= ro console=tty1 console=ttyS2,1500000n8 +``` + +Get the UUID: + +```bash +sudo blkid /dev/mmcblk0p4 +``` + +### Step 5: Boot and finish setup + +```bash +sudo umount /mnt/nixos_root/boot /mnt/nixos_root +sudo sync +``` + +Reboot, pick "NixOS" from the u-boot menu. Once booted: + +```bash +# Login as root with the password set in config +# Set up SSH keys, add a user, etc. +# Do NOT run nixos-rebuild switch until /boot management is understood +``` + +## Boot Management After Install + +Once NixOS is running, kernel updates become a manual process because NixOS isn't managing /boot: + +1. On the build machine, build a new closure with the updated kernel +2. Extract the new kernel + initrd + DTB +3. Copy them to the OPi3B's shared /boot via SD card or SSH+scp +4. Update the extlinux.conf `LINUX` / `INITRD` paths if filenames changed +5. Reboot + +Alternatively, if the kernel and initrd paths stay constant (same filenames), you can overwrite in place and the existing extlinux.conf entry keeps working. + +## NixOS on ARM — Board Support + +The Orange Pi 3B doesn't have a dedicated NixOS wiki page yet. The [NixOS on ARM wiki](https://wiki.nixos.org/wiki/NixOS_on_ARM) confirms AArch64 has full upstream support. The generic aarch64 images from Hydra should work on the OPi3B with the correct DTB. The Orange Pi 5 (RK3588) wiki page at serves as a reference — the approach is the same, just different SoC. + +## Verification Checklist + +- [ ] NixOS boots from u-boot menu +- [ ] Network works (the kernel needs the right DTB for the Ethernet PHY) +- [ ] SSH access (configured in the NixOS flake) +- [ ] extlinux.conf entries for all three OSes still present after NixOS boots +- [ ] Kernel update works (build, extract, copy, reboot cycle) + +## References + +- [NixOS Anywhere](https://github.com/nix-community/nixos-anywhere) — evaluated, not suitable (see above) +- [Disko](https://github.com/nix-community/disko) — declarative partitioning, destructive by default +- [NixOS on ARM wiki](https://wiki.nixos.org/wiki/NixOS_on_ARM) — AArch64 support status +- [Orange Pi 5 Plus wiki](https://wiki.nixos.org/wiki/NixOS_on_ARM/Orange_Pi_5_Plus) — reference for Rockchip NixOS approach +- [nix-community/nixos-images](https://github.com/nix-community/nixos-images) — aarch64 kexec installer images +- eMMC multi-boot setup: `docs/emmc-multiboot.md` +- Armbian install pattern: `docs/armbian-install.md` +- Bring-up reference: `docs/initial-bringup.md` \ No newline at end of file -- 2.49.1