Using LLMs to unleash the embedded Linux beast

Disclaimer: I used an LLM to help write about 50% of this post. All copyediting and final review were done by humans.

TL;DR

Our active modules used to take about 21 seconds from power-on to a usable touchscreen. It now takes 5.6. I didn’t write any of the code that made that happen. My job was to wire up a test bench so an AI agent could power-cycle the unit, read its console and watch its screen, then to give it a goal, a time budget and a cloud budget, and to make the calls that needed taste.

If you’re an existing customer: see for yourself and update your modules to the latest firmware.

Motivation

Production test rack gets power-cycled all the time: every time the test rack is moved, or every time a test technician decides the cleanest recovery is to pull the plug. Time spent waiting for a PD8X chassis full of modules to come online is half a minute of a test station doing nothing, again and again.

At Nine Fives we’re big fans of embedded Linux. It’s the secret sauce that lets us offer best-in-class networking features: the USB-C port as a driverless Ethernet dongle, a rich web UI self-hosted on the device, etc. This isn’t free though: embedded Linux has a well-earned reputation for complexity. Check out the Linux kernel from GitHub and you’ll find make menuconfig has 20,000+ options to configure. In the pre-LLM era, properly configuring an embedded Linux platform required a team of skilled software engineers. As a result, a lot of our competitors used technically inferior but simpler-to-develop microcontroller-based systems.

Another downside to embedded Linux is that, without tuning, it can take a while to boot. Most of this comes down to knowing exactly which kernel options to enable or disable. For our devices, some will render the system unbootable or introduce strange bugs, while others were important during the debug and platform-bringup phase of development but are now unnecessary. Determining which kernel config options fall into which category is a well-known craft: you profile, you find the slow thing, you try disabling or bypassing it, you reboot, and you do it a few hundred more times. It’s important, but it’s slow and tedious, so it rarely tops anyone’s list. It seemed like a good thing to hand to an agent.

This is a summary of the prompts I used, not a word-for-word transcript. There was a lot more back and forth than I’ve shown.

Step zero: give the agent hands and eyes

An agent can’t make a board boot faster if it can’t boot the board. Typing reboot at a Linux shell doesn’t capture early boot behavior, a true analog power cycle is needed.

What makes it more complicated is that our hardware can be powered via USB-C or Power-Over-Ethernet. To really give the LLM all the tools required it needs to be able to power the unit via either.

• Power. The unit runs on PoE or USB, whichever is live. The PoE comes from a MikroTik switch whose Power Sourcing Equipment ports can be turned on and off over its API, and the USB from a Coolgear switchable hub. Two small scripts give the agent an on/off switch.
• A serial console. A USB-to-serial adapter on the debug header, so the agent can see every line the bootloader and kernel print and log in when the network is down.
• Eyes. A 4K USB camera pointed at the touchscreen, streamed over the network. Most of the boot is visible only on the screen, and I wanted the agent to measure what a person standing at the bench would see.

Most of this was itself done by Claude. The camera streaming server, for example, was one prompt to set up and several more to keep alive. The camera feed dropping out halfway through a session turned out to be a recurring theme.

Step one: measure

The first session was just about measuring where the time went.

It wrote a profiler that cold-boots the unit through the power switches, timestamps every console line against the moment power was applied, splits the boot into phases and lists the longest silences. It then ran it a handful of times on the firmware we shipped at the time. The four boots agreed to within 10 milliseconds, which is a good sign that the method is sound. The headline: about 21 seconds to a login prompt on the serial console, of which a bit over 2 seconds was the PoE switch deciding whether it’s safe to supply power, and the rest was ours.

The ranked list of suggestions was a good read. A few highlights:

• The main application didn’t start until the network was fully configured, even though it doesn’t need the network to start.
• The bootloader paused for a full second in case someone wanted to press a key. Claude checked U-Boot’s source and confirmed that holding a key at power-on still works without the pause, so our factory recovery tools would keep working.
• The kernel was compressed with a slow algorithm and still included support for network file systems, CAN bus, Wi-Fi and a number of other things this product will never use.

That got the main application starting about 10 seconds sooner. It was a solid first pass, but the touchscreen barely moved: the UI still waited for the boot animation to finish, so the screen was still the last thing to come up.

Step two: the twelve-hour leash

A few days later I came back with a bigger ask, a time budget and a credit card. I had noticed in the previous run that the LLM spent a lot of time waiting for Linux kernel rebuilds while iterating, so I set it up with an AWS CodeBuild template that let it trigger much faster rebuilds while changing kernel code.

The first thing it did was check that it could actually control the bench. The second was to take a fresh baseline on the Step one branch: 19.4 seconds to a usable screen, down from about 21 on the shipped firmware, with an animation that stuttered and then went black for almost two seconds before the UI appeared. The third was to ask me some questions before touching anything outside the board’s own folder:

• May the UI cut the boot animation short, or play it faster? (yes)
• May it change the animation player and the network code in our application repository? (yes)
• May it flash a new bootloader, and is USB recovery wired up if that goes wrong? (yes)
• May it quiet the kernel’s console output and drop its tracing support? (yes)

I find that the questions the LLM asks are often a good way to clarify my thought process when I am overly vague initially.

Four and a quarter hours in, it stopped on its own, with most of its time budget unused, because, as it put it: the remaining gains are either inside nf_runtime (the application software, not Linux) or under 0.2 s each.

What was actually slow

Some of what it found was the usual embedded Linux housekeeping. Some of it was the kind of thing you only find by being stubborn. My favorites:

• The processor ran at half speed until the kernel was well into its boot. The chip comes out of reset at half its rated clock, and nothing in the bootloader ever raised it.
• The bootloader read the flash with its slowest, safe-default timings, because a missing clock driver meant it never learned the chip’s real ones. Linux used the fast timings. Now the bootloader does too.
• The Ethernet link was negotiated twice. The bootloader brings up the link, and then Linux, then quietly reset it and started over. Each restart cost about two seconds of renegotiation. The fix touched the bootloader, the device tree and two kernel drivers.
• DHCP waited 10 seconds for nothing. The first DHCP request went out before the link could carry traffic, and the retry was 10 seconds later. Now it asks every second.
• Our application was compressed with an executable packer, to save flash space, and spent about a second unpacking itself on every start.
• The kernel was printing to a 115,200 baud serial port, and printing is synchronous. Turning the console down to warnings only saved most of a second.
• The black screen. The touch UI only started after the animation finished, so there was always a gap. Now the UI starts during the animation and draws behind it, and the animation hands the screen over the moment the UI is ready.

The stretch goal got done too. The animation used to load its frames from flash while the rest of the system was also hammering the flash, so it played at about 15 frames per second for the first second. It now preloads its frames, runs at real-time priority, and flips frames in step with the display refresh. It also logs its own frame timing, so you don’t need a camera to know whether a boot was smooth. Almost every boot it measured had zero late frames.

Step three: the app itself

With the operating system out of the way, the remaining time was in our own application software.

The touch UI and the main application were two separate programs that shared most of their code, and both had to be loaded from flash at boot. It merged them into a single program that behaves as one or the other depending on how it’s started. That cut the UI’s startup by more than half and the firmware image by 3 MB. It also made the UI wait for real data before drawing anything (no more brief “HARDWARE TYPE UNSET” flash), and changed the UI loop to redraw only when something changes. The UI’s idle CPU use went from 19% to 6.5%, and a tap now registers about 45 milliseconds faster.

Step four: make it safe to ship

A faster boot that occasionally bricks a unit is worse than a slow one. We already have a lot of safeguards: most notably RAUC A/B boot. Most of the remaining effort went into proving it doesn’t.

• A second agent reviewed the pull request. It left a long list of notes, and the first agent fixed the easy ones, tested each fix on the bench, and came back to me with the two that needed a decision.
• Old units get the speed-up too. Some of the gains needed a different flash layout, or a new bootloader. The update package now converts older units as part of a normal update, and only rewrites the bootloader when it has actually changed, since that’s the one step in an update that isn’t power-cut safe.
• Then I asked the LLM to try to break it. It ran 30 trials on a second unit, most of them power cuts at chosen points during firmware updates: mid-write, mid-conversion, right after verification, during the reboot. Every time the unit came back on its own, on a complete, consistent copy of its firmware.

That campaign also turned up an older problem: enough power cuts in a row early in the boot could leave a unit stuck at the bootloader with two perfectly good firmware copies. The faster boot shrinks that window (the boot is now marked good at about 7.5 seconds instead of about 21), but doesn’t close it. Fixing it properly, so a power cut no longer counts as a failed boot, plus a hardware watchdog, became its own project and its own pull request, and is a story for another post. The boot-time work and the safeguards above are part of 4.7.0.

Bonus: making the video

The hero video at the top of this post was made by an agent too.

It installed each firmware, did one unrecorded cold boot so first-boot work wouldn’t count, cut both power sources for 10 seconds, cleared the DHCP lease, then filmed the boot in 4K. To find the exact frame where power came on, it fitted a line through the camera’s frame timestamps (which wobble by tens of milliseconds) and matched it to the moment the switch confirmed PoE was on. Then, unprompted, it made the side-by-side version with the timer.

Firmware	Menu fully drawn	DHCP lease	SSH up
4.0.0	20.9 s	28.5 s	29.0 s
4.7.0	5.6 s	8.6 s	10.1 s

All times are seconds after PoE is switched on, and about 2.2 seconds of each is the switch’s own PoE handshake, before the unit has any power at all. In our bench measurements, on a network with a static IP the unit answers SCPI at around 5.5 seconds, before the menu is even on screen. With DHCP, our bench switch’s own startup delay is now the slowest part of the boot.

Conclusion

The thing that surprised me most wasn’t any single fix. It was how much of the work came down to the bench. Once the agent could power the unit, read it and see it, a boot-time project went from “someday” to a few evenings of me answering questions. The cloud builds cost about $16 of the $100 I set aside.

All of this is in firmware 4.7.0 for the POE-ATTEN-6G, POE-SPDT-6G, and POE-SP4T-6G. To see the programmable RF test equipment available today, visit ninefives.com.