Firmware Changes

Local changes to the ReefRhythm-SmartDoser firmware running the reef doser (see the Acropora page for the tank itself). This tree is a local working copy, not a git fork -- these fixes exist only as the comments/code in this checkout, found and fixed against real dosing behavior on this tank's own hardware (MKS SERVO42C stepper drivers, 750L reef system). Recorded here since there's no git history to carry them.

1. Low-flow calibration extrapolation could silently overdose. Below the lowest measured calibration point, the polynomial fit was extrapolating blind -- confirmed on real calibration data that it can swing back up instead of continuing toward zero. That defeats move_with_rpm's low-flow runtime-shortening safety check entirely: a near-zero flow request could get an RPM at or above the achievable minimum instead of below it, so the dose ran at full requested duration instead of a shortened one -- a silent overdose. Fixed by anchoring a real (0 RPM, 0 flow) point -- a peristaltic pump at 0 RPM delivers exactly 0 flow, a physical fact the fit had no way to know on its own.
2. Extrapolated flow curve could decrease with increasing RPM. A degree>1 polynomial fit can wiggle (local extrema), producing a flow-rate curve that dips as RPM increases even though real flow never decreases with RPM. Forced non-decreasing via a running max.
3. RPM-snap without runtime rescaling could silently overdose. find_combination snaps a requested RPM to the nearest achievable one in the calibration table -- for a low-flow dose this is very often higher than requested (there's a minimum achievable RPM, ~0.593 on this table). Running that higher RPM for the original, unadjusted runtime overdoses, since delivered volume scales with rpm * runtime. Fixed by rescaling runtime down proportionally whenever the snap goes up. Deliberately one-directional: if a request's RPM is above the table's max (e.g. an unvalidated /run or MQTT command with a typo'd rpm=99999), the snap goes the other way and scaling runtime by desired/achieved would be >1, amplifying runtime by the same factor instead. Failing safe there means delivering less than requested at the achievable top speed, never risking an unbounded overdose in the other direction.
4. Zero-step doses read as a driver fault. calc_steps truncates to int -- a request small enough that the runtime-shortening above collapses to well under one step's worth of rotation rounds down to exactly 0. Sending a real "run 0 steps" command to the driver isn't a no-op on this hardware: it came back as an outright failure (rejected reply or no completion confirmation within timeout), indistinguishable from a genuine driver fault, and triggered the same "DOSE FAILED -- driver unresponsive" alert for a request that was simply too small for this pump/duration to resolve. Fixed by checking steps <= 0 and failing cleanly before any UART traffic at all.

5. CRC check used decoded values instead of raw bytes. read_angle_error deliberately does NOT use check_crc=True: the existing CRC check summed the already-decoded field values, not the raw bytes actually transmitted -- only correct for single-byte fields. error is a signed 2-byte value (can be negative), so that comparison would not reconstruct the driver's real byte-level checksum and would reject valid replies. A correct fix needs the check computed over raw_data before struct.unpack, not after.
6. Delayed "straggler" reply corrupting the next command. Config changes (mode switches especially) can trigger a delayed side-effect reply/reset on the driver that arrives after the bounded read already gave up -- confirmed by observation: a dose sent immediately after a mode/current-set call failed, then an identical dose sent right after that succeeded cleanly. Fixed with a short settle + buffer flush before send_set_command/release_protection return.
7. Missing flush before make_steps when called with stop=False. move_with_rpm calls make_steps(..., stop=False), which otherwise skips the only buffer-clearing step (stop()'s own flush) -- without an explicit flush here, a previous dose's "run complete" reply that arrived after its completion-wait had already given up would still be sitting in the buffer and get misread as, or corrupt, this command's own reply.
8. Fast doses could hide their own completion reply. Low-flow doses are very often RPM-snapped up (see #3), and for a big snap this can shrink actual motor runtime to a fraction of a second -- fast enough that the driver's "complete" reply can already be sitting in the buffer alongside the "starting" ack, in the very same read_raw() call. Since that call already drained the buffer, waiting again afterward would never see it. Fixed by checking for complete_pattern in the initial read first, before waiting for anything new.
9. Negative step count could raise an uncaught OverflowError. steps.to_bytes(4, 'big') is unsigned -- a negative steps value (reachable if an RPM-snap ratio or a malformed schedule entry ever produces one upstream) would raise uncaught instead of failing the dose cleanly. Fixed with an explicit steps < 0 guard.

10. Version-comparison gate could leave files stale after an interrupted extraction. version.txt is only ever written after a full extraction completes. If an extraction is ever interrupted partway (an external reset mid-boot, a crash), a later boot's version comparison could wrongly conclude nothing had changed and skip re-writing files that were never actually updated. Fixed by always re-extracting unconditionally on boot -- costs a few extra seconds every boot but guarantees static files can't get silently stuck out of date.
11. A silently-empty RELEASE_TAG made real OTA updates no-op. RELEASE_TAG drives boot.py's "did the app actually change" check and is shown on the OTA page / published over MQTT. If two builds in a row both had an empty tag, boot.py saw no version change and skipped re-extracting the frozen app -- so a real code fix could survive a "successful" OTA update without ever actually taking effect. Fixed in init.sh by defaulting RELEASE_TAG to a timestamped <version>-manual-<UTC timestamp> value whenever it isn't explicitly set by CI.

12. GET /storage-status -- read-only endpoint exposing each pump's container_size_ml/remaining_ml/dosed_since_refill_ml, reflecting the same storage dict already used internally for the "container empty" alert. Added so cumulative dosing-over-time could be shown on this tank's own webpage without guessing it from the schedule alone -- a missed/failed dose or a manual off-schedule run wouldn't show up in a schedule-based estimate, but does here since remaining only ever decrements on an actual completed dose. No side effects, no new write paths.
13. Cancel Download button on the OTA page. Previously there was no way to stop an in-progress OTA download short of power-cycling the device. Added POST /ota-cancel, which sets a flag the existing download loop in /ota-upgrade checks once per chunk. Safe by construction, not just by care taken: OTA.write() only ever touches the inactive partition, and only OTA.close() calls part.set_boot() -- breaking out of the loop before close() leaves the currently-running (old, known-good) firmware as the boot target no matter how much or how little was written. The one thing that did need explicit handling: dosing is paused (scheduled_jobs = []) the moment an OTA starts, and a cancelled download now restores it via update_schedule(schedule), the same as the existing failure path -- without that, a cancelled OTA would have left real dosing silently disabled with no reboot to ever bring it back.
14. "Dosing now" indicator on doser.html. A pulsing indicator next to the currently-selected pump's name. Deliberately does NOT touch stepper_run (the actual dosing execution path, already the source of several of the bug fixes above) -- two independent, purely client-side/frontend sources instead: scheduled doses are inferred by polling the existing, unmodified GET /schedule-diag endpoint every 5 seconds (already exposes each job's exact computed dose_times and the device's own seconds_since_midnight); manual doses are shown for the exact duration of the "Start Dosing" button's own fetch('/dose') call, which already only resolves once the pump has actually finished (or failed) -- true start-to-finish coverage, not a guess based on the entered duration. Good enough for "does this look like something's happening right now", not a safety signal.
15. Settings save no longer reboots for cosmetic changes. /settings previously called a hard device reset unconditionally on every save, purely as a byproduct of rewriting every config field in one shot -- renaming a pump bounced the whole live doser exactly the same as changing WiFi credentials. Only WiFi credentials, hostname (mDNS re-registration) and MQTT broker/login genuinely require the re-init this firmware only does at boot, and pump count changes fixed-size hardware arrays allocated at import time -- everything else (pump names, color, theme, notification settings, current limits, direction inversion) is now rebound live in place instead of waiting for a reboot.
16. "Dosing now" indicator could miss short doses. The scheduled-dose polling above ran every 5 seconds -- fine for longer doses, but this tank's shortest scheduled dose (Magnesium, ~15-21s) could have up to a quarter of its actual run window missed at either edge. Tightened to a 1-second poll interval; manual doses were already unaffected.
17. Scheduled dose-amount ramps. A schedule entry can now carry a start date, end date, start amount and end amount; the doser linearly interpolates the actual dosed amount between them by calendar date, holding at the start amount before the start date and permanently at the end amount after the end date. Symmetric by construction -- an end amount lower than the start amount ramps down, not just up. A new background check runs once a minute, recomputing the live schedule whenever the calendar date rolls over on a ramping job (a no-op for every other schedule entry). Added because gradual dosing changes -- this tank's own carbon-dosing ramp-up, ongoing Ca/Mg/Alk corrections -- previously required manually recalculating and re-pushing a new amount every few days by hand. The schedule editor on doser.html also gained a "Ramp dose amount over time" option (start date, end date, end amount) when adding a new dose, which scales duration together with amount to keep the pump's flow rate roughly constant across the ramp, the same approach used for every manual dose correction on this tank so far.
18. Dose-log write failures could go completely unnoticed. Every dose gets persisted to an on-device history file for later review, but a write failure there was only ever visible on a serial console -- confirmed on this tank when real dosing kept running correctly (notifications went out the whole time) while the history file itself had a multi-hour gap with zero trace anything had gone wrong. Fixed by routing a write failure through the same notification channels already used for dose-failure alerts, rate-limited to once per failure episode rather than once per dose (this runs on every single dose across every pump, many times a day), with a matching one-time "recovered" message once writes succeed again.
19. New GET /fs-status. Read-only endpoint exposing the doser's internal filesystem usage (total/used/free bytes), added to check whether a near-full filesystem could explain the write failures above. On this tank it was ruled out (47.9% used at the time).
20. Ramping schedule points now show today's actual dose in the info modal. The dose-ramp feature above means the amount a pump actually runs today can differ from what's shown in the raw schedule list -- there was no way to see today's real interpolated dose without querying the device's diagnostics directly. Clicking a ramping point's info popup now shows "Today's actual dose" alongside the existing ramp-range text, fetched once at click time rather than continuously polled -- no extra network traffic for ordinary fixed-amount schedule points.

21. Dose failures were logged without saying why. The driver code already distinguished several real failure causes internally -- no reply at all, an explicit rejection, a run that started but never confirmed completion, the low-flow implausible-RPM safety refusal, a negative step count -- but only ever printed which one to a console nobody has open. The persisted dose-log history collapsed all of them into a bare FAILED, with no way to tell afterward which had actually happened. Fixed by threading a reason string through every failure path into the existing dose-log line, plus a fallback note for any future un-annotated failure path, so a gap in this coverage shows up as "reason not captured" instead of silently reproducing a blank FAILED again.
22. Per-dose waits could stall the web server. set_mstep's retry loop and make_steps's initial "run starting" acknowledgement both used a blocking, time.sleep_ms-based wait rather than the event-loop-yielding one already used for the (much longer) completion wait -- meaning the web server, WiFi, and every other pump's schedule could stall for up to roughly 900ms in the worst case, on every single dose, not just an occasional one. Converted both to the same async, yielding wait pattern already proven on the completion wait.
23. New /advanced-settings page. The MKS SERVO42C protocol documents a considerably larger command set than this firmware ever exposed -- working current, microstepping, position-loop PID gains (Kp/Ki/Kd), acceleration, max torque, locked-rotor protection, on-demand calibration, and more, all reachable over the same serial bus already used for dosing, with no UI to reach any of it. Added a dedicated page exposing these settings grouped by real consequence (cosmetic, operational, dangerous) rather than a flat list, each with a plain-language explanation of what it does and what breaks if it's set wrong. Dangerous settings require typing the target pump's name before the send control activates, and the same confirmation is enforced server-side too, independently of the page's own JS -- the request itself is refused without it, not just hidden behind a UI gate.
24. Work Mode, Serial Baud Rate, and Restore Factory Defaults deliberately left out. All three can take a driver out of CR_UART/38400 entirely, and the only way back is the driver's own physical OLED menu -- not a viable recovery path on this installation. For every other dangerous setting on the page (bus address, acceleration) a real software recovery exists; for these three there isn't one, so a wrong value here wouldn't just be risky, it would be unrecoverable. Left off the page entirely rather than hidden behind a warning.
25. Raw bus-address recovery. If two drivers ever end up answering the same address -- for instance if one was ever reconfigured via its own physical menu to an address another pump already uses -- the normal pump-name selector becomes actively misleading: it addresses a pump by its intended identity, not by whatever address its driver is actually sitting at. Added a direct raw-address override that talks to a specific bus address regardless of which pump is supposed to live there, so recovering from a collision doesn't depend on guessing which pump name happens to reach the colliding driver.

26. Making dose waits non-blocking reintroduced a race the old blocking version had accidentally prevented. Item #22 above fixed the web server stalling during a dose by making set_mstep and make_steps's initial acknowledgement wait yield the event loop instead of blocking it. What wasn't obvious at the time: the old blocking wait had an unintended side effect, freezing the single-threaded event loop for a dose's entire duration also happened to block every other UART-touching route (/pump-diag, /pump-config, the advanced-settings page) from ever running concurrently with a dose -- and all nine stepper drivers share one physical UART bus. Removing that freeze let diagnostics and config calls interleave their bytes with an in-progress dose's own traffic, corrupting both sides of whichever two commands collided. Confirmed by measurement on this tank: the dose failure rate roughly quadrupled, from 7.9% (20 of 252) in the days before this change to 34.5% (20 of 58) after, hitting nearly every pump. Fixed permanently -- not by reverting the event-loop fix, which stays in place -- by routing every one of these previously-direct driver calls through the same command_buffer queue /dose, /stop, and /run already used to serialize access to the shared bus, so diagnostics and config changes now queue behind an in-progress dose instead of racing it.

27. The driver's own transmission was being read back as the driver's reply. All nine drivers share one single-wire half-duplex UART, so every byte the ESP32 writes also lands in its own RX buffer. Nothing in the read path distinguished that echo from a genuine reply, so a command whose driver never answered still "succeeded" -- it just parsed its own outgoing bytes. The arithmetic made it unambiguous: read_protection_state returned 62, which is exactly the 0x3E function code it had just sent, and read_angle_error returned the bus address plus 14393, where 14393 is 0x3839 -- the 0x39 function code followed by its checksum byte. Fixed by recording each command as it is written and stripping that exact prefix from whatever comes back before parsing. This also re-labelled a class of failures that had previously been reported as bogus sensor values rather than as "no reply".
28. A failed read was unpacked into three variables, returning 500 instead of "no reply". read() signals failure by returning False, but read_angle_error, read_encoder and read_pulses immediately unpacked its result into multiple names. With a driver physically disconnected -- or simply not answering -- that raised a TypeError inside the request handler, so /pump-diag served an opaque "Internal server error" exactly when it was needed most. Each now checks for False explicitly and reports no-reply as a value.

29. Every driver config and diagnostics call blocked the single-threaded event loop for its full wait. Fixing the echo misread above had a side effect: a call that used to return instantly on its own echo now correctly waited out its entire bounded timeout. Because these methods still used time.sleep_ms and ran directly inside the event loop, one unresponsive driver could stall the web server, the scheduler and every other pump for up to ~1.5 s per call. Added *_async counterparts that yield the loop during their waits -- the same pattern the dosing path already used -- and routed every diagnostics and config route through them. The original synchronous methods are kept deliberately, because load_configs.py calls set_current() at boot before an event loop exists.
30. Those new async calls were silent no-ops for two builds, because MicroPython coroutines have no __await__. The dispatch helper decided whether to await a result with hasattr(result, "__await__"), which is correct on CPython. In MicroPython a coroutine is a generator and has no such attribute, so the test was always false, the coroutine was never awaited, and every configuration write silently did nothing while reporting success. /pump-diag made it visible by returning the string <generator object 'read_angle_error_async'> as a pump's angle error. Fixed by testing against the real generator type as well as __await__. Worth recording as a caution: a CPython idiom that merely returns the wrong answer here, rather than raising, produced two builds in which an entire feature appeared to work and did not.

31. The run command's acknowledgement had the tightest timeout on the bus and no retry behind it. make_steps waited 150 ms for the driver to acknowledge a run command and abandoned the whole dose if it did not arrive, while a neighbouring exchange (set_mstep) retried the same kind of read five times. Widened to 400 ms, which costs nothing when the acknowledgement is on time and carries none of the double-dose risk a retry on a motion command would. See item #34 for what was actually making those acknowledgements late.
32. The OTA download spent about 75 seconds doing nothing. The firmware download loop yielded with await asyncio.sleep(0.1) once per 4096-byte chunk, so a ~3 MB image paid a fixed 100 ms penalty roughly 750 times over. Changed to await asyncio.sleep(0), which yields to the event loop just as effectively without the delay. Chunk size is deliberately unchanged, so the yield frequency -- and therefore responsiveness during an upgrade -- is identical.

33. Matching the current second exactly meant a dose was lost outright if no tick landed on its second. Both scheduler loops ran if now in job["dose_times"] once per await asyncio.sleep(1). But that sleep guarantees at least a second, never exactly one, so whenever the loop had other ready work a tick advanced well over a second of wall clock and stepped straight over the seconds in between. A dose due on a skipped second was not retried, not logged, and raised nothing -- it simply never happened. Because scheduler_tick_count increments once per tick, dividing it by the device's own clock measures this directly: 0.705-0.746 ticks/s with two browser pages open (25-30% of seconds never examined) against 0.983-0.994 with none. Matching the schedule against the dose log over one 11.6 h window found 89 doses scheduled, 68 logged and every log row accounted for -- so 21 doses, 24%, about 106 mL, had never run, with the log sitting at 68 lines against a 200-line trim limit so nothing had been discarded. The decisive detail: two pumps that share a dose second disappeared together on all five occasions, which no per-pump comms or driver fault can produce. Fixed by firing on the half-open interval since the previous tick instead of on an instant, which makes dispatch correct however far behind the loop runs. The interval is committed before any dose is dispatched, so a dispatch that raises cannot reopen it and dose twice; it is exclusive at the previous second and inclusive at the current one; it handles the midnight wrap; the first tick after boot starts one second back rather than replaying the day; and intervals wider than 180 s are discarded rather than honoured, since an NTP correction backwards reads as a ~86400 s jump forward and replaying a long stall would dump a burst of doses at once. Simulated against this doser's own 192-dose schedule, the new logic delivers 192/192 with no duplicates at 0, 20, 430, 1500 and 5000 ms of lag per tick, where the old exact match delivers 134/192 at 430 ms.
34. A once-a-second poll for a cosmetic indicator was what made the loop run late -- and it was also failing doses. The "dosing now" dot from items #14 and #16 polled /schedule-diag every 1000 ms from each open page and compared times client-side. That endpoint sorts every job's dose_times (192 entries across 7 jobs on this doser) and serialises ~2.6 KB on every call, so each open page bought the device that work once a second. The interval fix above stopped this costing whole doses, but it did not stop it failing them on the run-command acknowledgement of item #31: over one 5.5 h run, 1 of 16 doses failed with pages open and 26% of seconds skipped, against 0 of 33 with pages closed and 1% skipped. In other words, watching the pumps degraded the pumps. Replaced with a new GET /pump-running?id=N that returns a single boolean -- 18 bytes against 2629, with no sort and no large serialise -- and the page polls that at the same 1 s cadence. Semantics are deliberately unchanged: still inferred from the schedule, still unable to tell a real dose from one that silently failed to start, still blind to a manual off-schedule dose. Computing it server-side also fixed a real edge case the client-side comparison got wrong, a dose that starts before midnight and is still running after it. Measured afterwards: a simulated page polling once a second costs no measurable lag at all (+8 ms/tick, identical to an idle device), and adding a live /dose-sse stream on top brings it only to +17 ms, against +403 ms before. That last measurement is worth stating plainly, because it was deliberately a one-change-at-a-time test: the per-second poll was essentially the entire cause, and the SSE stream -- the original suspect -- contributes almost nothing.

35. Trimming the dose log destroyed it outright if interrupted. log_dose_event capped the log by reading the last 200 lines and then reopening the same file with "w" -- which truncates it to zero before those lines are written back, so a reset, a power cut or an exception anywhere in that gap left an empty file. This is not a cosmetic loss: the 24% silent dose-loss bug of item #33 was undetectable until this log could be compared against the schedule, and nothing else on the device records what was actually dosed. Now writes a .tmp file and renames it, so a reader sees either the previous log or the complete trimmed one.
36. Container levels persisted only once an hour, and the write itself could corrupt them. storage_tracker slept 3600 s between checks, so any reset discarded up to 60 minutes of per-dose remaining-mL decrements -- not theoretical, three reboots in one afternoon of firmware work made it a real loss. Reduced to 60 s, so the worst case is a single unaccounted dose. That change was only safe together with the second half: load_configs.py catches an unparseable storage.json and silently substitutes all-zero containers, which also disables the "container empty" alert entirely, because that check skips any pump whose container size is 0. A half-written file is therefore actively destructive rather than merely stale, and writing six times as often without fixing the write would have multiplied exactly that hazard. All four storage.json writers now go through one atomic temp-file-and-rename helper -- including the one on the OTA path, which flushes levels immediately before a firmware write, precisely when a reset is most likely. Both renames fall back to remove-then-rename, since this board's partition is labelled vfs (so initsetup.py formats it littlefs2, where rename replaces an existing destination atomically) while the partition table declares subtype fat (where rename refuses one) -- rather than depend on which is mounted, handle both. Verified on the device rather than assumed: a dose was allowed to land, then the doser was deliberately rebooted, and all nine pumps' levels came back exactly as they were, confirming both that the write succeeds and that the tracker task survives it.

37. A background task that raised an exception was silently gone for the rest of uptime. Every long-running task -- the web server, the dosing scheduler, the MQTT workers, the storage writer, the Wi-Fi keeper -- was launched bare into asyncio.gather(..., return_exceptions=True). Because all of them are infinite loops, that gather never returns, so the "Task error" report written after it never ran either. A task that hit an exception simply stopped: no log, no alert, no restart, and nothing anywhere showing that it had ever existed. This is not hypothetical -- a dead web-server task is precisely the "ping replies, HTTP dead, scheduler still ticking" state this doser was found in on 2026-10-08, which needed a manual power cycle. Each task is now wrapped in a supervisor that restarts it with bounded backoff, alerts on its first crash (first only -- a persistently crash-looping task would otherwise spam every configured channel forever), and records the count, time and message in task_failures on /schedule-diag. A clean return is not treated as a failure, because several of these legitimately return rather than loop (the MQTT worker with no broker configured, the Telegram worker with no token).
38. Nothing could recover a hang; it needed a human and a power cycle. Added the ESP32 hardware watchdog, 120s, armed lazily on the dosing scheduler's first real tick rather than at startup -- the watchdog on this port cannot be stopped or retimed once constructed, so arming it before the loop is known to be running would turn any startup failure into a reboot loop. It is fed from the dosing scheduler specifically, not from a dedicated feeder task, so that "the loop which dispatches real doses stopped" is itself a reset condition; a separate feeder would happily keep the device alive with dosing silently dead.

Why these two had to be designed together. An earlier build of exactly this pair never came back up at all, and was rolled back without the cause being found -- the traceback exists only on a serial console, and the stepper drivers on this hardware share pins with it, so reading it means powering the pumps down first. The source of that failed build was instead recovered out of its own archived firmware image (the frozen filesystem stores each file as raw deflate) and read directly. That ruled out the obvious candidates: every supervised task name existed, the restart contract was correct, the watchdog was already lazily armed and guarded, and every module-level addition was a harmless assignment. What it left was an interaction: the watchdog panics and reboots on a missed feed, it was fed from the scheduler loop, and the supervisor's backoff grew to 300s -- past the 120s watchdog. A crash-looping scheduler would then be rebooted by its own watchdog, and because that build also made boot skip re-extracting the app, each reboot took seconds instead of ~97s. The result is a fast reboot loop that never stays up long enough to answer a ping, which is exactly what was observed, and which none of the three changes produces on its own.

That remains a hypothesis rather than a proven cause. So this build does not rely on it being right: the backoff is capped at 30s against the 120s watchdog (a 4x margin), the watchdog feed is the first statement in the loop body so every restart attempt feeds at least once before anything that could raise, boot.py is byte-identical to the previous build and still extracts unconditionally, and startup falls back to launching the tasks exactly the old way if the supervision wrapper throws for any reason -- an unreachable doser cannot be rolled back over the air, so that path must never depend on new code. Both web.py and boot.py were extracted back out of the finished firmware image and checked byte-for-byte against the reviewed source before it was installed.

39. A pump whose container is empty keeps being commanded to dose, and the only warning was one that fires on every dose. The remaining-volume counter cannot detect a dry bottle -- it only ever subtracts the doses it was told to perform -- so the best available signal is "this pump has been reading zero for a long time while still being dosed". There was already a "Container empty" message, but it fired on every single dose, which is the shape people learn to tune out, and in practice it was tuned out: on this tank the magnesium container read 0 mL for ten days (2026-09-27 to 10-07) while 46 doses a day were commanded into an empty bottle, and magnesium fell about 60 ppm before anyone noticed. Added a separate, state-aware alert that fires once when a pump first starts dosing against an empty counter and once a day after that, stating how many days and how many thrown-away doses, and exposes empty_for_days and doses_into_empty per pump on /storage-status. Worth stating the limit precisely: this detects dosing into an empty container, not an empty container -- a pump that never doses (this doser has one with a failed driver board) stays at zero without ever being flagged, which is the right behaviour for that case but means the feature is not a tank-wide level monitor.

40. Three ordinary malformed requests returned HTTP 500 instead of the 400 the handler was written to give. request.args.get(name, type=int) looks safe and is not: it raises on a value that is present but not numeric, raises on an empty value, and request.args is absent altogether when the URL carries no query string at all. So /pump-diag, /pump-config and /pump-running all answered an opaque "Internal server error" to exactly the kind of hand-typed request someone makes while debugging something else, which is how it was found. Now routed through a helper that returns None for anything unusable and lets each handler's own validation produce the real message. Deliberately not applied to the arguments that carry a real default (/dose and /run's pump id and direction): there a malformed value raising is the safe failure, because coercing it to a default would mean a request with a bad id silently dosing pump 1 instead of refusing. A first attempt at this blamed the wrong thing -- the return {...}, 400 return shape -- which the web framework actually handles correctly; that fix was reverted before it shipped once a live request proved a real 400 came back fine.
41. The machine module was never actually imported. machine.reset() (the reboot route) and now machine.WDT() only resolved because a wildcard import of the config module happened to pull in that module's own import transitively. It worked, but a watchdog whose construction raised NameError would have been swallowed by its own guard and reported as unavailable while the device looked healthy -- a silently inert watchdog is worse than none, because it is believed. Now imported explicitly.
42. The OTA page warned "rollback pending" after every reboot, including for builds already confirmed. The flag was a plain in-memory value reset on every boot, so a confirmed firmware kept being reported as unconfirmed -- the kind of false alarm that teaches people to ignore a real one. Now persisted per release tag, written atomically.

43. The dose-failure alert collapsed every cause into one sentence, after the log had already been fixed not to. Item #21 threaded a specific reason through every failure path into the persisted dose log. The notification that actually reaches a person kept sending a single fixed string, "driver unresponsive or rejected command", for all of them -- reproducing in the one channel someone reads the exact problem the log had just stopped having. Observed on this tank on 2026-10-11 22:54, where the log said the alkalinity dose had most likely landed and the alert said only that the driver was unresponsive. The distinction matters at 3am: "run started but no completion confirmation" needs no action, "no reply to run command -- motor very likely never started" does.
44. The constant-speed command existed but could not tell refusal from silence. move() wrote the driver's 0xF6 "run in constant speed" command and never read anything back, so it could not distinguish a driver that explicitly refused from one that was not listening at all -- the same distinction that made the two failed driver boards on this tank legible in the first place. Added move_async(), which reads the documented status byte and returns the three-state accepted / refused / no-reply used everywhere else in this driver.
45. New POST /pump-constant. A bounded constant-speed run, for two purposes: diagnosing a driver that refuses counted moves but might accept constant speed, and proving out continuous-feed duty where a pump turns at a fixed rate and step accuracy is irrelevant. Refused outright for any pump that has a dosing schedule, because a constant run is not dose-accounted and would empty a container with nothing in the dose log. The stop is issued from a finally block and the duration is capped regardless of what the caller asks for. The one design point that matters: the sleep happens outside the bus lock, so a dose falling due mid-run queues behind a single short write instead of the whole duration.

46. One corrupted byte in the driver's reply threw away a dose that had already started. With reason strings now in the log (#43), the next alkalinity failure was readable for the first time, and it was not a failure. The driver's "run starting" acknowledgement arrived as 95 01 e6 where e5 01 e6 was expected -- status byte correct, checksum correct, only the address byte corrupted on the wire. Because the checksum is computed over address plus status, a valid checksum proves the driver itself had the right address, so the corruption happened in transit and the motor had genuinely started. Nothing on that code path ever sends a stop, so the full dose was delivered; the firmware then logged FAILED, sent an alert, and skipped the volume accounting. The acknowledgement is now matched on status and checksum when the full three-byte pattern does not match. That stays pump-specific rather than a blanket relaxation -- pump N's two-byte form is unique because the checksum carries the address, and a false match would require a frame ending in a checksum of 0x01, meaning a bus address of 1 or lower, which no driver here can produce. Scoped to the run command alone: every other command either retries or is a diagnostic whose failure costs nothing, so relaxing them would be loosening a check with no evidence behind it. The count of acknowledgements accepted this way is exposed per pump on /schedule-diag, because the parsing absorbs the symptom while the cause stays physical.
47. "Motor most likely finished" was filed as a failed dose and never deducted from the bottle. When the completion acknowledgement never arrives, the firmware's own reason string said the motor had most likely finished and this was a comms problem rather than a missed dose -- and then returned failure anyway, which skipped the remaining-volume update. Over 2026-10-09 to 10-13 that mislabelled four of this pump's five logged failures, roughly 5.9 mL of hydroxide delivered into the tank but never subtracted from the container. Such a dose now counts the volume and is logged as UNCONFIRMED, a third state that is neither OK nor FAILED, with an alert worded to match. The two copies of the outcome-handling block were also collapsed into one, since this file's own history records three rounds of patching individual failure paths that still missed one.
48. A reported RPM that was arithmetic on a flag. /pump-constant reported the speed it was running at by dividing by the pump's microstepping -- which, for any pump that has never run a counted dose, is still the sentinel value meaning "leave microstepping alone" rather than a real subdivision. The result was a confident, wrong number: 3.003 RPM where the true figure at a normal setting is 187.5, out by a factor of 62. It now returns nothing rather than a fabricated figure when the subdivision is unknown, and accepts an explicit value to set it and get a real one. This mattered immediately, because that route is how a continuous feed pump's rate gets calibrated.
49. Nothing bounded how long a dose could sit in the queue before running. There was already a cap on how wide a catch-up window the scheduler will replay. That is a different thing from how long a dispatched dose then waits in the command queue behind the single worker that owns the shared bus -- every pump shares it, and the longest dose on this tank runs 133 seconds. A backed-up queue delivered stale doses in a burst whenever it drained. A dose more than 600 seconds late is now dropped and logged STALE rather than delivered. The number is measured rather than chosen: simulating this tank's live schedule, 196 doses a day across seven jobs, through one worker gives a worst legitimate wait of 247 seconds and a 95th percentile of 109, with nothing above 300 -- against a tightest per-pump interval of 1878 seconds, so a dropped dose can never crowd the next one.

A wrong turn worth recording. While looking for what the corrupted replies had in common, the dose log appeared to show a perfect correlation: every failure on one pump happened within seconds of another pump's dose, and none of its 171 isolated doses had ever failed. That was an artifact of reading the log wrong. Its timestamps are written when a row is written, which for a successful dose is when the dose ends and for a failed one is when it aborts -- so treating them all as start times made a routine back-to-back pairing look like a rare and perfectly predictive one. Recomputed on reconstructed start times the correlation disappears entirely: 2.1% failures among doses starting while another motor was running, against 2.5% on an idle bus, and the pump pairing that looked decisive occurs 175 times with six failures, which is simply that pump's base rate. The residual ~2.3% across healthy pumps remains unexplained by anything visible in the schedule.

50. Constant mode on any channel, with a watchdog that checks the shaft rather than the driver. A denitrification reactor needs its feed pump turning continuously at a fixed rate instead of running counted doses. The bounded test route from #45 is deliberately capped and is not a run mode, so this is a real per-pump setting -- on or off, speed, direction -- available on every channel rather than special-cased for one. Desired and actual state are reconciled every five seconds, so a change made in the interface takes effect in seconds, and one branch covers first start after a boot, a speed change, and re-asserting after a driver has lost the command. The motion check is the part worth explaining: it reads the magnetic encoder, not a commanded-pulse counter, because the two failed boards on this tank proved a driver can accept a command, report success and never turn -- a pulse count would have agreed with the driver. Two consecutive confirmed-stationary reads raise one alert and re-issue the command; "no reply at all" is counted separately and says so, because it is a different fault. A pump with a dosing schedule cannot be switched into constant mode, enforced at the single point where schedules are built so the rule holds however a schedule arrived -- interface, MQTT, an add-on, or a hand-edited file. The interface shows the resulting RPM as the speed is typed, which is what makes the setting dialable to a real flow rate instead of guessed at.
51. The nine-pump limit was inherited, not electrical. The driver's own address setting runs 0xe0 to 0xe9 -- ten devices, per the manufacturer's manual -- and every stepper shares one serial bus, so a tenth driver needs no additional pin. The cap was simply what the upstream project shipped. Raising it was not a one-line change, which is why it had not been done: two configuration loaders filled in missing per-pump entries only in their load-failure branch, and three per-pump arrays were written as literal nine-element lists. A device with existing configuration files would therefore have failed during startup -- before the web server or the scheduler exist, which is to say unreachable for an over-the-air rollback -- while a factory-fresh one came up fine. Every per-pump array now pads on read, and both loaders fill gaps on the success path too. Verified against this tank's real configuration before shipping.

52. A continuous-running mode that could not report the rate it was running at. The previous build added a microstepping argument to the bounded test route but not to constant mode itself, so the interface could only say "rpm unknown -- this firmware has not set microstepping on this pump". That defeats the purpose: the rate is displayed precisely so a reactor feed can be dialled against a measured flow. Microstepping is now part of each pump's constant-mode configuration and is applied before the run command, which makes the figure derivable — and if setting it fails the pump deliberately does not start, because a mode whose entire point is a known rate should refuse to run at an unknown one. The interface computes the resulting RPM from speed and microstepping as either field is changed, with no round trip to the device. A second fix came out of the same session: after a stall the loop re-issues the run command as its one automatic recovery, which was resetting the displayed state back to "running" — on a permanently stalled pump that produced a display flipping between "running" and "not turning" every few minutes, contradicting the alert that had already been sent. It now says "re-asserted after stall" until real motion clears it.

53. Setting the acceleration parameter to 0 stops a loaded pump turning, while every test short of actually dosing still passes. Two of this tank's nine pumps had been written off as failed driver boards. Both accepted every parameter write, both ran encoder calibration that physically turned the motor through two clean revolutions, both reported success to every motion command, and neither moved. A counted move trailed off at exactly 18° — ten full steps — and merely enabling the motor saturated the position error with nothing touching the shaft. Replacement boards were ordered.
    
    The cause was this firmware. The advanced-settings page declares a default for every driver parameter it exposes, and for acceleration it declared 0. That value was taken from the manufacturer's on-screen menu, which lists the option as "Disable / 286 / ... / 1042 (Default: Disable)". The serial-command section of the same manual states the default is 0x11e, which is 286. The two halves contradict each other, and the wrong one was followed. "Set the defaults" was then run against those two pumps on 2026-10-07 and 10-11 respectively, and against no other pump — which is exactly why two independent boards failed identically, the one detail that never made sense.
    
    Acceleration 0 means no ramp: the driver commands the target speed from a standing start, and a loaded peristaltic head cannot follow it, so the rotor loses sync within a few steps. That is the 18° trip. Open-loop calibration is unaffected because it steps the motor blindly and never asks it to accelerate, which is why every diagnostic short of a real dose kept saying the hardware was fine.
    
    Setting 286 brought the first pump back immediately: twelve minutes of continuous running with the encoder confirming rotation on every check and no stalls, where before it had tripped within one check every time, followed by a counted move that completed and logged normally. The second pump was reconnected and given the same value on 2026-10-14 and came back the same way. Both pumps are fully recovered; no hardware was ever at fault, and the replacement boards ordered earlier are not needed. The spec now declares 286 in both places it existed — the server-side route handler and a duplicate copy hardcoded into the settings page's own JavaScript, which the first attempt at this fix missed, since the page's own comment warning they must be kept in sync by hand went unread.
    
    Two things worth keeping from how long this took. The firmware already recorded every advanced setting it had ever written, per pump, and that record named the culprit in one line — it was never consulted, because the hardware was assumed faulty from the start. And the manufacturer's Chinese documentation is materially more accurate than the English translation: it states that "Phase Line Error" and "Reverse Lookup Error" both mean the motor wire sequence is wrong (the English says "check magnet and motor shaft" for the second), that "Magnet Error" is an encoder-IC communication fault on the board rather than a magnet problem, that the supply range is 7–28 V rather than 12–24, and it carries a warning about PID parameters that the English edition omits entirely. It also documents a screen-free way to tell a wiring fault from a healthy one: during calibration a correctly wired motor turns one full revolution each way, while a mis-wired one moves back and forth. That test is what ruled wiring out here and forced the search somewhere else.

Current build hosted at /downloads/firmware_smartdoser/ is built from this exact fixed source tree (verified by sha256 match against the local build output before publishing).