Snapshot/restore flash to recover known-good Maslow position after a reboot?

Follow-up to the controller-reboot thread I just posted.

After the unclean reboot, the boot log showed:

[MSG:INFO: TL encoder moved 540 counts (-5.797mm) since save, adjusting position]
[MSG:INFO: TR encoder angle difference too large (-1744 counts), treating belt positions as stale]

Machine then sat in Alarm with NaN MPos and forced a full Retract → Extend → Tension recovery — even though physically nothing moved during the ~27 ms gap between the last <Idle> status and the firmware boot banner. I think the TR “movement” was the firmware comparing a stale snapshot to a fresh encoder reading, not actual belt motion.

Here’s my mental model of the storage architecture on the Maslow S3 board, based on poking around the FluidNC files tab and reading boot logs — happy to be corrected:

  • Program flash partitionfirmware.bin itself, uploaded with the orange-cloud icon in FluidNC.
  • LittleFS data partitionmaslow.yaml (kinematics + calibration grid), preferences.json, config-bak.yaml, index.html.gz. These are the files we back up before flashing anything.
  • NVS partition (ESP32 key-value store) — where $$ settings live, and presumably where the periodically-saved encoder counts and Z position live, since the boot log clearly reads back Current z-axis position loaded as: 70.000 and the per-axis “moved N counts since save” lines.

So my actual question:

Could esptool.py over USB be used to snapshot the partition that holds the live encoder/Z save, when the machine is happily in state 7, and then write_flash that same image back after a glitch — so the firmware comes up trusting the cached state and skipping the re-tension cycle?

Three related questions I’d love answers to:

  1. Which partition actually holds the runtime encoder/position snapshots? NVS, LittleFS, a custom partition, or somewhere else?
  2. Does FluidNC integrity-check whatever it’s reading on boot? i.e. would a hand-restored copy from esptool.py write_flash be accepted, or rejected as tampered?
  3. Has anyone scripted periodic NVS snapshots as a “save game” for the controller — write the snapshot to the host whenever the machine reaches state 7, restore it after an unclean reboot? Or is there a known reason this won’t work even in principle?

End goal: when the controller glitches in a way that I can confirm wasn’t a physical-motion event, I’d like to be able to restore the just-prior known-good state and skip the 90-second tension dance.

Thanks.

1 Like

Wes wrote:

Follow-up to the controller-reboot thread I just posted.

After the unclean reboot, the boot log showed:

[MSG:INFO: TL encoder moved 540 counts (-5.797mm) since save, adjusting position]
[MSG:INFO: TR encoder angle difference too large (-1744 counts), treating belt positions as stale]

Machine then sat in Alarm with NaN MPos and forced a full Retract → Extend → Tension recovery — even though physically nothing moved during the ~27 ms gap between the last <Idle> status and the firmware boot banner. I think the TR “movement” was the firmware comparing a stale snapshot to a fresh encoder reading, not actual belt motion.

there are two ways that the position can be invalid on startup.

  1. as the machine starts to move, it marks the current data as invalid (it’s not
    possible to keep it update in real time the chips have a finite number of writes
    in their lifetime, it’s hundreds of thousands or millions of writes, but it’s
    finite. trying to store it contually just doesn’t work. We save the position
    when the machine switches in to idle mode

  2. what you ran into here. It is reporting that the position of the encoders at
    startup does not match the position they wee in when the last position was
    saved. the encoders are 4096 counts/rev. Each rev is ~40mm (I don’t remember if
    it’s exact or not). note this is an instantanious position, not a cumulative
    position, each full rotation results in the exact same position being reported.

The code allows up to 10mm of error between the two (1/4 rotation of the encoder
either way). in theory we can correct for errors up to 1/2 a rotation, but if
you go just past 1/2 a rotation, it will think you rotated the other direction
just under 1/2 a rotation, so I was generous is setting a dead zone that is just
considered invalid.

In this case, it’s reporting that one encoder was different by 5.7mm (~1/8
rotation) and it adjusted for that, but the other was off by ~17mm (almost half
a rotation)

so, you may not have realized it, but the encoders did move from the point where
they were last saved.

Here’s my mental model of the storage architecture on the Maslow S3 board,
based on poking around the FluidNC files tab and reading boot logs — happy to
be corrected:

  • Program flash partitionfirmware.bin itself, uploaded with the orange-cloud icon in FluidNC.
  • LittleFS data partitionmaslow.yaml (kinematics + calibration grid), preferences.json, config-bak.yaml, index.html.gz. These are the files we back up before flashing anything.
  • NVS partition (ESP32 key-value store) — where $$ settings live, and presumably where the periodically-saved encoder counts and Z position live, since the boot log clearly reads back Current z-axis position loaded as: 70.000 and the per-axis “moved N counts since save” lines.

the encoder position is not in the flash, that’s in a separte eeprom. eeprom can
be written a byte at a time while flash has to be written a block at a time
(commonly 128k last I looked)

there are actually two firmware partitions and two data partitions. This is how
an update can fail without bricking the machine. it writes to the new
firmware/data partition and then switches to using them (with some other
trickery to sync the data partition. I had a bug one time that broke the data
partition, every other reboot worked, but the odd reboots had a different config
and failed. That was ugly to track down until I realized it was exactly every
other reboot :slight_smile:

So my actual question:

Could esptool.py over USB be used to snapshot the partition that holds the
live encoder/Z save, when the machine is happily in state 7, and then
write_flash that same image back after a glitch — so the firmware comes up
trusting the cached state and skipping the re-tension cycle?

I’d need to check, but if it’s idle when it shuts down, I think it still
considers the saved position valid

Three related questions I’d love answers to:

  1. Which partition actually holds the runtime encoder/position snapshots?
    NVS, LittleFS, a custom partition, or somewhere else?

I believe it’s a different chip

  1. Does FluidNC integrity-check whatever it’s reading on boot? i.e. would
    a hand-restored copy from esptool.py write_flash be accepted, or rejected as
    tampered?

the error you show above is the first round (and was initially only round) of
integrity checking. it checks that the angles on the encoders haven’t moved too
much. In addition, when you apply tension, it checks the belt lengths to see if
they compute to a valid position based on the anchor locations it is configured
with. (that is another common error you see people reporting)

The initial version of saving the belt positions didn’t allow you to relax
tension, it was designed only to work if you were fully retracted or left
tension on the belts.

we could widen the allowed error and rely on the apply tension results to catch
everything, but I’ll admit to being a bit chicken to try that.

  1. Has anyone scripted periodic NVS snapshots as a “save game” for the
    controller
    — write the snapshot to the host whenever the machine reaches
    state 7, restore it after an unclean reboot? Or is there a known reason this
    won’t work even in principle?

the problem is that in the time between moves, the belts can move a long way and
the data you are trying to save just isn’t anywhere close to valid.

End goal: when the controller glitches in a way that I can confirm wasn’t a
physical-motion event, I’d like to be able to restore the just-prior
known-good state and skip the 90-second tension dance.

once you lose track of the belt position, the only current way of retriving it
is to do a full retract.

I’ve though about the idea of having a bolt somewhere that you could clamp the
spindle too and pull tight there instead of retracting to 0. the big advantage
would be not having to detach all the belts. but I haven’t experimented to test
if this is really accurate enough in practice.

It may be possible to just do apply tension and see if the belt position
computes with ‘sufficient’ accuracy, but today that alarm is triggering on a
belt being off by 12 of 15mm, if you were going to try to use it to get a truely
accurate position, you would want to have your answer down to sub 1mm (even
harder to convince myself it’s going to be valid)

David Lang