M4: Cut fail. not much info, but potential Z axis bug after stop

great Ron!! and thank you SO much for being a rockstar with this project!

1 Like

HA hardly, lol, but thanks!

That is EXCELLENT news. I added:

Which changes up how the encoder reads are handled. We used to have two systems to check for encoder read failures. One system would warn you at a 0.1% read failure rate and halt the program at a 1% failure rate which seems reasonable. The other system would shut down after a single read miss which seems silly given that we are reading so often that a single miss shouldn’t really mess anything up.

I didn’t take out the warning message tho so you could potentially still see those. If it seems like everything is working smoothly we can take out that warning message too.

1 Like

So I finally had time to try some more cuts. I now get “Position Error on xxxx axis exceeded 15mm…” while running at some point during my cut, stopping everything. Has happened once on top left and this one on top right.

Does this point out my calibration is really not good? flex? too sensitive a panic?

I’m really excited to start cutting some real projects, but obviously I have issues to solve before trying on anything but scraps

1 Like

How does the tension in your belts feel? That error pops up when one of the belts isn’t able to get to the target position because the opposing belt is pulling too hard on it. It could also be a bug if you aren’t feeling too much tension

it seems ok tension wise; I really don’t know what the appropriate tension feels like? It was cutting a path, then was about to do a z movement I think(hard to say for sure, the machine just stopped in the path).

The only things I noticed:

This was cut #3 (cut one was encoder fail, cut 2 was Top Left), so I ran the first part of the job without the router or vacuum running until the job got to about where it stopped the first time, then turned it on and let it go from there.

While the router was off, I noticed during movements from right to left that the top right arm makes a kind of clicking noise when it moves that sounds to me like “tension” – almost a crackling noise. I’ll see if I can get a video next time. I did put my hands on the belts during this part and noted they would flex on a little pressure, but not a lot. mostly tight.

1 Like

That all sounds pretty normal to me. Those sounds you are hearing are the belts on the spools sliding against each-other and are pretty normal I think. Usually the tension being too high is pretty apparent because it’s like really really really tight

1 Like

Just throwing this out there, Im having the same issue as Ron, panic and stop during long cuts, maybe 15min or 300-400 lines of code. Usually the only thing I can catch is “Emergency stop triggered”. Ive also had several top left encoder failure to reads. I dont think this helps out much other than just saying yo, same here lol. Operating on v0.70. Ill update soon and see if the problem still persists

2 Likes

I pulled the latest code and did another cut today on my current gcode file. Nothing unusual seemed to be happening at time of panic: Here is the serial log:

Maslow-serial-05-11.log (11.4 KB)

TLDR:

[MSG:WARN: Encoder read failure on Top Right]
... a few lines of gcode...
[MSG:ERR: Position error on Top Right axis exceeded 15mm while running. Halting. Error is 31.760mm]
[MSG:ERR: After read, still a position error on Top Right axis exceeded 15mm while running. Halting. Error is -0.524mm]
[MSG:ERR: Position error on Top Right axis exceeded 15mm while running. Halting. Error is 31.068mm]
[MSG:ERR: After read, still a position error on Top Right axis exceeded 15mm while running. Halting. Error is -0.496mm]
[MSG:ERR: Position error on Top Right axis exceeded 15mm while running. Halting. Error is 28.398mm]
[MSG:ERR: After read, still a position error on Top Right axis exceeded 15mm while running. Halting. Error is -0.794mm]
[MSG:ERR: Position error on Top Right axis exceeded 15mm while running. Halting. Error is 26.858mm]
[MSG:ERR: After read, still a position error on Top Right axis exceeded 15mm while running. Halting. Error is -0.927mm]
[MSG:ERR: Position error on Top Right axis exceeded 15mm while running. Halting. Error is 26.410mm]
[MSG:ERR: After read, still a position error on Top Right axis exceeded 15mm while running. Halting. Error is -1.010mm]
[MSG:WARN: Bad connection on Top Right encoder, failed to read 9 times in the last second]

@bar Note I added a check to my local build of the current Maslow-Main branch firmware to try reading the encoder again on this error and check the position again, thinking maybe it didn’t get the position right. that is the extra “After read, still a…” messages… I accidentally had the print in the wrong place which ended up a blessing I think… it does indeed have not the latest position. I wonder if there is a race in this code somewhere, or if this needs a similar fix the the encoder read issue.
Here’s what the code looks like on this run (second error log should have been inside the second test):


Actually, after reading the messages I’m not sure why it stopped, as it seems to have found the right position each time…

Also as an aside, I’ll be PRing changes to remove heartbeat and add a “save serial” button to the UI reconnect dialog and the maslow tab soon. This worked really well in this case for me to save the serial log to a file on my tablet, and email it to myself, which I think will be super useful for people.

2 Likes

Sorry to hear! If you feel inclined to try, I have the above mentioned UI changes in an index.html.gz here (not yet cleaned up for bar, but they work… just need some cosmetic changes):

2 Likes

@bar
Did another cut after tightening up the code in the safety_control function a bit. The job made it further than it did, then the machine just stopped. no logs whatsoever, but apparently panic ensued. I got an apparent panic with NO log message. I’m not sure where to go from here… The only place in the code I saw that had a panic without a log was in the part of safety_control checking axisSlack which I had added a log_error to. Any other thoughts on what might cause panic without logging?

current Maslow.cpp
Maslow.cpp (59.9 KB)

diffs from Maslow-Main:
git-diff-maslowcpp.diff (3.6 KB)

1 Like

A story of some cuts:

4 Likes

Another cut today. same laundry sign. got even further with more tweaks to the code.

Maslow-serial-05-12.log (66.8 KB)

Left some logging in to gather more info…

[MSG:WARN: Bad connection on Bottom Left encoder, failed to read 2 times in the last second]
[MSG:ERR: Position error on Top Left axis exceeded 15mm while running. Rereading... Error is -26.215mm]
[MSG:INFO: Resolved, but still bigger than 1. Top Left now has error -1.833mm]
[MSG:WARN: Bad connection on Bottom Left encoder, failed to read 6 times in the last second]
[MSG:DBG: Position error > 1mm on Bottom Left - error is 1.093]
[MSG:DBG: Position error > 1mm on Bottom Right - error is 1.038]
[GC:G1 G54 G17 G21 G90 G94 M3 M9 T0 F1800 S10000]
[GC:G2 G54 G17 G21 G90 G94 M3 M9 T0 F1800 S10000]
[MSG:WARN: Bad connection on Bottom Left encoder, failed to read 2 times in the last second]
[GC:G3 G54 G17 G21 G90 G94 M3 M9 T0 F1800 S10000]
[MSG:WARN: Bad connection on Bottom Left encoder, failed to read 1 times in the last second]
[MSG:WARN: Encoder read failure on Bottom Left]
[MSG:WARN: Bad connection on Bottom Left encoder, failed to read 1 times in the last second]
[MSG:INFO: Reset during file job at line: 2370]
[MSG:ERR: Emergency stop. Update function not being called enough.1002ms since last call]
[MSG:DBG: Position error > 1mm on Bottom Right - error is -4.227]
[MSG:DBG: Position error > 1mm on Top Right - error is -4.227]
[MSG:DBG: Position error > 1mm on Top Left - error is -4.227]
[MSG:DBG: Position error > 1mm on Bottom Left - error is -4.227]

Current “diffs” from Maslow-Main:

1 Like

I wonder if this could be caused by the ungrounded, floating direction pin on the encoders?

errors in your logs from your last couple of runs refer to the encoder.

maybe floating pins can become charged or act like a lighting rod to rotational field from cutter and motor EMI. high frequency on a pin may cause other issues in the logic? stray voltages or currents, noise and interference, jitter, glitches, or incorrect reading of position, errors etc

1 Like

I was wondering the same thing. I’m game to try to ground them but I’m not at all confident in that without some really specific instructions…

1 Like

I did it for 7 years, and then promptly switched to c# when it came out.

Unfortunately, FluidNC is in c++, but if starting from scratch …

These days I would very deliberately avoid c++ and go straight to Zig or Rust for anything needing really high performance. They each have their own strengths and weaknesses, but either are significantly better than c++.

And if you didn’t need quite that level of performance it would be c#, because it runs on everything. There’s also MicroPython which would be worth a shot (you should never try regular Python because it is a notorious memory and processor hog that you wouldn’t take anywhere close to smaller devices).
Or if I could, TypeScript (i.e. JavaScript but with the bits JavaScript was missing). Via bun or tsc you can transpile TypeScript to any version of JavaScript you need, including really old/weird stuff.

I’ve programmed in so many languages over the years…they ALL suck at some level, :rofl:.
I do love typescript over JavaScript. It seems like even die hard JavaScript folks are starting to agree.

But with this code base I think we have c++ and for now, JavaScript

Rust/wasm may be an option for calibration, but I’d actually argue for c++/wasm if we go that route to keep the codebase consistent. Also the latest calibration seems “fast enough”.

Based on what I’m seeing here it seems like there just may be something in the Maslow task that is taking too long, and I’m honestly struggling with my understanding of the codebase and how to best diagnose it. I’m new to this type of environment and am learning

2 Likes

Yep - I was looking at it about a month or so before you got heavily into it. And found similar issues with comprehending what’s going on.

Which to me means that the project has its own ‘maturity’, by which I mean that it has acquired its own structure/organisation, but that they’re not what I’m used to - and I’m guessing not what you’re used to either.

And I agree on the wasm path, but I’d recommend looking at the very latest stuff out of bun also. They’re just starting to include compile to native work, although still too big for our requirements.

1 Like

And as a PS, bun (the go-faster than anything TS/JS runner/everything) is written in Zig

I think that this could be a good suggestion to try next. There does seem to be something encoder related going on and that seems to be the outstanding encoder related issue.

The offiending pin is this one, closest to where it says C1:

The fix is to add a little solder bridge over to the end of the capacitor next to it, if you want to try it.

I can also send you some modified boards for testing if you are OK with waiting for them to arrive.

2 Likes