I think @dlang is right that we do have an issue with new releases breaking old features, but Iām not 100% sure more manual testing is the solution.
with hardware involved, manual testing is unavoidable. The key is to plan out a
test plan that tests as many different things in one run as possible so that you
do as few manual runs as you can get away with.
Having multiple machines, some of which are much more limited than the full
machine, but are enough to perform specific tests can help.
I feel like I have so little time, and there are soooo many things to test
that it would take a whole day each week just to be sure the release has no
bugs.
What many software projects do is to have a āmerge windowā for changes that
occupies part of the release cycle, after that window, only bugfixes/reversions
are accepted, any further improvemetns must wait for the next cycle.
In our case, I would suggest that the merge window open on Wednesday[1] when you
cut a release, and close Saturday morninng. This would mean that as of Saturday
morning, the version in Git is expected to be the version that will be released
the following Wednesday. People can then test this version over the weekend and
report problems. Meanwhile, people can experiment and develop on different
branches to produce PRs, but the PRs will not be merged until after the next
release is cut (unless they are fixes to problems discovered in testing)
This lets everyone else perform tests so you donāt have to do them all
I tried playing with an automated testing system like CircleCI, but because
hardware is a factor itās tough.
Yep, most of the real problems we are having are hardware related and automated
testing in simulations would not catch them.
Iām hoping soon we can stop adding so many new features and move to a release
every two weeks or even every month. That way I can dedicate multiple days to
testing each new version.
I donāt I hope the development speed stays high
David Lang
[1] with this sort of testing, it make make sense to move the release day up to
Monday or Tuesday to give a little more time for adding things. But since we are
almost all hobbiests, having a release candidate available for testing over the
weekend is valuable