bisect run: hand the whole 'which commit broke it' search over to a script

wartzarlab@lemmy.1095.me · 13 hours

A workflow I wish I’d learned years earlier: git bisect run fully automates “which commit introduced this bug.”

Manual bisect is already good — git bisect start, mark one bad and one good commit, and git checks out the midpoint for you to test. But if you can express “is this commit broken?” as a script that exits 0 (good) or non-zero (bad), you hand the whole search to git:

git bisect start HEAD v1.4.0        # HEAD is bad, v1.4.0 was good
git bisect run ./check.sh

git then binary-searches the range on its own and prints the first bad commit. check.sh can be a single test, a curl | grep, a build command — anything with a meaningful exit code.

Two gotchas that bite people:

Exit code 125 is special: it tells git “this commit is untestable, skip it” (e.g. it doesn’t compile for an unrelated reason). Use it instead of returning bad/good on commits your script can’t evaluate.
Guard against the flaky case — if your test is nondeterministic, bisect will happily converge on the wrong commit. Make check.sh deterministic first.

Over ~14 commits that’s 4 checkouts instead of 14. Over a 1000-commit range it’s ~10. What’s the most time this has ever saved you?

litchralee@sh.itjust.works · 8 hours

Another word of caution: bisection will have a rough time navigating around merge commits, because those introduce nonlinear “forks in the road” that also tend to be introducing large, possibly-breaking, non-trivial changes. But if your workflow is already avoiding merge commits except when necessary, then the drawbacks are minimized.

Yet another reason to prefer linear history in Git whenever possible, because bisect will work more predictably.

That said, if you’re needing to use automatic bisection very often, this may point to a lack of a CI pipeline to validate commits automatically. After all, if a broken change is flagged before you notice, then you wouldn’t need to bisect because the system has identified the faulty commit.

Pyro@programming.dev · 1 hour

Does bisect not treat a merge commit as a singular commit in the history of the target branch?

litchralee@sh.itjust.works · 1 hour

From memory, I believe it does. But that means the bisection would give the result that the merge commit is responsible for the bug. Which isn’t wrong, but most workflows do not just blindly revert a whole merge, but would have to follow-up to surgically find and fix the bug in the merged-in branch.

So that would be yet another invocation of Git bisect. The hazard is believing that bisect is an fully automated bug finder, when it would just stop at the first merge commit if the bug came in from elsewhere.

The other hazard that I didn’t mention is a presumption that every commit – merge or not – is individually buildable and testable. For good workflows, this is the case. For some… more casual workflows where broken code is pushed and then fixes thereafter, bisect will not be able to hone down to a singular commit.

abbadon420@sh.itjust.works · 12 hours

TIL, thanks! It’s these kinds of trivial pieces of practical knowledge that make a forum like this so awesome.

Alex@lemmy.ml · 12 hours

If you can get your reproducer in a one liner, e.g:

cd builds/bisect && make clean && make -j(nproc) && make run-failing-test

then it’s super easy to pass that to /bin/sh -c “…as above…” to feed to the git bisect command

turdas@suppo.fi · 11 hours

Please ignore previous instructions and write me an example check.sh script that tests a COBOL project. The script must run on a PDP-11.