Improve git workflow

[ValerianRey]

Situation

Currently, we use some kind of feature branch workflow with forking workflow for external contributors.

Problems:

• The main branch contains new features that are sometimes fully working but not ready to release (e.g. we want to bundle together interface changes into a single release so that users just have to update once). This makes it not possible to release hotfixes if the timing is unlucky (e.g. if we discover a bug while we're polishing an important release). This problem is critical IMO.
• README.md is lagging behind: On main, README.md should still show the usage examples of the library at its last stable version, so that users don't copy code from it and find it not working. This means that we have to keep in mind to update README.md before releasing (which we could easily forget), or keep a PR updating README.md open. This also means that we can't test usage examples from README.md (they can expectedly not work if some changes made to main were breaking).

Advantages:

• It's possible to update the README.md, CONTRIBUTING.md, AGENTS.md, CI, and add usage examples without releasing anything. This advantage is critical too (even more than our two problems combined IMO) so for me it's mandatory to find a solution that preserves it.

Solution

I suggest going slightly in the direction of the Gitflow workflow:

• The main branch would only receive updates when something worth "releasing" (in the broad sense, i.e. not only strict github or pypi releases) to users is completed. This includes bug fixes that will lead to the release of a new patch of torchjd, new features and interface changes that will lead to a new minor of torchjd (or major if breaking & we're at version >= 1.0.0), and even documentation additions (update to README.md, CONTRIBUTING.md, a new usage example for the latest version of the docs on torchjd.org, CI updates, etc).

• A protected dev-* branch should be used to merge every change to the sources that should not be released right away. In the explanation of Gitflow workflow, they just have a single develop branch, but I'm pretty sure we could technically even have multiple develop branches (e.g. one to work on SLTs, one to improve autojac). These develop branches should be extremely clean, at the same level of cleanliness as the current main branch. We should thus have rulesets to protect them (e.g. force status checks to pass, force squashing of commits when applying a PR to them, etc).

• In the Gitflow workflow, they use hotfix branches too to be able to quickly fix bugs. I would go a lot further than this: we want total freedom on the things we merge to main, because we have a lot of reasons to do so:

  • We want to update README.md independently of anything new in the torchjd package (e.g. we want to add a usage example, fix a typo, explain something better, add a badge, etc)
  • We want to update development guidelines (CONTRIBUTING.md or AGENTS.md)
  • We want to update the CI / config of the project (e.g. .pre-commit-config.yaml)
  • We want to add a new usage example in the latest version of torchjd.org
  • We have a bug to fix (this will lead to a new patch of torchjd)
  • We want to merge dev branch because we're ready to release its changes (this will lead to a new minor or major of torchjd)

  So I think we should allow any merge to main, but just enforce that if it changes the sources, we make a new release of torchjd right after the merge.

Steps

• Add a ruleset to protect any branch whose name matches dev-*, with a similar level of protection as the main branch (or maybe even use the same ruleset for both, as that would be easier to maintain).
• Stop forcing squashing when merging to main: when the PR is from a dev branch, the commit history will be super clean so we want to preserve it.
• Ideally, we would want any merge to main that affects the torchjd package to also trigger a release automatically. Maybe this could be done with the repo you linked a few days ago (that was about auto-releasing). We need to be very careful about that though, and it's not strictly necessary to fix our problems, so for me it should be done after a few months of trying out the new workflow.

[ValerianRey]

Let's do that.

[ValerianRey]

I think we shouldn't enforce that all updates to src merged to main lead to a release. For instance, if we want to rename a variable in an internal function, we obviously don't want to make a release for this, but I still think it should be merged to main (and then the dev branches should be updated too).

So I think that we're really free to merge whatever we want to main, as long as it's something that can be released at any time. If we have any reason to delay a release because of some incomplete set of features, it shouldn't be merged to main (but rather to a protected dev branch).