Ticket #1783 (closed feature: fixed)
Opened 2013-04-08T12:40:57-05:00
Last modified 2014-05-01T16:19:56-05:00
Make version binding tighter for reproducible builds
| Reported by: | dscho | Owned by: | dscho |
|---|---|---|---|
| Priority: | major | Milestone: | imagej-2.5.0 |
| Component: | Build System | Version: | |
| Severity: | major | Keywords: | |
| Cc: | curtis | Blocked By: | #1784, #1785, #1786, #1787, #1788 |
| Blocking: |
Description
ImageJ2 is not a stand-alone project; it depends on ImgLib2, scijava-common, pom-scijava and SCIFIO. Fiji depends on all of them.
Builds, however, are not reproducible (as can be seen by failing git bisect and Jenkins' Fiji build failures). The reason is that there are -SNAPSHOT version couplings.
Goal: get rid of all -SNAPSHOT dependencies in all of above-mentioned projects.
To maintain ease of development, this implies introducing some tools:
- just like http://jenkins.imagej.net/job/Synchronize-and-deploy-IJ1/ synchronizes ImageJ 1.x fully automatically, commits, pushes and deploys, we need Jenkins jobs (most likely manually triggered, however) to help bumping up the version in one of our projects.
- the Jenkins job should also trigger dependent projects to update, e.g. a scijava-common bump should bump pom-scijava which in turn should bump the rest of 'em.
- this Jenkins job should avoid pushing updated 'master' branches when builds fail, but simply mark the builds as failed.
- we need a Jenkins job to warn about API changes in -SNAPSHOT versions. Most likely, this job should make use of the Animal Sniffer Maven plugin: http://mojo.codehaus.org/animal-sniffer-maven-plugin/.
- the Animal Sniffer job could automatically push branches as appropriate for micro, minor and major version increases to a special namespace (e.g. refs/suggested/) to ease the maintenance burden.
- it would probably be a good idea to extend http://jenkins.imagej.net/job/ImageJ-style/ to check also for SNAPSHOT dependencies pushed to master by mistake.
- http://jenkins.imagej.net/job/ImageJ-style/ could be extended to fix easy-to-fix problems and commit & push them to master.
The idea is really to help the development process without adding a substantial burden on the developers. Tools should be helpful, not create more work.
Change History
comment:1 Changed 2013-04-08T12:42:12-05:00 by dscho
comment:2 Changed 2013-04-08T12:58:52-05:00 by curtis
We will also write up a public rationale for how and why we do versioning this way for SciJava projects (SciJava Common, SCIFIO, ImgLib, ImageJ, Fiji, LOCI projects, etc.):
- Conformity to SemVer versioning system.
- Fixed release version couplings (i.e., no SNAPSHOTs, no version ranges) for build reproducibility, and hence bisectability, along with many other benefits.
- There is nothing wrong with SNAPSHOT or version range couplings, as long as you understand the tradeoffs. Any downstream projects can continue to use these with SciJava projects, as long as they understand that their builds will not be reproducible.
- It is not our intent to enforce any particular workflow on people developing on master. If people want to break API left and right during development on master, that is their prerogative. However, it does incur a slight (temporary) technical debt when making the next release, since either: A) any breaking changes would need to be "undone" (i.e., readding any previous method signatures with deprecation tags); or B) the next release would need a major version bump indicating lack of backwards compatibility. Obviously (A) is preferred, and the Animal Sniffer Maven plugin and related tooling (discussed above) will be very helpful in assisting in pursuit of (A).
- Of course, it is strongly preferred that for each commit to master (including all commits as part of a merge commit), code still compile, hence the CI jobs for testing this. For a variety of reasons, broken builds on master should be fixed as soon as possible.
- The only new requirement process-wise is that code on master should not introduce any SNAPSHOT dependencies, because those commits will then not have a reproducible build. Any SNAPSHOT dependencies on master should be reverted to fixed release dependencies as soon as possible, just like failing builds. We may create a Jenkins job to check this, which fails in the case of invalid dependency versions. (To put another way: we want it to be possible to cut a release from master as easily as possible. This means code with only fixed release dependencies, which compiles and has passing unit tests.)
Lastly, the versions-maven-plugin is a great tool for verifying that you are using the latest release version of both Maven plugins and project dependencies. We can make use of it when performing prerelease checks.
comment:3 Changed 2013-04-08T13:16:23-05:00 by curtis
One other thing that dscho & I discussed: we ideally want to use the Maven Release plugin to perform our releases. However, the default behavior of mvn release:prepare cuts the release on master and then pushes it. We would prefer to cut the release to the (orphaned) tag itself, and only commit the SNAPSHOT version bump on master. I.e., no commit on master should have a release version in the POM. This is how we've been doing it with our tag-release.sh script. The mvn release:branch goal might be a solution there; further investigation should be a subticket of this feature.
comment:9 Changed 2014-05-01T16:19:56-05:00 by curtis
- Status changed from new to closed
- Resolution set to fixed
We did it! SciJava, SCIFIO, ImgLib2 and ImageJ now use only release couplings between components.
There is still some work to be done splitting ImgLib2 to separate repositories, as well as getting Fiji fully transitioned away from snapshot couplings, but that will come with time. No need to keep this ticket around reminding us.
Ah, and of course: the Jenkins jobs should be written as .sh scripts in scijava-common's bin/ directory for transparency.