Ticket #832 (closed enhancement: fixed)

I think tickets #487 and #488 will be largely solved by such integration.

Work has begun in fiji.git on that; it's all in the 'ij2-launcher' branch:  http://fiji.sc/cgi-bin/gitweb.cgi?p=fiji.git;a=shortlog;h=refs/heads/ij2-launcher

Very close now: the Fiji launcher was renamed to ImageJ and is able to run ImageJ2 already. I'm not quite clear yet how to integrate the thing into the imagej source tree, being native and all, but it is basically working and ready for a beta.

For an example of native C code structured for compilation by Maven and/or CMake, see  the slim-curve project. This project uses the maven-nar-plugin to bundle the resultant artifact as something consumable by other Maven Java projects. The CMakeLists.txt provides an alternative cross-platform compilation mechanism as well.

Would something like this work for the ImageJ launcher code?

The last obstacle to adding this to ImageJ2's repository is that I need to figure out how the Maven native plugin works. For starters, I will probably punt and use the Maven executer plugin instead (calling make, and uploading artifacts manually for all supported platforms, using Maven profiles to prevent compilation on non-matching platforms). When this happens, I will add another ticket to make sure that the native plugin is used eventually, and will figure out how we can still build things using the cross-compile scripts provided by Fiji.

Progress: the 'ij-launcher' branch has an initial version of the ImageJ launcher.

Notes:

• it uses the NAR plugin ( http://duns.github.com/maven-nar-plugin/) to compile
• maven-nar-plugin was preferred over maven-native-plugin because the latter is apparently not able to compile executables, and more importantly, it seems to not have been developed since an alpha state in 2006
• it was only compiled on Linux x86_64 so far (which the NAR plugin calls amd64)
• for now, JNI_ENOMEM was hard-coded; this is wrong, I have no idea yet why including jni.h is not enough on my system to have that symbol defined
• the future plan is to add support for the other platforms' flags by adding properties for each option and using Maven profiles (yay for profiles! Profiles for president!)
• the NAR plugin is nice enough to bundle the platform-dependent components (in this case, the executable) in individual .nar files (which are just .jar files, but do not contain .class files). That will allow us to deploy the artifacts from several different machines, using trusty Jenna Jenkins
• in theory, it would be possible to put the ij-launcher.jar sources into the same project, I think, but I'd like to keep the C sources and the Java sources separate for cleanness sake
• the launcher is not yet included into the packaging of the imagej2-*.zip
• the executable should probably be named ImageJ instead of 'launcher-program' (I still need to discuss with Curtis whether it is okay to name the project/subdirectory 'ImageJ' instead of 'launcher-program')
• Since I override the default compiler options, the many compile warnings regarding casts between signed and unsigned variables are hidden for now
• I changed ImageJ.c originally in fiji.git, changes are in fiji.git's 'ij-launcher' branch for now but will be merged to 'master' soon (basically, the requirement to define JAVA_HOME and JAVA_LIBRARY_PATH was dropped)
• I haven't made up my mind yet whether the launcher should be changed so it can run from target/nar/

As always with my work in topic branches, this is very much work in progress and all of the commit might change before being merged into the mainline branch.

Much progress was made, in particular with non-Linux and non-64-bit builds.

In particular, ImageJ.c was adjusted so it can compile without platform-specific #defines. It also learnt how to behave correctly on MacOSX when jars/ is not inside the .app/ directory but alongside.

Then came the hardest part: setting up Jenkins to compile all the platforms.

1) Windows

• A virtual machine was set up with Windows 7 64-bit to run in VirtualBox.
• msysGit (the Git for Windows development environment) was installed via the net installer from  http://msysgit.github.com/ (*not* Git for Windows which is a barebones Git and lacks in particular the gcc compiler which we need)
• Fiji was cloned to /c/fiji/
• Fiji was built to get a JDK for Windows 64-bit
• In /c/fiji/, java/win32/ was cloned to get a JDK for Windows 32-bit
• Apache Maven was installed into /c/apache-maven-3.0.4/
• Two nodes were added to Jenkins (Jenkins>Manage Jenkins>Manage Nodes>New Node), Win32 and Win64, both with one processor and only taking jobs bound to that node
• The slave.jar was downloaded into the Windows machine's /c/fiji/ directory using

  curl -O http://jenkins.imagej.net/jnlpJars/slave.jar
  

• The following shell script was installed into /c/fiji/start-jenkins-slave.sh

  #!/bin/sh
  
  MODE=64
  test "win32" = "$1" && MODE=32
  
  if test 64 = "$MODE"
  then
          SYSROOT=/src/mingw-w64/sysroot/bin
          test -x $SYSROOT/gcc.exe ||
          cp $SYSROOT/x86_64-w64-mingw32-gcc.exe $SYSROOT/gcc.exe
          export PATH=$SYSROOT:$PATH
  fi
  
  export FIJI_HOME=/c/fiji
  export JAVA_HOME=$FIJI_HOME/java/win$MODE/jdk1.6.0_24
  export MAVEN_HOME=/c/apache-maven-3.0.4
  export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH
  JENKINS_URL=http://jenkins.imagej.net/computer/Win$MODE/slave-agent.jnlp
  
  java -Xdebug \
          -jar "${0%/*}"/slave.jar \
          -cp $FIJI/jars/jna.jar \
          -jnlpUrl $JENKINS_URL
  

• To auto-start the nodes upon login, two Shortcuts were created in Start>All Programs>Startup (right mouse click lets you open an Explorer there), with the Target

  c:\msysgit\bin\sh.exe -c "/c/fiji/start-jenkins-slave.sh win32"
  

  and

  c:\msysgit\bin\sh.exe -c "/c/fiji/start-jenkins-slave.sh win64"
  

  respectively
• Auto-Login was enabled via Windows+R, launching "control userpasswords2" and turning off the check box requiring passwords
• Still missing:

  VBoxHeadless -s "Win 7 64-bit" -n -m 5998
  

  should be started from upstart on the host machine whenever the machine is (re-)booted.

2) MacOSX

• A virtual machine was set up to run MacOSX in VirtualBox
• Maven was upgraded manually to 3.0.4 by unpacking the .zip into /usr/share/java/ and relinking /usr/share/maven to it
• The user account jenkins was created
• The directory /var/jenkins was created and jenkins made owner of it
• The file slave.jar was downloaded as for Windows and put into /var/jenkins/
• JNA was copied over from a Fiji installation's jars/jna.jar into /var/jenkins/ (this might not be strictly necessary, but makes for nicer logs in Finder's Utilities>Console.app)
• The following text was written to /Library/LaunchDaemons/jenkins.plist:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
          http://www.apple.com/DTDs/PropertyList-1.0.dtd >
  <plist version="1.0">
          <dict>
                  <key>Label</key>
                  <string>org.jenkins</string>
                  <key>UserName</key>
                  <string>jenkins</string>
                  <key>WorkingDirectory</key>
                  <string>/var/jenkins</string>
                  <key>ProgramArguments</key>
                  <array>
                          <string>java</string>
                          <string>-jar</string>
                          <string>slave.jar</string>
                          <string>-cp</string>
                          <string>jna.jar</string>
                          <string>-jnlpUrl</string>
                          <string>http://jenkins.imagej.net/computer/MacOSX/slave-agent.jnlp</string>
                  </array>
                  <key>KeepAlive</key>
                  <true/>
                  <key>RunAtLoad</key>
                  <true/>
          </dict>
  </plist>
  

• The permissions of the plist were set to executable via

  sudo chmod 0755 /Library/LaunchDaemons/jenkins.plist
  

• Still missing:

  VBoxHeadless -s "MacOSX" -n -m 5997
  

  should be started from upstart on the host machine whenever the machine is (re-)booted.

After the successful setup of the Jenkins nodes:

• A Jenkins new job was created, and a Slaves axis was added to the Configuration Matrix checking all the individual nodes.
• A Maven build step was added to the Jenkins job, specifying core/launcher/pom.xml as POM
• A shell script build step was added to the Jenkins job to compile 32-bit Linux and pre-Leopard MacOSX launchers:

  # also build the 32-bit stuff on MacOSX and Linux
  
  UNAME="$(uname -s)"
  case "$UNAME" in
  Darwin)
      (cd core/launcher &&
       mvn -Dos.arch=i386 -P "MacOSX Tiger" -P !"MacOSX Leopard")
      ;;
  Linux)
      (cd core/launcher &&
       export JAVA_HOME=/home/dscho/fiji/java/linux/jdk1.6.0_24 &&
       export PATH=$JAVA32_HOME/bin:$PATH &&
       mvn -P "i386-Linux" -P !"amd64-Linux")
      ;;
  esac
  

Next steps:

• make a hacky Jenkins job that deploys the different launchers in one go so that they get the same snapshot version
• make sure that the Jenkins nodes are always started
• possibly put those Jenkins nodes onto a MacMini rather than this developer's work machine
• in the not-so-distant future, patch maven-nar-plugin to be able to allow for a multi-step deployment of the same snapshot version

Actually, one rather crucial thing I forgot: On MacOSX,  git://github.com/scijava/cpptasks-parallel must be cloned and installed as jenkins user, otherwise the -arch option is not recognized correctly as a compiler option by the nar plugin.

And a not so crucial thing: there is an off-by-one in the nar plugin itself (when writing the nar.properties), so one might want to clone and install  git://github.com/scijava/maven-nar-plugin, too. It is not that critical, because the nar plugin never interprets the nar.properties file.

Next steps: deploy our own version of cpptasks-parallel and fix our version of the maven-nar-plugin in our Maven repository.

Oh, another thing I forgot: when launching the Windows slaves that way, the msysGit bin/ directories need to be added to the PATH explicitly. In other words, at the beginning, there should be these two lines:

export MSYSGIT_HOME=/c/msysgit
export PATH=$MSYSGIT_HOME/bin:$MSYSGIT_HOME/mingw/bin:$PATH

In total, the script start-jenkins-slave.sh now reads therefore:

#!/bin/sh

export MSYSGIT_HOME=/c/msysgit
export PATH=$MSYSGIT_HOME/bin:$MSYSGIT_HOME/mingw/bin:$PATH

MODE=64
test "win32" = "$1" && MODE=32

if test 64 = "$MODE"
then
        SYSROOT=/src/mingw-w64/sysroot/bin
        test -x $SYSROOT/gcc.exe ||
        cp $SYSROOT/x86_64-w64-mingw32-gcc.exe $SYSROOT/gcc.exe
        export PATH=$SYSROOT:$PATH
fi

export FIJI_HOME=/c/fiji
export JAVA_HOME=$FIJI_HOME/java/win$MODE/jdk1.6.0_24
export MAVEN_HOME=/c/apache-maven-3.0.4
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH
JENKINS_URL=http://jenkins.imagej.net/computer/Win$MODE/slave-agent.jnlp

java -Xdebug \
        -jar "${0%/*}"/slave.jar \
        -cp $FIJI/jars/jna.jar \
        -jnlpUrl $JENKINS_URL

And then there are some good news!

The Jenkins job to collect all the ImageJ Launcher artifacts from the slaves is basically working (the ImageJ-Launcher job is configured to publish all files in core/launcher/target/ and the Deploy-ImageJ-Launcher job re-uses ImageJ-Launcher's workspace to wrap everything up and deploys all the platform-specific files nicely).

As even my concoction to assemble the files in the correct places for the application.zip using the maven-dependency-plugin seems to work, I dcommitted everything and deleted the topic branch.

Case closed (for the moment).

One piece of information that would have saved me a lot of grief, and which I therefore reproduce here to avoid IJ2 developers from having to repeat my experience:

When binding jobs (which are sometimes also called "projects" in Jenkins speak) to nodes (which are sometimes also called "slaves" in Jenkins speak) by adding a Slaves axis in the configuration matrix of a job, the console output of the slave can never be seen in the Console Output of the job itself. Instead, one has to click on the date in the build history, then click on the links under the Configuration section on the right-hand side (the titles correspond to the node names *except* if the job is bound to a single node in which case the link title is confusingly changed to default), and then again on the date in the build history.

It does make sense once one realises that the set of bound nodes can vary over the build history, and once one keeps in mind that the build information is only copied into the subdirectory configurations/ of the job workspace when there are multiple bound nodes.

Before one realises those two things, it does not make sense at all.

Another setting I had to make in the Windows VM is to choose "Never" in Control Panel's "Choose how to check for solutions". Otherwise a crash in, say, gcc.exe will just open an interactive dialog and hold any Jenkins job.

It would have been too easy if the setup worked, would it not?

Apparently Windows (or at least my small VM) has problems to run two nodes in parallel. It even blocked upon a gcc.exe crash in spite of choosing "Never" for "how to check for solutions".

So I had to make the Windows node build 64-bit and 32-bit consecutively. As a consequence, there is only one Shortcut left in Start Menu>Startup which calls start-jenkins-slave.sh that now reads:

#!/bin/sh

export MSYSGIT_HOME=/c/msysgit
export PATH=$MSYSGIT_HOME/bin:$MSYSGIT_HOME/mingw/bin:$PATH

SYSROOT=/src/mingw-w64/sysroot/bin
test -x $SYSROOT/gcc.exe ||
cp $SYSROOT/x86_64-w64-mingw32-gcc.exe $SYSROOT/gcc.exe
export PATH=$SYSROOT:$PATH

export FIJI_HOME=/c/fiji
export JAVA_HOME=$FIJI_HOME/java/win64/jdk1.6.0_24
export MAVEN_HOME=/c/apache-maven-3.0.4
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

java -Xdebug \
	-jar "${0%/*}"/slave.jar \
	-cp $FIJI/jars/jna.jar \
	-jnlpUrl http://jenkins.imagej.net/computer/Windows/slave-agent.jnlp

And the configuration in Jenkins changed: there is now only one node called Windows (instead of a Win64 and a Win32 one), and the shell script build step (executed after the Maven build step building the 64-bit Launchers) reads like this:

# also build the 32-bit stuff on MacOSX and Linux

UNAME="$(uname -s)"
case "$UNAME" in
Darwin)
    (cd core/launcher &&
     mvn -Dos.arch=i386 -P "MacOSX Tiger" -P !"MacOSX Leopard")
    ;;
Linux)
    (cd core/launcher &&
     export JAVA_HOME=/home/dscho/fiji/java/linux/jdk1.6.0_24 &&
     export PATH=$JAVA_HOME/bin:$PATH &&
     mvn -P "i386-Linux" -P !"amd64-Linux")
    ;;
MINGW*)
    (cd core/launcher &&
     export JAVA_HOME=$(cd /c/fiji/java/win32/jdk1.6.0_24 && pwd -W) &&
     export PATH=/c/msysgit/mingw/bin:/c/msysgit/bin:$JAVA_HOME/bin:$PATH &&
     mvn -Dos.arch=x86 -X -e)
esac

I finally added the startup configuration to the machine running the VMs: it is an Ubuntu machine, so it provides upstart (a SYSV-compatible init replacement). The configurations live in /etc/init/jenkins-macosx.conf and /etc/init/jenkins-windows.conf, respectively:

# The MacOSX VM that connects to jenkins.imagej.net as MacOSX node

description "Jenkins MacOSX node"

respawn
start on filesystem
stop on runlevel [!2345]

exec sudo -u gene099 VBoxHeadless -n -m 5977 -o no-password -s MacOSX

The Windows node configuration is a copy-edited version thereof.

After an OS upgrade, JAVA_HOME was no longer properly set on the master (and setting the environment variable globally would override the nodes' settings erroneously).

So now we build the 64-bit launchers also via a shell script that calls mvn explicitly (and that sets JAVA_HOME if $(uname -s) is equal to Linux).

Another thing: the MacOSX guest system uses outrageously many CPU cycles when run inside VirtualBox. The secret is to remove a kernel extension from the MacOSX guest: simply

rm -rf /System/Library/Extensions/AppleIntelCPUPowerManagement.kext

brings down the CPU load from 100% (on the one core on which I allowed VirtualBox to run the MacOSX guest) to ~8%.

This tip came from  https://forums.virtualbox.org/viewtopic.php?f=22&t=31176#p139617 which also warns that even a minor MacOSX update (i.e. not only from 10.6 to 10.7 but already from 10.6.8 to 10.6.9) might bring back the bad behavior.

Yet another thing: when rebooting the machine, upstart tried to run the virtual machines too early. The trick is to change the start on line in /etc/init/jenkins-{macosx,windows}.conf to

start on net-device-up

(I think, will test on next reboot ;-)

You did not think that we were done with this ticket yet, did you?

No, we need the following in /etc/init/jenkins-macosx.conf:

pre-stop script
        sudo -u gene099 VBoxManage controlvm "MacOSX" poweroff
        sleep 35
end script

and the following in /etc/init/jenkins-windows.conf:

pre-stop script
        sudo -u gene099 VBoxManage controlvm "Windows 7 64-bit" acpipowerbutton
        sleep 35
end script

For good measure, I also put in a

respawn limit 5 10
start on (local-filesystems and net-device-up IFACE=eth0)

So I moved the Jenkins Node VM from my machine to the new Mac Mini. It is started automatically upon reboot and allows access via RDP (xfreerdp is what I recommend on Linux), password are the five smallest natural numbers, in ascending order. Since it is on cyber.microscopy.wisc.edu (i.e. behind the firewall), only LOCI can access it.

The file responsible for making it be started automatically is in /Library/LaunchDaemons/jenkins-vm.plist and looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC -//Apple Computer//DTD PLIST 1.0//EN
	http://www.apple.com/DTDs/PropertyList-1.0.dtd >
<plist version="1.0">
	<dict>
		<key>Label</key>
		<string>org.jenkins.vm</string>
		<key>UserName</key>
		<string>LOCI</string>
		<key>WorkingDirectory</key>
		<string>/Users/LOCI</string>
		<key>ProgramArguments</key>
		<array>
			<string>VBoxHeadless</string>
			<string>-s</string>
			<string>MacOSX Jenkins Node</string>
		</array>
		<key>KeepAlive</key>
		<true/>
		<key>RunAtLoad</key>
		<true/>
	</dict>
</plist>