NOTICE! This is a static HTML version of a legacy ImageJ Trac ticket.

The ImageJ project now uses GitHub Issues for issue tracking.

Please file all new issues there.

Ticket #1731 (closed defect: fixed)

Opened 2013-03-25T15:01:25-05:00

Last modified 2013-04-18T17:42:07-05:00

Make sure Fiji infrastructure is properly backed up

Reported by: dscho Owned by: dscho
Priority: major Milestone: imagej2-b7-ndim-data
Component: Server Admin Version:
Severity: serious Keywords:
Cc: curtis Blocked By:
Blocking: #1729, #1757

Description

Probably as a Jenkins job, we should make sure that fiji.sc things are written properly to the /backup/ directory. The following databases and file system dumps need to be made:

  • Fiji Wiki (including images/, extensions/ and skins/)
  • Bugzilla
  • IRC logs (shinybot)
  • Git mirrors (or remove them?)

Change History

comment:1 Changed 2013-03-25T15:30:01-05:00 by dscho

Bugzilla needs the bugzilla MySQL database, the /var/www/bugzilla/.git repository and the /etc/apache2/conf.d/bugzilla file. The libapache2-mod-perl2 module needs to be enabled, too.

In addition, when re-installing, the lib/ directory needs to be blown away and install-modules.pl --all needs to be run as root.

Last edited 2013-03-25T16:24:35-05:00 by dscho

comment:2 Changed 2013-03-25T15:32:02-05:00 by dscho

Fiji Wiki requires the wikidb3 database, the /var/www/mediawiki/phase3/images/ directory (owned by www-data.www-data) and the /var/www/mediawiki/{,phase3/extensions/,phase3/skins/}.git repositories.

comment:3 Changed 2013-03-27T13:05:39-05:00 by dscho

  • Blocking 1757 added

(In #1757) It seems that in addition to #1733, this required copying fiji-dot.png and openlayers/ to the appropriate directory.

Leaving this open until #1731 is fixed (and includes backing up the stuff for the usage map).

comment:4 Changed 2013-03-28T09:34:11-05:00 by dscho

To be precise, the usage map requires the file fiji-dot.png and the Git checkout openlayers/ in awesome/ to be backed up, too.

While at it, favicon.ico and downloads/ should be backed up for the Fiji Wiki to work, too. Oh, and LocalSettings.php would be rather crucial, too...

comment:5 Changed 2013-03-28T09:36:55-05:00 by dscho

For GitWeb to work, the static symlink and httpd.include are important. The repositories are expected to live in /data/git, and Git is expected to be installed from Git for Windows' source ( https://github.com/msysgit/git); The httpd.include includes all the information required to run the smart HTTP backend and GitWeb.

comment:6 Changed 2013-03-28T11:41:44-05:00 by dscho

  • Milestone changed from imagej2-unscheduled to imagej2-b7-ndim-data

comment:7 Changed 2013-03-28T13:19:52-05:00 by dscho

The code search facility (e.g.  http://fiji.sc/IJ.java) requires cgi-bin/search.cgi to be backed up in addition to httpd.include.

comment:8 Changed 2013-04-01T19:48:56-05:00 by dscho

Fiji's samples reside in the samples/ directory which also needs to be backed up.

comment:9 Changed 2013-04-14T23:51:40-05:00 by dscho

comment:10 Changed 2013-04-18T17:19:19-05:00 by dscho

The uploaded sample images from  http://upload.imagej.net/ need to be backed up, too (since Fiji's Help>Upload Sample Image uses ImageJ2's command to do the job).

comment:11 Changed 2013-04-18T17:34:24-05:00 by dscho

The Backup-fiji.sc job now makes sure that the Wiki database, the BugZilla database, the Geolocation database, the Wiki files, the virtual host configuration, and the GitWeb configuration are written into timestamped files in /backup/fiji/.

It also makes sure that files that are not supposed to change are rsync'ed into /backup/fiji-eden/' (to avoid unnecessary copying). These files are in downloads/, images/, javadoc/, samples/ and update/.

Finally, it clones the .git/ directories into bare mirror repositories in /backup/fiji-eden/.

Once all that is done, the Backup-fiji.sc job cleans up files older than 2 weeks in /backup/fiji.

The Backup-Uploaded-Sample-Images job backs those images up, into /backup/sample-images/. I also copied the uploaded sample images from pacific.

It appears that rsync is really slow, despite using the -u option (only newer files)... :-(

comment:12 Changed 2013-04-18T17:42:07-05:00 by dscho

  • Status changed from new to closed
  • Resolution set to fixed

Luckily, rsync is only slow the first time. The Backup-fiji.sc job took 1h11min for the run that finally put all things into the correct places (actually, the Git repositories were already cloned by the same script but outside Jenkins), but now it only takes 2min45sec.

At the time of writing, the Backup-Uploaded-Sample-Images job still is crunching away on its initial run.