Ticket #1731 (closed defect: fixed)
Opened 2013-03-25T15:01:25-05:00
Last modified 2013-04-18T17:42:07-05:00
Make sure Fiji infrastructure is properly backed up
| Reported by: | dscho | Owned by: | dscho |
|---|---|---|---|
| Priority: | major | Milestone: |
|
| Component: | Server Admin | Version: | |
| Severity: | serious | Keywords: | |
| Cc: | curtis | Blocked By: | |
| Blocking: | #1729, #1757 |
Description
Probably as a Jenkins job, we should make sure that fiji.sc things are written properly to the /backup/ directory. The following databases and file system dumps need to be made:
- Fiji Wiki (including images/, extensions/ and skins/)
- Bugzilla
- IRC logs (shinybot)
- Git mirrors (or remove them?)
Change History
comment:1 Changed 2013-03-25T15:30:01-05:00 by dscho
comment:2 Changed 2013-03-25T15:32:02-05:00 by dscho
Fiji Wiki requires the wikidb3 database, the /var/www/mediawiki/phase3/images/ directory (owned by www-data.www-data) and the /var/www/mediawiki/{,phase3/extensions/,phase3/skins/}.git repositories.
comment:3 Changed 2013-03-27T13:05:39-05:00 by dscho
- Blocking 1757 added
comment:4 Changed 2013-03-28T09:34:11-05:00 by dscho
To be precise, the usage map requires the file fiji-dot.png and the Git checkout openlayers/ in awesome/ to be backed up, too.
While at it, favicon.ico and downloads/ should be backed up for the Fiji Wiki to work, too. Oh, and LocalSettings.php would be rather crucial, too...
comment:5 Changed 2013-03-28T09:36:55-05:00 by dscho
For GitWeb to work, the static symlink and httpd.include are important. The repositories are expected to live in /data/git, and Git is expected to be installed from Git for Windows' source ( https://github.com/msysgit/git); The httpd.include includes all the information required to run the smart HTTP backend and GitWeb.
comment:6 Changed 2013-03-28T11:41:44-05:00 by dscho
- Milestone changed from imagej2-unscheduled to imagej2-b7-ndim-data
comment:7 Changed 2013-03-28T13:19:52-05:00 by dscho
The code search facility (e.g. http://fiji.sc/IJ.java) requires cgi-bin/search.cgi to be backed up in addition to httpd.include.
comment:8 Changed 2013-04-01T19:48:56-05:00 by dscho
Fiji's samples reside in the samples/ directory which also needs to be backed up.
comment:9 Changed 2013-04-14T23:51:40-05:00 by dscho
Started work in http://jenkins.imagej.net/job/Backup-fiji.sc/
comment:10 Changed 2013-04-18T17:19:19-05:00 by dscho
The uploaded sample images from http://upload.imagej.net/ need to be backed up, too (since Fiji's Help>Upload Sample Image uses ImageJ2's command to do the job).
comment:11 Changed 2013-04-18T17:34:24-05:00 by dscho
The Backup-fiji.sc job now makes sure that the Wiki database, the BugZilla database, the Geolocation database, the Wiki files, the virtual host configuration, and the GitWeb configuration are written into timestamped files in /backup/fiji/.
It also makes sure that files that are not supposed to change are rsync'ed into /backup/fiji-eden/' (to avoid unnecessary copying). These files are in downloads/, images/, javadoc/, samples/ and update/.
Finally, it clones the .git/ directories into bare mirror repositories in /backup/fiji-eden/.
Once all that is done, the Backup-fiji.sc job cleans up files older than 2 weeks in /backup/fiji.
The Backup-Uploaded-Sample-Images job backs those images up, into /backup/sample-images/. I also copied the uploaded sample images from pacific.
It appears that rsync is really slow, despite using the -u option (only newer files)... :-(
comment:12 Changed 2013-04-18T17:42:07-05:00 by dscho
- Status changed from new to closed
- Resolution set to fixed
Luckily, rsync is only slow the first time. The Backup-fiji.sc job took 1h11min for the run that finally put all things into the correct places (actually, the Git repositories were already cloned by the same script but outside Jenkins), but now it only takes 2min45sec.
At the time of writing, the Backup-Uploaded-Sample-Images job still is crunching away on its initial run.
Bugzilla needs the bugzilla MySQL database, the /var/www/bugzilla/.git repository and the /etc/apache2/conf.d/bugzilla file. The libapache2-mod-perl2 module needs to be enabled, too.
In addition, when re-installing, the lib/ directory needs to be blown away and install-modules.pl --all needs to be run as root.