Ticket #1863 (closed defect: fixed)
Opened 2013-05-11T10:42:38-05:00
Last modified 2013-05-11T11:31:57-05:00
Fix Jenkins' increasingly frequent "out of PermSpace" errors
| Reported by: | dscho | Owned by: | dscho |
|---|---|---|---|
| Priority: | critical | Milestone: |
|
| Component: | Server Admin | Version: | |
| Severity: | critical | Keywords: | |
| Cc: | curtis | Blocked By: | |
| Blocking: | #1742 |
Description
It seems that for some days (weeks?) Jenkins freezes at some stage, running out of PermSpace. We need to fix this.
Change History
comment:1 Changed 2013-05-11T10:53:49-05:00 by dscho
comment:2 Changed 2013-05-11T11:14:17-05:00 by dscho
In the process of working on this issue, I wrote a little script to virtually click Build Now on all currently-failed jobs:
#!/usr/bin/jenkins-cli groovy
map = jenkins.model.Jenkins.instance.getItemMap()
map.each() {
name, item ->
if (item instanceof hudson.model.Job) {
if (item.getLastBuild().getResult().isWorseOrEqualTo(hudson.model.Result.FAILURE)) {
item.scheduleBuild(new hudson.model.Cause.RemoteCause("localhost", "Started by rebuild-all-failed.groovy"))
}
}
}
It requires the JENKINS_URL environment variable to be set; adding '-s <URL>' to the shebang line fails for some obscure reason (jenkins-cli claims that it does not know the command).
comment:3 Changed 2013-05-11T11:31:57-05:00 by dscho
- Status changed from new to closed
- Resolution set to fixed
Inspired by https://issues.jenkins-ci.org/secure/attachment/20266/GroovyCommand_patch.txt (attached to https://issues.jenkins-ci.org/browse/JENKINS-8892) I now use Jenkins' own "uber" class loader:
This should help with PermSpace being cluttered.
In case this is not enough, we need to add the Java options -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled to JAVA_ARGS in /etc/default/jenkins.
But let's do that only if we absolutely must, since I have no idea how that interferes (or not) with webapp-style class loading. So I'll close this for now, to be reopened if the problem actually persists.
The reason seems to be the hack in Stable-Fiji to generate pseudo SVn changes; to do so, there is a Groovy Postbuild script to that instantiates URLClassLoaders pretty frequently...