Change Details

For each operations/puppet changeset, a short Jenkins job runs that performs pep8/pyflake/puppet-lint etc. For most changesets, you would expect such a job to run in a few seconds, less than half a minute in the vast majority of the cases. However, it's often the case that such job takes minutes, sometimes many minutes (e.g 5mins for a couple of changesets of mine just now) to run, slowing us down. Since operations/puppet is ff-only, it's often the case that someone needs to just rebase and then wait in order to merge a changeset (or worse: more than one!). Moreover, since changes have to be followed by a manual puppet-merge on the puppetmaster (and sometimes babysat on), this means that we can't have Jenkins automatically merge changesets with C+2/V+2. All of the above means that for a lot of us, we have to daily go through a process for that is essentially us hitting "rebase", then sitting on our screens idle while waiting for Jenkins to finish for some minutes, then hit submit, then puppet-merge, then do that all over again for the next changeset, and the next and so on and so forth. This is a waste of time, energy and focus and incredibly frustrating. It was unacceptable 4 years ago when I first raised it (searched my IRC logs: Feb 19th and Mar 12th, 2013 was when I first pointed this out) and it definitely is now, with a far larger puppet tree and contributor base. operations/puppet is one of the most active repositories Wikimedia has, and one of the most critical ones as far as errors creeping in to production go, so that manually forcing V+2 is dangerous enough. It doesn't look like the current CI plans are going to address this particular CI pipeline -- at least not anytime soon. We expect better than this. I realize this is the result of some poor architectural choices made in the past (some of them against our strong recommendations at the time) and can't be fixed anytime soon without a major rearchitecturing effort. I'd still be interested in an shorter-term fix though, like for example bypassing NodePool entirely and having 1-2 VMs dedicated to puppet-linting (or any other short-term fix you folks can think of).

For each `operations/puppet` changeset, a short Jenkins job runs that performs pep8/pyflake/puppet-lint etc. For most changesets, you would expect such a job to run in a few seconds, less than half a minute in the vast majority of the cases. However, it's often the case that such job takes minutes, sometimes many minutes (e.g 5mins for a couple of changesets of mine just now) to run, slowing us down. Since `operations/puppet` is configured to be {nav Fast forward only}, it's often the case that someone needs to just rebase and then wait in order to merge a changeset (or worse: more than one!). Moreover, since changes have to be followed by a manual `puppet-merge` on the puppetmaster (and sometimes babysat on), this means that we can't have Jenkins automatically merge changesets with C+2/V+2. All of the above means that for a lot of us, we have to daily go through a process for that is essentially us hitting {nav Rebase}, then sitting on our screens idle while waiting for Jenkins to finish for some minutes, then hit submit, then puppet-merge, then do that all over again for the next changeset, and the next and so on and so forth. This is a waste of time, energy and focus and incredibly frustrating. It was unacceptable 4 years ago when I first raised it (searched my IRC logs: Feb 19th and Mar 12th, 2013 was when I first pointed this out) and it definitely is now, with a far larger puppet tree and contributor base. `operations/puppet` is one of the most active repositories Wikimedia has, and one of the most critical ones as far as errors creeping in to production go, so that manually forcing V+2 is dangerous enough. It doesn't look like the current CI plans are going to address this particular CI pipeline -- at least not anytime soon. We expect better than this. I realize this is the result of some poor architectural choices made in the past (some of them against our strong recommendations at the time) and can't be fixed anytime soon without a major rearchitecturing effort. I'd still be interested in an shorter-term fix though, like for example bypassing #NodePool entirely and having 1-2 VMs dedicated to puppet-linting (or any other short-term fix you folks can think of).

For each `operations/puppet` changeset, a short Jenkins job runs that performs pep8/pyflake/puppet-lint etc. For most changesets, you would expect such a job to run in a few seconds, less than half a minute in the vast majority of the cases. However, it's often the case that such job takes minutes, sometimes many minutes (e.g 5mins for a couple of changesets of mine just now) to run, slowing us down. Since `operations/puppet` is ff-configured to be {nav Fast forward only}, it's often the case that someone needs to just rebase and then wait in order to merge a changeset (or worse: more than one!). Moreover, since changes have to be followed by a manual `puppet-merge` on the puppetmaster (and sometimes babysat on), this means that we can't have Jenkins automatically merge changesets with C+2/V+2. All of the above means that for a lot of us, we have to daily go through a process for that is essentially us hitting "r{nav Rebase"}, then sitting on our screens idle while waiting for Jenkins to finish for some minutes, then hit submit, then puppet-merge, then do that all over again for the next changeset, and the next and so on and so forth. This is a waste of time, energy and focus and incredibly frustrating. It was unacceptable 4 years ago when I first raised it (searched my IRC logs: Feb 19th and Mar 12th, 2013 was when I first pointed this out) and it definitely is now, with a far larger puppet tree and contributor base. `operations/puppet` is one of the most active repositories Wikimedia has, and one of the most critical ones as far as errors creeping in to production go, so that manually forcing V+2 is dangerous enough. It doesn't look like the current CI plans are going to address this particular CI pipeline -- at least not anytime soon. We expect better than this. I realize this is the result of some poor architectural choices made in the past (some of them against our strong recommendations at the time) and can't be fixed anytime soon without a major rearchitecturing effort. I'd still be interested in an shorter-term fix though, like for example bypassing #NodePool entirely and having 1-2 VMs dedicated to puppet-linting (or any other short-term fix you folks can think of).