Task #2228
closed
Improve nodepool node launch demand calculation
0%
Description
Because of a needed tweak to nodepool[0] we already running a fork. After chatting with elyezer, there are some further improvements I'd like to include on the fork.
Currently, nodepoold uses a tool called "geard" to determine node launch demand, based on the configure label demands of queued jobs. I believe that when nodepool was first being used, this information was not obtainable via the Jenkins API. Now, however, it is[1]. Furthermore, the geard integration simply doesn't work, afaict, for our use-case, unless we want to go nuts and bring in more pieces of openstack automation (e.g. zuul).
I've already done some preliminary work[2] to try to determine label demand based on the current jenkins build queue, so all that remains is wiring it into the nodepool fork and getting rid of the geard stuff that doesn't meet our needs.
[0]: https://github.com/seandst/nodepool/commit/1a807c8def36d62e80aaaa0e4e961a176e9ce746
[1]: https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.get_queue_info
[2]: https://gist.github.com/seandst/8e9b62f7b46bb5baa88a8bf5fbb11215
Updated by semyers over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to semyers
- Priority changed from Normal to Urgent
For at least two days in a row, probably more, all of the automation has been blocked because of deadlocking in nodepool due to a lack of capacity to spawn nodes. This lack of capacity is largely due to having unused nodes spawned in our CI environment in combination with our nearness to the quotas enforced in that environment. We need to be spawning nodes beyond min-ready entirely based on label demand, allowing us to (finally!) set some label min-ready values to 0 while still seeing nodes with that label get spawned on-demand.
Updated by semyers over 7 years ago
- Status changed from ASSIGNED to POST
Updated by semyers over 7 years ago
- Status changed from POST to CLOSED - CURRENTRELEASE
This change has been merged, and is now running in our automation environment.
.