Project

Profile

Help

Task #2228

Improve nodepool node launch demand calculation

Added by semyers about 4 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Urgent
Assignee:
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Target Release - Packaging:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Because of a needed tweak to nodepool[0] we already running a fork. After chatting with elyezer, there are some further improvements I'd like to include on the fork.

Currently, nodepoold uses a tool called "geard" to determine node launch demand, based on the configure label demands of queued jobs. I believe that when nodepool was first being used, this information was not obtainable via the Jenkins API. Now, however, it is[1]. Furthermore, the geard integration simply doesn't work, afaict, for our use-case, unless we want to go nuts and bring in more pieces of openstack automation (e.g. zuul).

I've already done some preliminary work[2] to try to determine label demand based on the current jenkins build queue, so all that remains is wiring it into the nodepool fork and getting rid of the geard stuff that doesn't meet our needs.

[0]: https://github.com/seandst/nodepool/commit/1a807c8def36d62e80aaaa0e4e961a176e9ce746
[1]: https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.get_queue_info
[2]: https://gist.github.com/seandst/8e9b62f7b46bb5baa88a8bf5fbb11215

History

#1 Updated by semyers about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to semyers
  • Priority changed from Normal to Urgent

For at least two days in a row, probably more, all of the automation has been blocked because of deadlocking in nodepool due to a lack of capacity to spawn nodes. This lack of capacity is largely due to having unused nodes spawned in our CI environment in combination with our nearness to the quotas enforced in that environment. We need to be spawning nodes beyond min-ready entirely based on label demand, allowing us to (finally!) set some label min-ready values to 0 while still seeing nodes with that label get spawned on-demand.

#2 Updated by semyers about 4 years ago

  • Status changed from ASSIGNED to POST

#3 Updated by semyers about 4 years ago

  • Status changed from POST to CLOSED - CURRENTRELEASE

This change has been merged, and is now running in our automation environment.

#4 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF