Task #3407

Upgrade the Celery stack that Pulp ships on EL7 to Celery + Kombu 4.x

Added by dalley over 1 year ago. Updated 2 months ago.

% Done:


Platform Release:
Blocks Release:
Target Release - Packaging:
Backwards Incompatible:
Sprint Candidate:
Pulp 2
QA Contact:
Smash Test:
Verification Required:


Celery 3.x is EOL, and has a couple of known issues which are not resolvable without some very risky backports. Additionally, Pulp is currently supporting both Celery 3.x and 4.x, resulting in double the testing load when we come across an issue relating to Celery.

Fedora has been shipping Celery 4.x for several months, ever since the release of Fedora 26 last year. Thus, we know Pulp is fairly stable with these versions of the dependencies.

I propose that we upgrade the carried versions of Celery, Kombu, and their dependencies to the ones currently being shipped in Fedora 27.

The full list of packages to build is:

  • python-celery 4.0.2-3 (will change pending below changes)
  • python-kombu 4.0.2-8 (will change pending below changes)
  • python-amqp 2.2.2-1
  • python-billiard
  • python-vine 1.1.3-3



#1 Updated by bmbouter over 1 year ago

  • Subject changed from Upgrade the Celery stack that Pulp uses on RHEL/CentOS to Celery + Kombu 4.x to Upgrade the Celery stack that Pulp ships on EL7 to Celery + Kombu 4.x

I'm +1 to this, but I wrote some questions and ideas here:

1. Do you also want the pulp spec file to depend on celery>=4.0 and remove the celery 3.x legacy codepaths? I think we should because with this change we are officially dropping celery/kombu 3 support. If so can it be added in here as the dev part of this ticket?

2. Can @pcreech or @ehelms comment on if they can build these things for EL7 and distribute them with the Pulp2 bits?

+1 to applying the patch for #2927 to the Fedora builds.

For the patch for #2927, and the other packaging updates, can we remove those from this ticket and create a bug for each in Fedora's issue tracker. It's made for packaging issues like this. Filing it there would let the package maintainer hopefully fix it. Here are the celery and kombu packaging tracker links:

Any questions or ideas are welcome. Thanks for bringing this up!

#2 Updated by dalley over 1 year ago

re: Fedora issue tracker, sure, I will do that

#3 Updated by dalley over 1 year ago

FYI, I found a currently-open bz to get these versions of these packages into EPEL

I would assume that gives us two courses of action, either go ahead and do the work ourselves, but let upstream know so that they can piggyback off of it, or the reverse, we help get it sorted out upstream and then pull in those packages to distribute ourselves for non-users of EPEL.

(Either way, I think getting it done would require our involvement, so I'm not sure the distinction matters much, and choice #1 is my preference)

#5 Updated by bmbouter over 1 year ago

@dalley thank you for breaking out the Fedora stuff into those other tickets, that looks resolved. What about removing the 3.0 codepaths altogether? I added two checklist items assuming the answer is yes.

I'm +1 to grooming this after the build team acks it.

@pcreech, @ehelms: Can you upgrade the stack as described here for 2.16? Also would you want to get it into EPEL (see comment 3) or self-distribute via fedorapeople in the long term? In the near tearm I think we have to self-distribute because upgrading EPEL would take a long time at least.

#6 Updated by pcreech over 1 year ago

Given current workload, I think it's more prudent to get this into pulp-packaging than pursuing the epel route ourselves. If at such time it is added in to epel we can stop shipping it. But there have been many times where we needed to update the version or backport a fix ourselves in the past anyways, which also leans me towards the upstream route instead of epel.

There's mentioning of patching the builds here as well... Do we need to makes sure the changes get propogated, or are we going to update to a newer celery version with the fix for the builds we control?

#7 Updated by dalley over 1 year ago

Well, unfortunately there isn't an official celery release yet which includes that fix. The most current release is 4.1.0, which was released prior to our patch being merged. However, they have offered Brian the opportunity to create a 4.1.1 release, so it remains an option.

edit: I asked the Celery developers if they have a timeline for a new release - apparently 4.2 is being released fairly soon, only one blocker left.

My thought was that it's probably better to stick with what is in Fedora for now, since that has been tested the most thoroughly. I'm not sure where using a very new release would fall in terms of risk management for downstream. I'm open to the idea of using a newer release mostly because it looks like there are almost no new features in 4.1 and 4.2 and a lot of bugfixes, some of which at a glance might be relevant.

@Brian, what are your thoughts?

Let's say Celery 4.2 gets released within the next month. I think we would want to minimize downstream risk as much as possible by making sure Pulp 2.16 would be using this code. What is Fedora's policy on updating packages like that within a release? Would their guidelines prohibit upgrading those packages significantly in a released version? If yes, we would have to vendor those packages ourselves for Fedora on top of EL7. I'm not sure how we feel about that.

I will sit down at some point and take a look through the bugfixes in the changelog to try to figure out how worthwhile such an upgrade would be

#8 Updated by pcreech over 1 year ago

I'm on board with taking the spec in the latest fedora, and building that and applying patches. Just wanted to make sure I knew what would be needed by releng

#9 Updated by bmbouter over 1 year ago

+1 to self-distributing EL7 and not pursuing the EPEL route. ty @pcreech for the comment.

For the risk reasons @dalley mentions, I'm +1 on using the existing versions in Fedora + the patch from

@dalley, Fedora can put any version in Rawhide prior to the branch point. I suspect we should just let Rawhide upgrade naturally after upstream Kombu 4.2 is released. For risk reasons, EL7 should lag relatively far behind so I don't think we should upgrade from celery/kombu 4.z+patches to celery/kombu 4.2 for a few months after 4.2 is out at least.

#10 Updated by dalley over 1 year ago

All of that sounds good to me.

#11 Updated by pcreech over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to pcreech

#12 Updated by pcreech over 1 year ago

  • Project changed from Pulp to Packaging
  • Status changed from ASSIGNED to POST

#13 Updated by pcreech over 1 year ago

  • Status changed from POST to MODIFIED

#14 Updated by ttereshc over 1 year ago

  • Platform Release set to 2.16.0

#15 Updated by bmbouter over 1 year ago

  • Status changed from MODIFIED to ON_QA

#16 Updated by bmbouter about 1 year ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#17 Updated by bmbouter 2 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF