RPM Support

It might be worth thinking about whether we can make a patch that will apply cleanly against 2.4 since there are users who are having problems with DB cursor timeouts. Patching against 2.6 might also be fine if we are comfortable requiring users to upgrade to a newer Pulp to fix this.

On 06/04/2015 11:00 AM, Pulp wrote:

It might be worth thinking about whether we can make a patch that will
apply cleanly against 2.4 since there are users who are having problems
with DB cursor timeouts. Patching against 2.6 might also be fine if we
are comfortable requiring users to upgrade to a newer Pulp to fix this.

On second thought, this might have to be done with "spawned tasks" which
would change the API to the task. One way to work around this not being
backwards-incompatible would be to add an optional boolean to the API
call that lets the user state whether they want to do the calculation in
parallel or not, and if the bool isn't provided we default to the
current behavior. Then, with Pulp 3.0 we can just change to always doing
it in parallel and drop the boolean.

--
Randy Barlow

Here is a possible implementation:

Define TaskMonitorTask as a regular celery task that takes two parameters: 'parent_task_id' and 'tasks'. 'tasks' is a list of task id's for tasks that need to be monitored. The task will check the status of all tasks in the list and then update the status of parent task. If not all of the tasks are in a final state, the task dispatches itself again with a list of remaining tasks and the same parent task id. Each time this task is dispatched with a delay of 5 minutes or another configurable value.

Define RepoProfileApplicabilityCalculation task as a celery task that takes an existing repo profile applicability and perform the work here [0]

Create a new RepoApplicabilityCalculationTask as a Pulp Task that will dispatch 1 RepoProfileApplicabilityCalculation task for each repo applicability profile that needs to be updated. Then it dispatches TaskMonitorTask and passes it the list of RepoProfileApplicabilityCalculation tasks that were dispatched and the id of itself (RepoApplicabilityCalculationTask)

[0] https://github.com/pulp/pulp/blob/2.6-dev/server/pulp/server/managers/consumer/applicability.py#L141:L158

@dkliban When you say RepoApplicabilityCalculationTask is a Pulp task do you mean it inherits from Pulp's base Task? If so then it will be auto-marked as completed as soon as it is finished because of the on_success or on_failure handlers that provides.

That would need to be somehow disabled and the final call to TaskMonitorTask would need to set it specifically. What can we do in that area?

Also one other important point to consider is using apply_async versus apply_async_with_reservation. Does the task RepoApplicabilityCalculationTask need a reservation to ensure a repo operation doesn't happen underneath it? What do you think?

On 07/21/2015 02:41 PM, Pulp wrote:

Since we are trying to move away from depending on the results backend

IMO, it's OK to use the broker as a results backend for this purpose.
Have you considered that since it may be easier?