Pulp

I believe users have needs to understand the overall progress of a group of tasks and distinctly see the progress of a single task. To the extent that is true, the design to have TaskGroupProgress be totally new objects that don't aggregate will force plugin writers to intentionally think about what progress is reported at each level. This is an application of the explicit is better than implicit design principal. The aggregation design would work in most cases, and be easier for plugin writers, but I don't think as strong for users.

For the question on figuring out 'total', generally the plugin writer sets it based on the workload they understand. I don't expect GroupProgressReport to make this race condition free in all cases. Here are three scenarios I think about regarding how various workloads could handle this depending on various needs.

• There is no race condition on 'total'. The 'total' is known at some point and is only ever set once. No one but the user ever reads it.

• There is only a write-read race condition on 'total'. In this case the writer sets it, calls save() and other tasks that read it, they can't be guaranteed 'total' is up to date unless it's set once. In the case it's set multiple times, they would need to have some sort of synchronization but TaskGroupProgress would not handle this for them in any way. I think this is also unlikely to be needed.

• There is a write-write race condition. In this case multiple processes are writing to 'total' based on portions of the work they are discovering. In this case the F() values are the way. Or someone can use a database transaction and handle the transaction-failed errors when one saves and the other doesn't.

Overall I don't think TaskGroupProgress needs to provide much except a F() based implementation for implementing the 'done' count because that's the one that is likeliest to be incremented across multiple processes.

The TaskGroupProgress seems reasonable to me. I'm guessing we'll probably have to use F() to update totals.

The 'total' is known at some point and is only ever set once.

Could you give more information about how this would work?

daviddavis wrote:

The TaskGroupProgress seems reasonable to me. I'm guessing we'll probably have to use F() to update totals.

The 'total' is known at some point and is only ever set once.

Could you give more information about how this would work?

Sure. The import/export example I think is this case actually. IIRC, for imports, the first task to run reads the archive to import and determines how many/which repos need updating and dispatches one "sub-task" for each of them. The first task after reading the number of repos that need updating would set total when it is known. The sub-tasks work through the work but do not modify total again.

I agree that we might find a way to set the total only once for the current use cases. I'm not entirely sure about the migration plugin because some of its work can be identified only during the migration itself but there is a chance that all those items are happening before subtasks for more well defined scope are dispatched.

+1 to start with F() for done. And if we have a use case, we can add it for total later.

https://github.com/pulp/pulpcore/pull/755