Pulp

The progress report aggregation would be useful in the following case:

The migration plugin dispatches 1000 migrate_repository tasks. Each migrate_repository task creates progress reports for "Create Repository", "Create Publication", "Create Distribution". Each of these progress reports has a "total" of 1. After the repository is created the "done" field for the related progress report is set to 1. The same follows publication and distribution.

The user looking at the task_group associated with these 1000 tasks does not want to see 3000 progress reports. The user wants to see 3 progress reports stating that 1000 out of 1000 items are done for each.

The progress report aggregation would not be as useful if we implemented a new API that lets users sync many repositories at once.

If the task group contains many sync tasks then the aggregated progress reports would look something like this:

Downloading Metadata Files: 10 done of 10 total
Optimizing Sync: 5 done of 5 total
Parsed Modulemd: 56 done of 56 total
Parsed Modulemd-defaults: 56 done of 56 total 
Parsed Comps: 14 done of 14 total
Parsed Packages: 35751 done of 35751 total
Parsed Advisories: 523 done of 523 total 

It would much better if the sync tasks could record progress in two types of progress reports. The one we have now, ProgressReport, and a GroupProgressReport. A sync task could then provide less granular progress for the TaskGroup API, e.g.: Syncing repository 'foo'. In that case the TaskGroup API could return something like this:

Syncing repository 'foo': 0 done of 1 total.
Syncing repository 'bar': 1 done of 1 total.
Syncing repository 'fedora': 0 done of 1 total.

For your sync task example, were you imagining that these would be regular sync tasks? In other words, could these sync tasks run as independent tasks not part of a TaskGroup and also as part of a TaskGroup? If so, I guess we'd have to introduce conditional reporting logic?

These would be regular sync tasks that can be run on their own or as part of a group. The sync task would always record ProgressReports. However, if a Group is passed in to the task, a GroupProgressReport would be recorded also. So yes, some conditional logic will be necessary.

I suppose where I'm at is the question "if conditional logic is required anyway, is there a sufficient reason to have a separate reporting structure rather than simply changing what is reported at the task level".

I can see that there might be reasons not to change the individual task level reporting, but are we sure that it is so necessary to justify adding more parallel reporting mechanisms. If we are, then that's fine.

Users will want to have both detailed ProgressReports and GroupProgressReports for sync tasks. I imagine when the sync task fails for one of the tasks in the Group, the user will want to see the detailed version of progress using the Tasks api.