Project

Profile

Help

Task #2092

closed

Create django model(s) for progress reporting

Added by mhrivnak over 7 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
master
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 7
Quarter:

Description

Create django models for the functionality that includes progress reporting. The model should support reporting of multiple metrics at a time. For example, both number of RPMs and number of bytes downloaded. It should also support reporting of progress at multiple stages of a workflow concurrently, such as a case where one step feeds into the next pipeline-style.

Once the model is done, make any required tasks to create an internal API for plugins to use it. It's possible that a helper API won't be valuable depending on how easy it is to use the model directly.

Although at least one feature could be helpful, which is throttling. Many plugins or other workflows may want to report progress as often as they like, but have the framework only attempt writing to the DB once or twice a second. When it's hard to predict if each download will take 10s or 10ms, it's very helpful to have one bit of tooling that meters the expensive DB writes for you.

Notes from the team on desired behaviors:

Q: What specific features will this have?
Tasks which show progress (total_count, current_completed, state)
Tasks that don't show progress, they only have a state
Q: How will the data be modeled? Presumably something associated with a Task.
+1 to a Task but can we please not call it a task. We have a lot of Tasks already in the data layer. Maybe a ProgressReport or ActivityReport?
Q: Are we sticking with linear progress reporting which can only show the state for 1 Task at a time.
I think supporting parallel progress reporting will be mostly easy.
Having parallel progress reporting will enable higher performing plugins using, stream processing or other concurrency models.
I (mhrivnak) think it's a great idea to make a model for the data that supports this, and I think using it will then be pretty straight-forward.

Related issues

Related to Pulp - Task #2226: Rename pulp.platform to pulp.appCLOSED - CURRENTRELEASEsemyers

Actions
Blocked by Pulp - Task #2086: add django DB boilerplateCLOSED - CURRENTRELEASE

Actions
Actions #1

Updated by mhrivnak over 7 years ago

  • Blocked by Task #2086: add django DB boilerplate added
Actions #3

Updated by mhrivnak over 7 years ago

  • Description updated (diff)
Actions #4

Updated by jortel@redhat.com over 7 years ago

  • Groomed changed from No to Yes
Actions #5

Updated by mhrivnak over 7 years ago

  • Sprint/Milestone set to 24
Actions #6

Updated by semyers over 7 years ago

There are several Django plugins that might help with this. My first thought was a state machine would be useful, so this popped into mind: https://www.djangopackages.com/packages/p/django-fsm/

The README for that project references something that might be a good fit for the modeling task, but right now it's only in beta: https://www.djangopackages.com/packages/p/viewflow/. It looks tempting, but their commercial license (https://www.gnu.org/licenses/agpl.html) appears to be in conflict with the code license (GNU AGPL: https://github.com/viewflow/viewflow/blob/master/LICENSE), so we'd probably need to get a lawyer involved to clear us to use it. Given that it's still in beta and might be such a legal hairball, it might be good to skip it. Having to log in to read the docs also stinks, and makes this whole project feel non-copyleft despite its GNU license.

No other "workflow" plugins that I can find support Python 3 (https://www.djangopackages.com/grids/g/workflow/).

Actions #7

Updated by bmbouter over 7 years ago

  • Status changed from NEW to ASSIGNED

I'm interested in this so I'm going to take it as assigned. I will get some starter ideas posted in a PR and then send around for discussion.

Actions #8

Updated by bmbouter over 7 years ago

  • Assignee set to bmbouter
Actions #9

Updated by mhrivnak over 7 years ago

  • Sprint/Milestone changed from 24 to 25
Actions #10

Updated by bmbouter over 7 years ago

  • Blocks Task #2178: Remove the conduits and port usage to use the progress API added
Actions #11

Updated by semyers over 7 years ago

  • Related to Task #2226: Rename pulp.platform to pulp.app added

Added by bmbouter over 7 years ago

Revision 6bb79329 | View on GitHub

Adds Progress Reporting models into pulp.platform.models.

This creates three new models: ProgressBar, ProgressReportingBase, and ProgressSpinner. These can be used to create progress reports.

https://pulp.plan.io/issues/2092 closes #2092

Added by bmbouter over 7 years ago

Revision 6bb79329 | View on GitHub

Adds Progress Reporting models into pulp.platform.models.

This creates three new models: ProgressBar, ProgressReportingBase, and ProgressSpinner. These can be used to create progress reports.

https://pulp.plan.io/issues/2092 closes #2092

Actions #12

Updated by bmbouter over 7 years ago

  • Status changed from ASSIGNED to POST
Actions #13

Updated by bmbouter over 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #14

Updated by semyers over 7 years ago

  • Platform Release set to master
Actions #15

Updated by semyers over 7 years ago

  • Tags Pulp 3, Pulp 3 Modeling added

Added by bmbouter over 7 years ago

Revision f4459998 | View on GitHub

Updates saving behaviors of ProgressReport

The context manager now provides saving in all cases. Also the docstrings have been updated to make this behavior clear to the user.

Also the short form of ProgressBar was fixed to create the 'pb' variable within the 'with' section.

https://pulp.plan.io/issues/2092 re #2092

Added by bmbouter over 7 years ago

Revision f4459998 | View on GitHub

Updates saving behaviors of ProgressReport

The context manager now provides saving in all cases. Also the docstrings have been updated to make this behavior clear to the user.

Also the short form of ProgressBar was fixed to create the 'pb' variable within the 'with' section.

https://pulp.plan.io/issues/2092 re #2092

Actions #16

Updated by bmbouter almost 7 years ago

  • Blocks deleted (Task #2178: Remove the conduits and port usage to use the progress API)
Actions #17

Updated by bmbouter over 6 years ago

  • Tags deleted (Pulp 3 Modeling)
Actions #18

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 7
Actions #19

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (25)
Actions #20

Updated by daviddavis almost 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #21

Updated by bmbouter almost 5 years ago

  • Tags deleted (Pulp 3)
Actions #22

Updated by bmbouter over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF