Project

Profile

Help

Task #2092

Create django model(s) for progress reporting

Added by mhrivnak over 3 years ago. Updated 7 months ago.

Status:
MODIFIED
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
master
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 7

Description

Create django models for the functionality that includes progress reporting. The model should support reporting of multiple metrics at a time. For example, both number of RPMs and number of bytes downloaded. It should also support reporting of progress at multiple stages of a workflow concurrently, such as a case where one step feeds into the next pipeline-style.

Once the model is done, make any required tasks to create an internal API for plugins to use it. It's possible that a helper API won't be valuable depending on how easy it is to use the model directly.

Although at least one feature could be helpful, which is throttling. Many plugins or other workflows may want to report progress as often as they like, but have the framework only attempt writing to the DB once or twice a second. When it's hard to predict if each download will take 10s or 10ms, it's very helpful to have one bit of tooling that meters the expensive DB writes for you.

Notes from the team on desired behaviors:

Q: What specific features will this have?
Tasks which show progress (total_count, current_completed, state)
Tasks that don't show progress, they only have a state
Q: How will the data be modeled? Presumably something associated with a Task.
+1 to a Task but can we please not call it a task. We have a lot of Tasks already in the data layer. Maybe a ProgressReport or ActivityReport?
Q: Are we sticking with linear progress reporting which can only show the state for 1 Task at a time.
I think supporting parallel progress reporting will be mostly easy.
Having parallel progress reporting will enable higher performing plugins using, stream processing or other concurrency models.
I (mhrivnak) think it's a great idea to make a model for the data that supports this, and I think using it will then be pretty straight-forward.

Related issues

Related to Pulp - Task #2226: Rename pulp.platform to pulp.app CLOSED - CURRENTRELEASE Actions
Blocked by Pulp - Task #2086: add django DB boilerplate CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision 6bb79329 View on GitHub
Added by bmbouter over 3 years ago

Adds Progress Reporting models into pulp.platform.models.

This creates three new models: ProgressBar, ProgressReportingBase,
and ProgressSpinner. These can be used to create progress reports.

https://pulp.plan.io/issues/2092
closes #2092

Revision 6bb79329 View on GitHub
Added by bmbouter over 3 years ago

Adds Progress Reporting models into pulp.platform.models.

This creates three new models: ProgressBar, ProgressReportingBase,
and ProgressSpinner. These can be used to create progress reports.

https://pulp.plan.io/issues/2092
closes #2092

Revision 6bb79329 View on GitHub
Added by bmbouter over 3 years ago

Adds Progress Reporting models into pulp.platform.models.

This creates three new models: ProgressBar, ProgressReportingBase,
and ProgressSpinner. These can be used to create progress reports.

https://pulp.plan.io/issues/2092
closes #2092

Revision f4459998 View on GitHub
Added by bmbouter about 3 years ago

Updates saving behaviors of `ProgressReport`

The context manager now provides saving in all cases.
Also the docstrings have been updated to make this
behavior clear to the user.

Also the short form of `ProgressBar` was fixed
to create the 'pb' variable within the 'with'
section.

https://pulp.plan.io/issues/2092
re #2092

Revision f4459998 View on GitHub
Added by bmbouter about 3 years ago

Updates saving behaviors of `ProgressReport`

The context manager now provides saving in all cases.
Also the docstrings have been updated to make this
behavior clear to the user.

Also the short form of `ProgressBar` was fixed
to create the 'pb' variable within the 'with'
section.

https://pulp.plan.io/issues/2092
re #2092

Revision f4459998 View on GitHub
Added by bmbouter about 3 years ago

Updates saving behaviors of `ProgressReport`

The context manager now provides saving in all cases.
Also the docstrings have been updated to make this
behavior clear to the user.

Also the short form of `ProgressBar` was fixed
to create the 'pb' variable within the 'with'
section.

https://pulp.plan.io/issues/2092
re #2092

History

#1 Updated by mhrivnak over 3 years ago

  • Blocked by Task #2086: add django DB boilerplate added

#3 Updated by mhrivnak over 3 years ago

  • Description updated (diff)

#4 Updated by jortel@redhat.com over 3 years ago

  • Groomed changed from No to Yes

#5 Updated by mhrivnak over 3 years ago

  • Sprint/Milestone set to 24

#6 Updated by semyers over 3 years ago

There are several Django plugins that might help with this. My first thought was a state machine would be useful, so this popped into mind: https://www.djangopackages.com/packages/p/django-fsm/

The README for that project references something that might be a good fit for the modeling task, but right now it's only in beta: https://www.djangopackages.com/packages/p/viewflow/. It looks tempting, but their commercial license (https://www.gnu.org/licenses/agpl.html) appears to be in conflict with the code license (GNU AGPL: https://github.com/viewflow/viewflow/blob/master/LICENSE), so we'd probably need to get a lawyer involved to clear us to use it. Given that it's still in beta and might be such a legal hairball, it might be good to skip it. Having to log in to read the docs also stinks, and makes this whole project feel non-copyleft despite its GNU license.

No other "workflow" plugins that I can find support Python 3 (https://www.djangopackages.com/grids/g/workflow/).

#7 Updated by bmbouter over 3 years ago

  • Status changed from NEW to ASSIGNED

I'm interested in this so I'm going to take it as assigned. I will get some starter ideas posted in a PR and then send around for discussion.

#8 Updated by bmbouter over 3 years ago

  • Assignee set to bmbouter

#9 Updated by mhrivnak over 3 years ago

  • Sprint/Milestone changed from 24 to 25

#10 Updated by bmbouter over 3 years ago

  • Blocks Task #2178: Remove the conduits and port usage to use the progress API added

#11 Updated by semyers over 3 years ago

  • Related to Task #2226: Rename pulp.platform to pulp.app added

#12 Updated by bmbouter over 3 years ago

  • Status changed from ASSIGNED to POST

#13 Updated by bmbouter about 3 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#14 Updated by semyers about 3 years ago

  • Platform Release set to master

#15 Updated by semyers about 3 years ago

  • Tags Pulp 3, Pulp 3 Modeling added

#16 Updated by bmbouter over 2 years ago

  • Blocks deleted (Task #2178: Remove the conduits and port usage to use the progress API)

#17 Updated by bmbouter over 2 years ago

  • Tags deleted (Pulp 3 Modeling)

#18 Updated by bmbouter over 1 year ago

  • Sprint set to Sprint 7

#19 Updated by bmbouter over 1 year ago

  • Sprint/Milestone deleted (25)

#20 Updated by daviddavis 7 months ago

  • Sprint/Milestone set to 3.0

#21 Updated by bmbouter 7 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF