Story #1939
closed
As a user, I would like to be able to profile Pulp tasks
100%
Description
Problem Overview¶
Often, users (both developers and people out in the "real world") want to be able to figure out why a task is taking a long time to complete. What bits are taking the majority of the time? It is possible to answer this question with profiling tools. Python's standard library contains ``profile`` and ``cProfile``. It would nice if I could flip a configuration setting and have every task Pulp dispatches be profiled.
Implementation Plan¶
It might be possible (and easy, even) to add to the task decorator we use on functions to check the server config for a setting, ``profile_tasks``. If true, it wraps the task function with cProfile and sends it on its merry way. There are, of course, other ways we could do this, but that's just one way that came to mind.
Updated by bmbouter over 8 years ago
@jcline, FYI dkliban had put this together: http://pulp.readthedocs.io/en/latest/dev-guide/debugging.html
We considered doing something similar to this story. We still can, but our reasoning not to was that a developer would need to read the report, so we expected a developer could add the 2-3 lines of import statements needed to generate the report.
What do you think?
Updated by jcline@redhat.com over 8 years ago
I do normally add the necessary lines, but I find it onerous. I also just encountered a situation where a user was experiencing slow tasks. Instead of being able to say "Hey, flip on profiling and send me these files", I had to either walk them through writing Python or (as was the case yesterday) get access to the system myself.
Updated by bmbouter over 8 years ago
I hadn't considered the remote debugging case; that makes sense, let's do it.
Updated by bmbouter almost 8 years ago
- Status changed from NEW to POST
- Assignee set to ehelms@redhat.com
- Sprint/Milestone set to 30
@ehelms put in a PR on this so I'm setting him as assignee.
PR available at: https://github.com/pulp/pulp/pull/2882/
Added by Eric D Helms almost 8 years ago
Added by Eric D Helms almost 8 years ago
Revision 07202a9b | View on GitHub
closes #1939: Enable cProfile of tasks
Adds the ability to turn on cProfiles of individual Pulp tasks and dumping of the tasks into a directory. This can have an impact on performance. The trade off is that users could investigate individual tasks or provide these cProfiles to developers to help detect anomalies or systematic issues.
Added by bmbouter almost 8 years ago
Added by bmbouter almost 8 years ago
Updated by Anonymous almost 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|07202a9b6cac4f37dc50e2f0f941b8c1ad371567.
Updated by bmbouter almost 8 years ago
- Status changed from 5 to ASSIGNED
I discovered this feature was not working as implemented so I'm pulling back to assigned.
- The code was using a slightly different spelling of the config value so it wasn't finding it
- The config value was being treated as a boolean when it was a string
- It did not provide directory management so it fails unless the directory is manually created
I'm making a PR now to fix all of these things.
Updated by bmbouter almost 8 years ago
Also the quoted section of docs weren't rendering as expected. I'm fixing that too.
Added by bmbouter almost 8 years ago
Revision b4db6138 | View on GitHub
Fixes the cProfile Feature
This feature was introduced in 2.12, and is being fixed prior to the release of 2.12.0. This fix includes:
- Updates the server.conf which used
enableinstead ofenabled - Updates the config file checking which treated received a str when it expected a boolean.
- Provides automatic directory creation.
- Documentation updates and fix for blockquote sections which did not render correctly.
Added by bmbouter almost 8 years ago
Revision b4db6138 | View on GitHub
Fixes the cProfile Feature
This feature was introduced in 2.12, and is being fixed prior to the release of 2.12.0. This fix includes:
- Updates the server.conf which used
enableinstead ofenabled - Updates the config file checking which treated received a str when it expected a boolean.
- Provides automatic directory creation.
- Documentation updates and fix for blockquote sections which did not render correctly.
Updated by bmbouter almost 8 years ago
- Status changed from ASSIGNED to POST
PR available at: https://github.com/pulp/pulp/pull/2929
Updated by bmbouter almost 8 years ago
- Status changed from POST to MODIFIED
Applied in changeset pulp|b4db6138611802ab65828193250143b7989c5daf.
Updated by semyers almost 8 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
closes #1939: Enable cProfile of tasks
Adds the ability to turn on cProfiles of individual Pulp tasks and dumping of the tasks into a directory. This can have an impact on performance. The trade off is that users could investigate individual tasks or provide these cProfiles to developers to help detect anomalies or systematic issues.
https://pulp.plan.io/issues/1939