Pulp

It would make the logs statements long, but we should have every Pulp log statement running inside a task did include the Task id. Pids are not as useful because the task id currently logged is the parent's pid and the rest of the log messages are the child's different pid. Also having task ids would be better for clustered installs where pids could collide. We could figure out how to implement this I think.

Regarding the pid's, if we did that we would publish a syslog config which would include the pids. But we should just add the task ids.

Instead of Dec 10 19:56:55 dev pulp[32137]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): repos.fedorapeople.org

It could be: Dec 10 19:56:55 dev pulp[32137]: requests.packages.urllib3.connectionpool:INFO: [0ca4399c-51eb-47c8-9a07-145e61e389a5] Starting new HTTPS connection (1): repos.fedorapeople.org

Or it could be: Dec 10 19:56:55 dev pulp[32137]: [0ca4399c-51eb-47c8-9a07-145e61e389a5] requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): repos.fedorapeople.org

I prefer the former I think so that all log statements have the position of the python package and log level in the same place when running inside or outside of a task. Log statements running outside of tasks will not get the UUID.

What do others think?

Just an idea: what about only logging the first 8 characters of the task ID? That should be plenty of uniqueness to avoid collision while providing a good reference.

Dec 10 19:56:55 dev pulp[32137]: requests.packages.urllib3.connectionpool:INFO: [0ca4399c-51eb-47c8-9a07-145e61e389a5] Starting new HTTPS connection (1): 

vs

Dec 10 19:56:55 dev pulp[32137]: requests.packages.urllib3.connectionpool:INFO: [0ca4399c] Starting new HTTPS connection (1): 

mhrivnak +1 to that idea. Retaining 8 characters will still keep collision probabilities low. 1 in 36^8 = 2,821,109,907,456 by my count.

My initial thought (and please tell me if this is a bad idea) was to write a Filter to set a task id on the LogRecord and then have the logger output that using a custom Formatter. However, that would require some global way to be able to get the task id but I didn't see any code for doing that. Is there a way to do that?

Here is a global way to get the task id [0]. I can't speak to the implementation approach, but whatever it does it needs to work well with our existing logger [1].

[0]: https://github.com/bmbouter/pulp/blob/f9355d21559fa880fd7654134d8f34dcf6b85acf/server/pulp/server/async/tasks.py#L643-L643
[1]: https://github.com/pulp/pulp/blob/master/server/pulp/server/logs.py