Level INFO messages not being logged in worker processes.
Level INFO messages not being logged in worker processes in Pulp3. Something with logging not setup?
#1 Updated by bmbouter almost 3 years ago
Supposedly this was working before, but it was hard to test because we were running workers in the foreground. We have configured the Django logger here
We should troubleshoot this while launching the workers with the systemd unit files in place and we can start/stop wokers not in the foreground. Maybe we need to configure logging on systemd?
#3 Updated by bmbouter almost 3 years ago
Yes I agree we probably need to use the syslog handler. The Django docs say that you can use any Python provided log handler, so I think the one we want is this one: https://docs.python.org/3.5/library/logging.handlers.html#sysloghandler
@mhrivnak: so for the Pulp3 MVP are we to use syslog as our default and over time add the ability to configure other log handlers (aka stdout when running in docker)?
#5 Updated by mhrivnak almost 3 years ago
I'm hoping we can just re-use our pulp 2 logging code, which supports both syslog and stream. The latter is particularly valuable for containerized deployments.
Is there anything about logging we need to change from Pulp 2? Or any reason we can't re-use that code?
#6 Updated by bmbouter almost 3 years ago
Early on in Pulp3, @smyers and I looked into this and we thought that the Pulp2 logger couldn't possibly be as good as the Django logger. It's at the very least not as well tested and something we have to maintain. Django supports both syslog and stream also.
I think we need to have a default (like syslog) and have a setting to use another logger. The existing settings "overlay" over the settings.py file at startup so we are well setup to allow the user to configure their logger the Django way. I don't think we want to bring along our custom logger from Pulp2.
#8 Updated by mhrivnak almost 3 years ago
The content app handles logging in the "django way", as opposed to the rest of Pulp. Perhaps we can just use this for all of Pulp?
#9 Updated by bmbouter almost 3 years ago
Yes I think we can use that logger everywhere and just have the settings.yaml overlay setting on top of it. I think the workers already are using those loggers even. For example, celery does all of that 'fixup django' strangeness due to it being Django aware.
The surprising thing to me is that the INFO log level is not sent to syslog with the default django logger as we have it configured. Maybe only routing WARN to syslog is normal? I'm really not sure.
#10 Updated by mhrivnak almost 3 years ago
Pulp 3 currently is configured to use the stream handler, not the syslog handler.
So that presumably means log output is going to stdout, where systemd is collecting it. If that's true, you'd be able to see it in "systemctl status $service". It could be that systemd has a policy where any WARN or higher log statements sent to stdout get automatically put in the system log, which would explain what you see. But that's just speculation.
In any case, we need to support the syslog handler. The pulp 2 content app config, linked above, is a good model for that.
#13 Updated by bmbouter almost 3 years ago
After talking some in #systemd it sounds like it routes nothing to syslog by default but "some distros" do route to syslog. I think if we want logs to go to syslog we should send them to syslog. Given the varying behavior I think defaulting to syslog for Pulp3 as we did in Pulp2 would be good.
Also the enabling of other loggers is already done, we just need to add the example I think to the yaml file here
So to accomplish this idea I added these checklist items.
Please register to edit this issue