Project

Profile

Help

Issue #2836

closed

Pulp makes a wall of logs during sync

Added by bmbouter almost 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

Pulp is putting out a log statement for every single unit downloaded. Syncing EPEL logs thousands of lines. These are too many log statements making the logs unusable. For example here is a small snippet while syncing EPEL:

Jun 23 11:32:07 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bouncycastle-pkix-javadoc-1.50-1.el7.noarch.rpm.
Jun 23 11:32:07 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/boxes-1.1.1-4.el7.x86_64.rpm.
Jun 23 11:32:07 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bpython-0.12-6.1.el7.noarch.rpm.
Jun 23 11:32:07 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bouncycastle-pkix-1.50-1.el7.noarch.rpm.
Jun 23 11:32:07 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/brightnessctl-0.2.1-1.el7.x86_64.rpm.
Jun 23 11:32:08 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bouncycastle-1.50-1.el7.noarch.rpm.
Jun 23 11:32:09 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bro-doc-2.4.1-3.el7.x86_64.rpm.
Jun 23 11:32:09 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/broccoli-2.4.1-3.el7.x86_64.rpm.
Jun 23 11:32:09 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/broccoli-devel-2.4.1-3.el7.x86_64.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/broctl-2.4.1-3.el7.noarch.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/btest-0.54-1.el7.noarch.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bti-034-3.el7.x86_64.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/btkbdd-1.3-5.el7.x86_64.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bubblewrap-0.1.7-1.el7.x86_64.rpm.
Jun 23 11:32:10 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bucardo-4.5.0-12.el7.noarch.rpm.
Jun 23 11:32:11 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bugwarrior-1.4.0-1.el7.noarch.rpm.
Jun 23 11:32:11 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bugyou-0.2.2-1.el7.noarch.rpm.
Jun 23 11:32:11 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bugyou_plugins-0.1.5-1.el7.noarch.rpm.
Jun 23 11:32:11 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bullet-2.82-2.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bro-2.4.1-3.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bullet-devel-2.82-2.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bullet-extras-devel-2.82-2.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/burp-client-1.4.40-1.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bullet-extras-2.82-2.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/burp-server-1.4.40-1.el7.x86_64.rpm.
Jun 23 11:32:12 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bwm-ng-0.6.1-2.el7.x86_64.rpm.
Jun 23 11:32:13 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/byobu-5.73-4.el7.noarch.rpm.
Jun 23 11:32:13 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bwa-0.7.12-1.el7.x86_64.rpm.
Jun 23 11:32:13 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/b/bzrtools-2.5-5.el7.x86_64.rpm.
Jun 23 11:32:13 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CBFlib-devel-0.9.5.1-1.el7.x86_64.rpm.
Jun 23 11:32:13 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CGSI-gSOAP-1.3.10-1.el7.x86_64.rpm.
Jun 23 11:32:14 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CBFlib-0.9.5.1-1.el7.x86_64.rpm.
Jun 23 11:32:14 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CGSI-gSOAP-devel-1.3.10-1.el7.x86_64.rpm.
Jun 23 11:32:14 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CUnit-2.1.3-8.el7.x86_64.rpm.
Jun 23 11:32:14 dev pulp[6563]: nectar.downloaders.threaded:INFO: Download succeeded: https://dl.fedoraproject.org/pub/epel/7/x86_64/c/CUnit-devel-2.1.3-8.el7.x86_64.rpm.

Thanks to @daviddavis for identifying that this behavior was introduced as part of Issue #2315. Successful downloads should not be logged at INFO.


Related issues

Related to Nectar - Issue #2315: Better logging at INFO levelCLOSED - CURRENTRELEASEdaviddavisActions
Actions #1

Updated by daviddavis almost 7 years ago

  • Related to Issue #2315: Better logging at INFO level added
Actions #2

Updated by mhrivnak almost 7 years ago

This behavior was specifically requested by multiple stakeholders. We should follow up with them before making any plans to remove it.

It's also worth noting that this behavior is not all that different from how other popular services operate. Web servers log each request. MTAs log each email. Etc. Another approach would be to focus on making it convenient to filter the types of statements you do or don't want.

Actions #3

Updated by bmbouter almost 7 years ago

mhrivnak wrote:

This behavior was specifically requested by multiple stakeholders. We should follow up with them before making any plans to remove it.

Who are the users desiring this behavior? We can't check with them without knowing who they are. Since you know who they are please ask them to comment on this issue.

It's also worth noting that this behavior is not all that different from how other popular services operate. Web servers log each request. MTAs log each email. Etc. Another approach would be to focus on making it convenient to filter the types of statements you do or don't want.

Note that those services don't send their logs to syslog like Pulp does. This allows them to write huge amounts of logs in a dedicated place. The issue here is that Pulp is filling up the system log which is not ok.

Actions #4

Updated by daviddavis almost 7 years ago

FWIW, I checked with the person who reported the original bug and he confirmed that he would prefer to log even successful downloads. I asked just setting his log level to DEBUG and he said that creates too much other noise.

Actions #5

Updated by bmbouter almost 7 years ago

One idea is to have Pulp log this line at DEBUG and that user can use rsyslogd to allow that single line to be matched and logged with the rest of the Pulp logs logging at INFO. That way Pulp can still have a sane default and that user can have the logs exactly the way they want them.

What about ^ as an idea?

Actions #6

Updated by bmbouter almost 7 years ago

  • Status changed from NEW to CLOSED - WONTFIX

After several conversations with other developers, this behavior isn't clearly wrong and some people value it. I'm going to close as WONTFIX for now and actual users can raise it if they think it's too much logging.

Actions #7

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF