Pulp

In regards to why the failure was not logged, seems like log_level of ERROR isn't high enough to report meaniful information. DEBUG did the job.

Regards

I stumbled onto https://media.readthedocs.org/pdf/pulp/stable/pulp.pdf while I was trying to rectify another problem I was facing with pulp, which elobrates a bit more on download policies. Does immediate depends on python-pulp-streamer? Which explains perhaps why it wasn't listed as a dependency. Although, I have Apache, Varnish, python-pulp-streamer functioning, I still can't run:

[root@pulp log]# pulp repo download --repo-id=webmin-noarch-base                                    
+----------------------------------------------------------------------+
              Downloading Repository [webmin-noarch-base]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Download Repository Content
[================                                  ] 33%
1 of 3 items
... failed

Task Succeeded

May 11 08:00:03 pulp pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): pulp.lhprojects.int
May 11 08:00:03 pulp pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): pulp.lhprojects.int
May 11 08:00:03 pulp pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): pulp.lhprojects.int
May 11 08:00:04 pulp pulp: nectar.downloaders.threaded:WARNING: Skipping requests to pulp.lhprojects.int:443 due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)
May 11 08:00:04 pulp pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/ec/49f00d534bec06478085013b0d9bfee4c0c81acd2c1315c00cba7ea6ab4bed/usermin-1.701-1.noarch.rpm failed: A connection error occurred.
May 11 08:00:04 pulp pulp: nectar.downloaders.threaded:WARNING: Skipping requests to pulp.lhprojects.int:443 due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)
May 11 08:00:04 pulp pulp: nectar.downloaders.threaded:WARNING: Skipping requests to pulp.lhprojects.int:443 due to repeated connection failures: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)
May 11 08:00:04 pulp pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/e4/f1bc99febc315c784368c1c66b7a614d1bef8bd8434becec026c6b023f9023/webmin-1.791-1.noarch.rpm failed: A connection error occurred.
May 11 08:00:04 pulp pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/91/8cad1b9b81a8c1fb5f85db579d7fac407cd733b24b4c436671edf4afd5e273/usermin-webmail-1.701-1.noarch.rpm failed: A connection error occurred.
May 11 08:00:04 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_repo[55c85f52-9b4c-4ab1-a32e-3e9596e8606b] succeeded in 21.5763582741s: None

Regards

Pulp is designed to have the streamer be an optional component which needs to be installed directly. Perhaps the docs should be more clear on that I could see purposing this bug for that, but the title and description would need to be edited.

Since it is an optional component, we should not include it in the pulp-server-qpid install which is reserved for required server-side bits when Pulp is run with the Qpid broker.

Immediate downloads are the default and should not require python-pulp-streamer which is what makes python-pulp-streamer optional.

Given ^ what do you want to do with this bug report?

With respect to your last comment about not being able to ``pulp repo download --repo-id=webmin-noarch-base``:

Pulp is making requests to the Apache reverse proxy and defaults to secure. This means it uses HTTPS and verifies the certificate that Apache offers. If you are running Pulp on the same host as the Apache reverse proxy/Varnish/Pulp streamer setup, you can safely use HTTP since all communication is happening locally. To do this, change the settings in the [lazy] section of ``/etc/pulp/server.conf``. Specifically, you can turn ``https_retrieval: true`` to ``https_retrieval: false``.

If, on the other hand, you want to use HTTPS, you'll need to configure the Pulp server to trust the CA that signed the SSL certificate the Apache reverse proxy is serving.

bmbouter wrote:

Pulp is designed to have the streamer be an optional component which needs to be installed directly. Perhaps the docs should be more clear on that I could see purposing this bug for that, but the title and description would need to be edited.

Yes, this would be fanstatic to save cost in time having 'Optional Components' section in http://pulp.readthedocs.io/en/latest/user-guide/installation.html#prerequisites, with a link pointing to http://pulp.readthedocs.io/en/latest/user-guide/deferred-download.html.

Since it is an optional component, we should not include it in the pulp-server-qpid install which is reserved for required server-side bits when Pulp is run with the Qpid broker

This is not a bug? Wouldn't it be cleaner to have /usr/lib/systemd/system/pulp_streamer.service be provided by python-pulp-streamer-2.8.2-1.el7.noarch.

[root@pulp ~]# rpm -qf /usr/lib/systemd/system/pulp_streamer.service 
pulp-server-2.8.2-1.el7.noarch
python-pulp-streamer-2.8.2-1.el7.noarch

Immediate downloads are the default and should not require python-pulp-streamer which is what makes python-pulp-streamer optional.

Uh, when I switch download policy on my ecryptfs-x86_64 repo (which is a repository located on my internal lan) from background to immediate. Two new tasks are created however content units are not downloaded:

May 11 17:42:11 pulp pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
May 11 17:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[6ab08684-894f-4a33-a083-055792165a99]
May 11 17:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[c72eac06-6c0d-4993-a175-b3c34bcacb2a]
May 11 17:42:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[6ab08684-894f-4a33-a083-055792165a99] succeeded in 0.0532712398563s: None
May 11 17:42:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[c72eac06-6c0d-4993-a175-b3c34bcacb2a] succeeded in 0.456320367055s: None
May 11 18:12:11 pulp pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
May 11 18:12:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[6f9dee02-2a89-406c-8529-95d57cc0b909]
May 11 18:12:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[d12ae7c8-1678-46cc-b495-a71d501b7fc4]
May 11 18:12:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[6f9dee02-2a89-406c-8529-95d57cc0b909] succeeded in 0.0420156719629s: None
May 11 18:12:12 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[d12ae7c8-1678-46cc-b495-a71d501b7fc4] succeeded in 1.1721588918s: None
May 11 18:42:11 pulp pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
May 11 18:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[c8e94407-9c0d-4f1c-9f59-55c59a2e0d97]
May 11 18:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[23d6ec21-952a-47f6-9803-2d6d9e25048e]
May 11 18:42:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[c8e94407-9c0d-4f1c-9f59-55c59a2e0d97] succeeded in 0.0333031900227s: None
May 11 18:42:12 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[23d6ec21-952a-47f6-9803-2d6d9e25048e] succeeded in 1.0592154311s: None
May 11 19:12:11 pulp pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
May 11 19:12:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[be6138e4-e20f-40bd-a08b-7db58c42226d]
May 11 19:12:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[f1976b9c-b149-4372-9a76-e93facc958c2]
May 11 19:12:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[be6138e4-e20f-40bd-a08b-7db58c42226d] succeeded in 0.0498800999485s: None
May 11 19:12:12 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[f1976b9c-b149-4372-9a76-e93facc958c2] succeeded in 1.40341140889s: None
May 11 19:42:11 pulp pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
May 11 19:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[9416b977-daf1-4dfc-b1ea-33343f4534ad]
May 11 19:42:11 pulp pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf]
May 11 19:42:11 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.queue_download_deferred[9416b977-daf1-4dfc-b1ea-33343f4534ad] succeeded in 0.0565808371175s: None
May 11 19:42:12 pulp pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf] succeeded in 1.14704286307s: None

[root@pulp ~]# pulp -vv tasks details --task-id=3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf
+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

2016-05-11 19:57:53,920 - DEBUG - sending GET request to /pulp/api/v2/tasks/3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf/
2016-05-11 19:57:54,178 - INFO - GET request to /pulp/api/v2/tasks/3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf/ with parameters None
2016-05-11 19:57:54,179 - INFO - Response status : 200 

2016-05-11 19:57:54,179 - INFO - Response body :
 {
  "exception": null, 
  "task_type": "pulp.server.controllers.repository.download_deferred", 
  "_href": "/pulp/api/v2/tasks/3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf/", 
  "task_id": "3d82ebb2-6cdb-4e85-b2b2-66c893cfefdf", 
  "tags": [
    "pulp:action:deferred_download"
  ], 
  "finish_time": "2016-05-11T19:42:12Z", 
  "_ns": "task_status", 
  "start_time": "2016-05-11T19:42:11Z", 
  "traceback": null, 
  "spawned_tasks": [], 
  "progress_report": {
    "on_demand_download": [
      {
        "num_success": 0, 
        "description": "Download Cached On-Demand Content", 
        "step_type": "on_demand_download",
        "items_total": 0, 
        "state": "FINISHED", 
        "error_details": [], 
        "details": "", 
        "num_failures": 0, 
        "step_id": "72f09462-8b9a-4193-a8a9-f64194a9fd3b", 
        "num_processed": 0
      }
    ]
  }, 
  "queue": "celery.dq", 
  "state": "finished", 
  "worker_name": "celery", 
  "result": null, 
  "error": null, 
  "_id": {
    "$oid": "57338b137faeb6060a6f6d93"
  }, 
  "id": "57338b137faeb6060a6f6d93"
}

As far as I can see, no content units are downloaded:

[root@pulp log]# ll /var/lib/pulp/content
ls: cannot access /var/lib/pulp/content: No such file or directory

Given ^ what do you want to do with this bug report?

Perhaps changing the title to the following, would be a more fitting description as to what is going on, in this issue: Cannot download content units.

And tag documentation.

Regards

jcline@redhat.com wrote:

With respect to your last comment about not being able to ``pulp repo download --repo-id=webmin-noarch-base``:

Pulp is making requests to the Apache reverse proxy and defaults to secure. This means it uses HTTPS and verifies the certificate that Apache offers. If you are running Pulp on the same host as the Apache reverse proxy/Varnish/Pulp streamer setup, you can safely use HTTP since all communication is happening locally. To do this, change the settings in the [lazy] section of ``/etc/pulp/server.conf``. Specifically, you can turn ``https_retrieval: true`` to ``https_retrieval: false``.

If, on the other hand, you want to use HTTPS, you'll need to configure the Pulp server to trust the CA that signed the SSL certificate the Apache reverse proxy is serving.

Thanks a bunch, I spent 4 hours figuring out this issue but unfortunately, no-one, seemed to encounter this before. Hopefully this issue shows up in Google searchs.

Regards

lhorace wrote:

This is not a bug? Wouldn't it be cleaner to have /usr/lib/systemd/system/pulp_streamer.service be provided by python-pulp-streamer-2.8.2-1.el7.noarch.

Having both pulp-server and python-pulp-streamer provide /usr/lib/systemd/system/pulp_streamer.service is definitely a packaging bug. I've filed https://pulp.plan.io/issues/1924 against our packaging project.

Uh, when I switch download policy on my ecryptfs-x86_64 repo (which is a repository located on my internal lan) from background to immediate. Two new tasks are created however content units are not downloaded:

[...]

[...]

As far as I can see, no content units are downloaded:

[...]

This is expected. That task is dispatched on a schedule (you can change it in /etc/pulp/server.conf) and checks if there is any cached content. If there isn't any, it doesn't do anything.

The problem, the Content Units are not downloadedd when I switch behavior (switch download policy)...Minor issue...

I have to pulp-admin repo download *

Consumer -> Pulp -> Apache -> Nginx (I have request store for about 30 minutes, then re-request) -> RPM Package

lhorace wrote:

The problem, the Content Units are not downloadedd when I switch behavior (switch download policy)...Minor issue...

I have to pulp-admin repo download *

Consumer -> Pulp -> Apache -> Nginx (I have request store for about 30 minutes, then re-request) -> RPM Package

Ah, I see. I believe you're experiencing https://pulp.plan.io/issues/1778 which appears to be in the upcoming 2.8.3 release.

You guys can close this ticket and I will watch 1778

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.