Pulp

Issue #4990

Updated by bmbouter almost 5 years ago

h3. Problem It is not stated in the docs that Ansible-collections supports lazy sync. 

 For a plugin However I was able to support on-demand it needs to take extra steps. Core should not allow sync a user by default to expect to use policy != 'immediate' remote with lazy sync policy. We need either raise an exception if a plugin hasn't done this is not supported or update the work to enable it. Currently values of policy docs. 


 Steps: 

 <pre> 
 git clone https://github.com/pulp/pulp_ansible.git 
 cd pulp_ansible/docs/_scripts/ 
 source base.sh 

 http POST $BASE_ADDR/pulp/api/v3/repositories/ name=foo 

 export REPO_HREF=$(http $BASE_ADDR/pulp/api/v3/repositories/ | \ 
   jq -r '.results[] | select(.name == 'on_demand' or policy "foo") | ._href') 

 http $BASE_ADDR$REPO_HREF 


 http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ \ 
     name='bar' \ 
     url='https://galaxy-dev.ansible.com' \ 
     whitelist='testing.ansible_testing_content' \ 
     policy='streamed' 

 export REMOTE_HREF=$(http $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ | jq -r '.results[] | select(.name == 'streamed' are also allowed. "bar") | ._href') 

 h3. Solution http $BASE_ADDR$REMOTE_HREF 

 1. Have core's Remote serializer (the Master one) fail validation if policy != 'immediate'. export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \ 
   | jq -r '.task') 

 wait_for_pulp $TASK_URL 
 2. Add docs on how plugins can adjust this validation in their Remote detail serializer to enable either export REPOVERSION_HREF=${CREATED_RESOURCE[0]} 

 http $BASE_ADDR$REPOVERSION_HREF 
 </pre> 

 outputs: 

 <pre> 
 root@localhost _scripts]# http POST $BASE_ADDR/pulp/api/v3/remotes/ansible/collection/ \ 
 >       name='bar' \ 
 >       url='https://galaxy-dev.ansible.com' \ 
 >       whitelist='testing.ansible_testing_content' \ 
 >       policy='streamed' and/or policy='on_deamand' 


 h3. Details 
 HTTP/1.1 201 Created 
 Allow: GET, POST, HEAD, OPTIONS 
 Connection: close 
 Content-Length: 482 
 Content-Type: application/json 
 Date: Wed, 12 Jun 2019 18:00:06 GMT 
 Location: /pulp/api/v3/remotes/ansible/collection/a1df15b4-5438-426a-9ca8-051fec031877/ 
 Server: gunicorn/19.9.0 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 We need to advertise to users that they can expect this here:    https://docs.pulpproject.org/en/3.0/nightly/workflows/lazy-downloading.html { 
     "_created": "2019-06-12T18:00:06.632076Z",  
     "_href": "/pulp/api/v3/remotes/ansible/collection/a1df15b4-5438-426a-9ca8-051fec031877/",  
     "_last_updated": "2019-06-12T18:00:06.632096Z",  
     "_type": "ansible.collection",  
     "download_concurrency": 20,  
     "name": "bar",  
     "policy": "streamed",  
     "proxy_url": null,  
     "ssl_ca_certificate": null,  
     "ssl_client_certificate": null,  
     "ssl_client_key": null,  
     "ssl_validation": true,  
     "url": "https://galaxy-dev.ansible.com",  
     "validate": true,  
     "whitelist": "testing.ansible_testing_content" 
 } 
 root@localhost _scripts]# http $BASE_ADDR$REMOTE_HREF 
 HTTP/1.1 200 OK 
 Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS 
 Connection: close 
 Content-Length: 482 
 Content-Type: application/json 
 Date: Wed, 12 Jun 2019 18:00:43 GMT 
 Server: gunicorn/19.9.0 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 We need to document to plugin writers what they should do here: https://docs.pulpproject.org/en/pulpcore-plugin/nightly/reference/lazy-support.html#lazy-support { 
     "_created": "2019-06-12T18:00:06.632076Z",  
     "_href": "/pulp/api/v3/remotes/ansible/collection/a1df15b4-5438-426a-9ca8-051fec031877/",  
     "_last_updated": "2019-06-12T18:00:06.632096Z",  
     "_type": "ansible.collection",  
     "download_concurrency": 20,  
     "name": "bar",  
     "policy": "streamed",  
     "proxy_url": null,  
     "ssl_ca_certificate": null,  
     "ssl_client_certificate": null,  
     "ssl_client_key": null,  
     "ssl_validation": true,  
     "url": "https://galaxy-dev.ansible.com",  
     "validate": true,  
     "whitelist": "testing.ansible_testing_content" 
 } 

 [root@localhost _scripts]# export TASK_URL=$(http POST $BASE_ADDR$REMOTE_HREF'sync/' repository=$REPO_HREF \ 
 >     | jq -r '.task') 
 [root@localhost _scripts]# wait_for_pulp $TASK_URL 
 [root@localhost _scripts]# export REPOVERSION_HREF=${CREATED_RESOURCE[0]} 
 [root@localhost _scripts]# http $BASE_ADDR$REPOVERSION_HREF 
 HTTP/1.1 200 OK 
 Allow: GET, DELETE, HEAD, OPTIONS 
 Connection: close 
 Content-Length: 575 
 Content-Type: application/json 
 Date: Wed, 12 Jun 2019 18:01:50 GMT 
 Server: gunicorn/19.9.0 
 Vary: Accept, Cookie 
 X-Frame-Options: SAMEORIGIN 

 { 
     "_created": "2019-06-12T18:01:35.431134Z",  
     "_href": "/pulp/api/v3/repositories/63cf03b0-109d-4326-b7c9-6f86614ada6d/versions/1/",  
     "base_version": null,  
     "content_summary": { 
         "added": { 
             "ansible.collection": { 
                 "count": 1,  
                 "href": "/pulp/api/v3/content/ansible/collections/?repository_version_added=/pulp/api/v3/repositories/63cf03b0-109d-4326-b7c9-6f86614ada6d/versions/1/" 
             } 
         },  
         "present": { 
             "ansible.collection": { 
                 "count": 1,  
                 "href": "/pulp/api/v3/content/ansible/collections/?repository_version=/pulp/api/v3/repositories/63cf03b0-109d-4326-b7c9-6f86614ada6d/versions/1/" 
             } 
         },  
         "removed": {} 
     },  
     "number": 1 
 } 

 </pre> 

 Pulp installation from source - June/18 
 <pre> 
 (pulp) [root@localhost pulp]# pip list | grep pulp 
 pulp-ansible                      0.2.0b1.dev0  
 pulp-certguard                    0.1.0rc1      
 pulp-docker                       4.0.0b5.dev0  
 pulp-file                         0.1.0b1.dev0  
 pulp-rpm                          3.0.0b4.dev0  
 pulpcore                          3.0.0rc3.dev0 
 pulpcore-plugin                   0.1.0rc3.dev0 

 </pre>

Back