Project

Profile

Help

Issue #1565

Pulp does not publish more than once

Added by dkliban@redhat.com over 5 years ago. Updated about 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.8.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

The simplest way to see this is to create a repository, publish it, and then sync it (with auto publish).

repo.json

{"id":"test-repo","display_name":"test-repo","importer_type_id":"yum_importer","importer_config":{"feed":"https://repos.fedorapeople.org/pulp/pulp/demo_repos/zoo/","ssl_ca_cert":null,"ssl_client_cert":null,"ssl_client_key":null},"notes":{"_repo-type":"rpm-repo"},"distributors":[{"distributor_type_id":"yum_distributor","distributor_config":{"relative_url":"test-repo","http":true,"https":true,"protected":true},"auto_publish":true,"distributor_id":"test-repo"},{"distributor_type_id":"export_distributor","distributor_config":{"http":false,"https":false},"auto_publish":false,"distributor_id":"export_distributor"}]}

publish.json

{"id":"test-repo","override_config":null}

sync.json

{"override_config":{"num_threads":4,"validate":true}}

Steps to reproduce:


$ http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ < repo.json

$ http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/test-repo/actions/publish/ < publish.json

$ http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/test-repo/actions/sync/ < sync.json

Associated revisions

Revision 9f0c6ecb View on GitHub
Added by dkliban@redhat.com over 5 years ago

Adds validation to the UTCDateTimeField for mongoengine

Due to a bug in datetime utils, we can't rely on mongonegine to convert a string representation of a timestamp with timezone represented as Z on end. As a result we now require the date to be supplied as a datetime object with tzinfo set.

https://pulp.plan.io/issues/1565 closes #1565

Revision 9f0c6ecb View on GitHub
Added by dkliban@redhat.com over 5 years ago

Adds validation to the UTCDateTimeField for mongoengine

Due to a bug in datetime utils, we can't rely on mongonegine to convert a string representation of a timestamp with timezone represented as Z on end. As a result we now require the date to be supplied as a datetime object with tzinfo set.

https://pulp.plan.io/issues/1565 closes #1565

History

#1 Updated by dkliban@redhat.com over 5 years ago

  • Description updated (diff)

#2 Updated by dkliban@redhat.com over 5 years ago

  • Description updated (diff)

#3 Updated by jortel@redhat.com over 5 years ago

  • Assignee set to dkliban@redhat.com
  • Priority changed from Normal to High
  • Triaged changed from No to Yes

#4 Updated by dkliban@redhat.com over 5 years ago

Pulp seems to expect that the date on the machine is in UTC timezone. I was able to resolve this problem by changing the timezone from EST to UTC.

sudo rm /etc/localtime
sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

I am still determining which date in Pulp codebase is wrong.

#5 Updated by dkliban@redhat.com over 5 years ago

The problem stems from a bug[0] in the dateutils.parser.parse(). This method is used by mongoengine to parse a string to create a python datetime object. It seems that parse() does not recognize the timezone part of the string and makes the assumption that it is in localtime. This works fine when a machine has it's local time set to UTC. When timezone on a machine is anything other than UTC, handing a UTC date to parse() causes the date to be converted to UTC from local time.

In this case[1], when we are saving the last published date, we first generate it as a string that looks like "2016-01-24T20:46:25Z". We then assign the value to 'last_publish' field of the Distributor object. During the save() of the distributor, the parse() method is used to convert the string to a datetime object. The Z part of the timezone is interpreted as being local time instead of UTC. So the database ends up storing a date that's 5 hours in the future (in the EST case).

[0] https://github.com/dateutil/dateutil/issues/70
[1] https://github.com/pulp/pulp/blob/master/server/pulp/server/controllers/repository.py#L1000-L1002

#6 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from NEW to POST

#7 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#8 Updated by rbarlow over 5 years ago

  • Status changed from MODIFIED to 5

#9 Updated by pthomas@redhat.com over 5 years ago

  • Status changed from 5 to 6

verified

root@mgmt5 ~]#
[root@mgmt5 ~]# http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/ < repo.json
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
HTTP/1.1 201 CREATED
Connection: Keep-Alive
Content-Length: 313
Content-Type: application/json; charset=utf-8
Date: Mon, 14 Mar 2016 21:24:47 GMT
Keep-Alive: timeout=5, max=10000
Location: https://localhost/pulp/api/v2/repositories/test-repo/
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5

{
"_href": "/pulp/api/v2/repositories/test-repo/",
"_id": {
"$oid": "56e72c1fbc1dbc60953fd076"
},
"_ns": "repos",
"content_unit_counts": {},
"description": null,
"display_name": "test-repo",
"id": "test-repo",
"last_unit_added": null,
"last_unit_removed": null,
"notes": {
"_repo-type": "rpm-repo"
},
"scratchpad": {}
}

[root@mgmt5 ~]# http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/test-repo/actions/publish/ < publish.json
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
HTTP/1.1 202 ACCEPTED
Connection: Keep-Alive
Content-Length: 172
Content-Type: application/json; charset=utf-8
Date: Mon, 14 Mar 2016 21:24:52 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5

{
"error": null,
"result": null,
"spawned_tasks": [
{
"_href": "/pulp/api/v2/tasks/54f41813-b4bb-41c7-8133-9ac2e576935c/",
"task_id": "54f41813-b4bb-41c7-8133-9ac2e576935c"
}
]
}

[root@mgmt5 ~]# http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/test-repo/actions/sync/ < sync.json
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
HTTP/1.1 202 ACCEPTED
Connection: Keep-Alive
Content-Length: 172
Content-Type: application/json; charset=utf-8
Date: Mon, 14 Mar 2016 21:25:04 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5

{
"error": null,
"result": null,
"spawned_tasks": [
{
"_href": "/pulp/api/v2/tasks/0aa72df6-5719-436b-a0da-bdfb4bda2d53/",
"task_id": "0aa72df6-5719-436b-a0da-bdfb4bda2d53"
}
]
}

#10 Updated by dkliban@redhat.com over 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF