Project

Profile

Help

Issue #3586

Filtering workers is indeterministic

Added by dkliban@redhat.com about 2 years ago. Updated 6 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 38

Description

pulp-smash test to filter workers is failing on python 3.5 from time to time.

It looks like when you filter by either 'online' or 'missing' AND 'name' you can get 2 results when only 1 worker should be returned when filtering by name.

Associated revisions

Revision d62bf5a2 View on GitHub
Added by dalley about 2 years ago

Fix indeterministic worker filtering

closes #3586 https://pulp.plan.io/issues/3586

Revision d62bf5a2 View on GitHub
Added by dalley about 2 years ago

Fix indeterministic worker filtering

closes #3586 https://pulp.plan.io/issues/3586

History

#1 Updated by dkliban@redhat.com about 2 years ago

  • Description updated (diff)

#2 Updated by dalley about 2 years ago

It fails like 90% of the time, but it does work like 10% of the time on Python 3.5.

It works (seemingly) 100% of the time on Python 3.6

#3 Updated by daviddavis about 2 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 36

I think the next step is to try to reproduce this locally and debug. I tried with python 3.5.4 and couldn't reproduce though. I think Travis is using python 3.5.5 so maybe that's why.

#4 Updated by jortel@redhat.com about 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

#6 Updated by rchan about 2 years ago

  • Sprint changed from Sprint 36 to Sprint 37

#7 Updated by jortel@redhat.com about 2 years ago

Unable to reproduce so far using:

  • django-crispy-forms (1.7.2)
  • django-extensions (2.0.7)
  • django-filter (1.1.0)
  • djangorestframework (3.8.2)
  • djangorestframework-jwt (1.11.0)

#8 Updated by dalley about 2 years ago

  • Assignee changed from jortel@redhat.com to dalley
  • Tags Pulp 3 added

#9 Updated by dalley about 2 years ago

Reproduced:

CPython/3.5.2
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:34:23.902196Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/d5096dd8-8126-4bb6-917d-7a4ffe9932d9/",
            "id": "d5096dd8-8126-4bb6-917d-7a4ffe9932d9",
            "last_heartbeat": "2018-05-15T22:23:19.829143Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:20.987260Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:51.110808Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

CPython/3.6.5
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39/",
            "id": "bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39",
            "last_heartbeat": "2018-05-15T23:43:48.028704Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:44:12.789671Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:46:28.543678Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"

{
    "next": null,
    "previous": null,
    "results": []
}

#10 Updated by dalley about 2 years ago

  • Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulp/pull/3500

Intersection limits what you can do in further queries. If the intersection happens first, the name filter doesn't get applied.

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:24
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:52:57.894782+00:00))
[22/May/2018 15:53:27] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 290
^C(pulp) [vagrant@pulp3 pulp]$ python manage.py runserver 0.0.0.0:8000
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Performing system checks...

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:35
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker") INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:53:19.456311+00:00))
[22/May/2018 15:53:49] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 808

In [11]: fmt(Worker.objects.filter(name='resource_manager@pulp3.dev').intersection(Worker.objects.online_workers()))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:41:50.722037+00:00))

In [12]: fmt(Worker.objects.online_workers().intersection(Worker.objects.filter(name='resource_manager@pulp3.dev')))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:42:25.810284+00:00)) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev)

#11 Updated by dalley about 2 years ago

  • Status changed from POST to MODIFIED

#12 Updated by bizhang about 2 years ago

  • Status changed from MODIFIED to ASSIGNED

It looks like this issue is still popping up in the current travis tests

#13 Updated by Ichimonji10 almost 2 years ago

  • Subject changed from Filtering workers is indetermanistic to Filtering workers is indeterministic

#14 Updated by rchan almost 2 years ago

  • Sprint changed from Sprint 37 to Sprint 38

#15 Updated by dalley almost 2 years ago

  • Status changed from ASSIGNED to MODIFIED

I can't reproduce this anymore locally, despite an awful lot of trying. Going to close it out and re-enable the tests.

#16 Updated by bmbouter about 1 year ago

  • Tags deleted (Pulp 3)

#17 Updated by bmbouter 6 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF