Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #3586

closed

Filtering workers is indeterministic

Added by dkliban@redhat.com over 6 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
dalley
Category:
-
Sprint/Milestone:
3.0.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 38
Quarter:

Description

pulp-smash test to filter workers is failing on python 3.5 from time to time.

It looks like when you filter by either 'online' or 'missing' AND 'name' you can get 2 results when only 1 worker should be returned when filtering by name.

Actions
Copy link
 #1

Updated by dkliban@redhat.com over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by dalley over 6 years ago

It fails like 90% of the time, but it does work like 10% of the time on Python 3.5.

It works (seemingly) 100% of the time on Python 3.6

Actions
Copy link
 #3

Updated by daviddavis over 6 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 36

I think the next step is to try to reproduce this locally and debug. I tried with python 3.5.4 and couldn't reproduce though. I think Travis is using python 3.5.5 so maybe that's why.

Actions
Copy link
 #4

Updated by jortel@redhat.com over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
Actions
Copy link
 #6

Updated by rchan over 6 years ago

  • Sprint changed from Sprint 36 to Sprint 37
Actions
Copy link
 #7

Updated by jortel@redhat.com over 6 years ago

Unable to reproduce so far using:

  • django-crispy-forms (1.7.2)
  • django-extensions (2.0.7)
  • django-filter (1.1.0)
  • djangorestframework (3.8.2)
  • djangorestframework-jwt (1.11.0)
Actions
Copy link
 #8

Updated by dalley over 6 years ago

  • Assignee changed from jortel@redhat.com to dalley
  • Tags Pulp 3 added
Actions
Copy link
 #9

Updated by dalley over 6 years ago

Reproduced:

CPython/3.5.2
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:34:23.902196Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/d5096dd8-8126-4bb6-917d-7a4ffe9932d9/",
            "id": "d5096dd8-8126-4bb6-917d-7a4ffe9932d9",
            "last_heartbeat": "2018-05-15T22:23:19.829143Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:20.987260Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/267e7528-43c4-4e98-aa43-5e39e72a94c3/",
            "id": "267e7528-43c4-4e98-aa43-5e39e72a94c3",
            "last_heartbeat": "2018-05-15T22:23:19.825483Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/454f6b10-e4bc-46bd-9102-d5b2384c4be1/",
            "id": "454f6b10-e4bc-46bd-9102-d5b2384c4be1",
            "last_heartbeat": "2018-05-15T22:23:51.110808Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

CPython/3.6.5
=============

pip list | grep -i django
-------------------------
Django                   2.0.5
django-crispy-forms      1.7.2
django-extensions        2.0.7
django-filter            1.1.0
djangorestframework      3.8.2

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39/",
            "id": "bbc7abd1-7c5e-49fa-9231-2e9a42d5bf39",
            "last_heartbeat": "2018-05-15T23:43:48.028704Z",
            "missing": false,
            "name": "resource_manager@pulp3.dev",
            "online": false
        },
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:44:12.789671Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?online=True"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/db161f9d-b67a-444e-8280-f8dfd5a0dc97/",
            "id": "db161f9d-b67a-444e-8280-f8dfd5a0dc97",
            "last_heartbeat": "2018-05-15T23:46:28.543678Z",
            "missing": false,
            "name": "reserved_resource_worker_1@pulp3.dev",
            "online": true
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev"

{
    "next": null,
    "previous": null,
    "results": [
        {
            "_href": "http://localhost:8000/pulp/api/v3/workers/4be920ab-2458-4955-a54c-bfe673f94d1a/",
            "id": "4be920ab-2458-4955-a54c-bfe673f94d1a",
            "last_heartbeat": "2018-05-15T23:43:48.026657Z",
            "missing": false,
            "name": "reserved_resource_worker_2@pulp3.dev",
            "online": false
        }
    ]
}

(pulp) [vagrant@pulp3 pulp]$ http GET ":8000/pulp/api/v3/workers/?name=reserved_resource_worker_2@pulp3.dev&online=True"

{
    "next": null,
    "previous": null,
    "results": []
}

Added by dalley over 6 years ago

Revision d62bf5a2 | View on GitHub

Fix indeterministic worker filtering

closes #3586 https://pulp.plan.io/issues/3586

Added by dalley over 6 years ago

Revision d62bf5a2 | View on GitHub

Fix indeterministic worker filtering

closes #3586 https://pulp.plan.io/issues/3586

Actions
Copy link
 #10

Updated by dalley over 6 years ago

  • Status changed from ASSIGNED to POST

PR: https://github.com/pulp/pulp/pull/3500

Intersection limits what you can do in further queries. If the intersection happens first, the name filter doesn't get applied.

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:24
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:52:57.894782+00:00))
[22/May/2018 15:53:27] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 290
^C(pulp) [vagrant@pulp3 pulp]$ python manage.py runserver 0.0.0.0:8000
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
/home/vagrant/.virtualenvs/pulp/lib64/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Performing system checks...

System check identified no issues (0 silenced).
May 22, 2018 - 15:53:35
Django version 2.0.5, using settings 'pulpcore.app.settings'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker") INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:53:19.456311+00:00))
[22/May/2018 15:53:49] "GET /pulp/api/v3/workers/?name=resource_manager@pulp3.dev&online=True HTTP/1.1" 200 808

In [11]: fmt(Worker.objects.filter(name='resource_manager@pulp3.dev').intersection(Worker.objects.online_workers()))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:41:50.722037+00:00))

In [12]: fmt(Worker.objects.online_workers().intersection(Worker.objects.filter(name='resource_manager@pulp3.dev')))

  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE ("pulp_app_worker"."gracefully_stopped" = FALSE
          AND "pulp_app_worker"."last_heartbeat" >= 2018-05-22 15:42:25.810284+00:00)) INTERSECT
  (SELECT "pulp_app_worker"."id",
          "pulp_app_worker"."created",
          "pulp_app_worker"."last_updated",
          "pulp_app_worker"."name",
          "pulp_app_worker"."last_heartbeat",
          "pulp_app_worker"."gracefully_stopped",
          "pulp_app_worker"."cleaned_up"
   FROM "pulp_app_worker"
   WHERE "pulp_app_worker"."name" = resource_manager@pulp3.dev)
Actions
Copy link
 #11

Updated by dalley over 6 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #12

Updated by bizhang over 6 years ago

  • Status changed from MODIFIED to ASSIGNED

It looks like this issue is still popping up in the current travis tests

Actions
Copy link
 #13

Updated by Ichimonji10 over 6 years ago

  • Subject changed from Filtering workers is indetermanistic to Filtering workers is indeterministic
Actions
Copy link
 #14

Updated by rchan over 6 years ago

  • Sprint changed from Sprint 37 to Sprint 38
Actions
Copy link
 #15

Updated by dalley over 6 years ago

  • Status changed from ASSIGNED to MODIFIED

I can't reproduce this anymore locally, despite an awful lot of trying. Going to close it out and re-enable the tests.

Actions
Copy link
 #16

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions
Copy link
 #17

Updated by bmbouter about 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF