Project

Profile

Help

Issue #6496

closed

Created Resources created and removed later are displayed as null in the task

Added by dalley about 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 76
Quarter:

Description

Create a distribution and delete it. Poll on the task that have created the distribution

$ http POST $BASE_ADDR/pulp/api/v3/distributions/rpm/rpm/     publication=/pulp/api/v3/publications/rpm/rpm/2f46a52c-bb47-4d15-a208-fec84107fa19/ name=centos71 base_path=centos71
HTTP/1.1 202 Accepted
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:57:04 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/3c6697d1-f54e-4338-b0eb-f6db91cf7df1/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ http GET $BASE_ADDR/pulp/api/v3/tasks/3c6697d1-f54e-4338-b0eb-f6db91cf7df1/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 583
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:57:12 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [
        "/pulp/api/v3/distributions/rpm/rpm/a587fe48-c0aa-40ba-8017-153cf2e3ac96/"
    ],
    "error": null,
    "finished_at": "2020-04-21T15:57:05.243262Z",
    "name": "pulpcore.app.tasks.base.general_create",
    "parent_task": null,
    "progress_reports": [],
    "pulp_created": "2020-04-21T15:57:04.651024Z",
    "pulp_href": "/pulp/api/v3/tasks/3c6697d1-f54e-4338-b0eb-f6db91cf7df1/",
    "reserved_resources_record": [
        "/api/v3/distributions/"
    ],
    "started_at": "2020-04-21T15:57:04.799416Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/972ff6d6-7b2d-458d-b507-d4e065262760/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ http DELETE $BASE_ADDR/pulp/api/v3/distributions/rpm/rpm/a587fe48-c0aa-40ba-8017-153cf2e3ac96/
HTTP/1.1 202 Accepted
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:57:36 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/a2530273-17f9-41a7-839e-158329300951/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp_rpm]$ http GET $BASE_ADDR/pulp/api/v3/tasks/3c6697d1-f54e-4338-b0eb-f6db91cf7df1/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 513
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:57:39 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [
        null
    ],
    "error": null,
    "finished_at": "2020-04-21T15:57:05.243262Z",
    "name": "pulpcore.app.tasks.base.general_create",
    "parent_task": null,
    "progress_reports": [],
    "pulp_created": "2020-04-21T15:57:04.651024Z",
    "pulp_href": "/pulp/api/v3/tasks/3c6697d1-f54e-4338-b0eb-f6db91cf7df1/",
    "reserved_resources_record": [
        "/api/v3/distributions/"
    ],
    "started_at": "2020-04-21T15:57:04.799416Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/972ff6d6-7b2d-458d-b507-d4e065262760/"
}



Actions #1

Updated by ttereshc about 4 years ago

  • Triaged changed from No to Yes
Actions #2

Updated by ipanova@redhat.com about 4 years ago

  • Project changed from Migration Plugin to Pulp
  • Subject changed from Created Resources created by the migration plugin don't serialize properly to Created Resources created and removed later are displayed as null in the task
  • Description updated (diff)
Actions #3

Updated by ipanova@redhat.com about 4 years ago

  • Triaged changed from Yes to No
Actions #4

Updated by fao89 almost 4 years ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 71
Actions #5

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 71 to Sprint 72
Actions #6

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 72 to Sprint 73
Actions #7

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 73 to Sprint 74
Actions #8

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 74 to Sprint 75
Actions #9

Updated by rchan almost 4 years ago

  • Sprint changed from Sprint 75 to Sprint 76
Actions #10

Updated by ggainey almost 4 years ago

What is the expected behavior here? The created-resource is gone, what should the task-record show?

Actions #11

Updated by dalley almost 4 years ago

ggainey they should disappear from the list, instead of appearing as "null" in the list.

Actions #12

Updated by ggainey almost 4 years ago

The underlying issue here is that there is no direct foreign-key relationship between a created-resource and core_createdresource table. This means that when you delete something "pointed to" by CreatedResource, the entry in that table still exists (because the entity-being-deleted doesn't know it's being pointed-to).

Rather than return 'null' when we can't find the Thing-Pointed-To - would it make sense to return a string like <deleted resource>?

Actions #13

Updated by ggainey almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ggainey
Actions #14

Updated by dalley almost 4 years ago

I'm not sure. That's slightly better for humans, probably, but might be worse for bindings. And it still seems sub-optimal (maybe that's the best we can do).

Actions #15

Updated by dalley almost 4 years ago

Part of me thinks this entire feature was designed backwards, and that it should have been done more like the repository version endpoint where it returns a map of queries to objects. But changing that now would be backwards incompatible.

Actions #16

Updated by ggainey almost 4 years ago

dalley wrote:

Part of me thinks this entire feature was designed backwards, and that it should have been done more like the repository version endpoint where it returns a map of queries to objects. But changing that now would be backwards incompatible.

Def too late - CreatedResource works the way it does, at least for 3.x.

I'm not sure. That's slightly better for humans, probably, but might be worse for bindings. And it still seems sub-optimal (maybe that's the best we can do).

Returning an array with a null entry is certainly better than a human-readable string (ie, leave things as they are now).

What use-case does this impact? The task ran, Long Ago, and did its thing. At some point since, one (or more) of the resources created by that task are deleted. Now, we're looking at the task - to do what?

If this is just "it looks odd" - then we're talking about people, and making it a little clearer in text is useful. If we expect code to be involved, it doesn't care about what it looks like, and 'null' makes way more sense.

Actions #17

Updated by dalley almost 4 years ago

I believe the issue is that re-migrations delete a bunch of things, so the created resources on old migration tasks get wiped out constantly.

Actions #18

Updated by ggainey almost 4 years ago

dalley wrote:

I believe the issue is that re-migrations delete a bunch of things, so the created resources on old migration tasks get wiped out constantly.

That says to me this is for humans, and "deleted resource" is as good as it gets. It's visible as long as the old task is still around (should we think about deleting old-migration-tasks when there is a new one? Not sure I'd suggest that, these are useful audit trails). At the end of the day, the resources pointed to by this task were created, and are now gone. As long as the task is around, it feels like it should record that it did in fact do a thing, even if the results are no longer with us.

I am leaning towards "make the change to deleted-resource" - but that might just be because I have a PR ready to push :)

Actions #19

Updated by ggainey almost 4 years ago

  • Status changed from ASSIGNED to CLOSED - WONTFIX

Based on discussions in IRC, we're going to leave the current behavior as-is and close this WONTFIX. Adding the discussion for posterity:

<dalley> dkliban, as resident binding expert, could you weigh in on this idea? https://pulp.plan.io/issues/6496#note-18
<dalley> will that cause problems?
<dkliban> i don't quite follow
<dkliban> is the suggestion to add a field that's called deleted-resources?
<dkliban> or is the suggestion to remove old tasks?
<dalley> right now the deleted "created resources" become "null"
<dkliban> yep
<dalley> ggainey is saying its difficult to actually get rid of them, but we could replace "null" with "deleted resource"
<dkliban> oh
<ggainey> dkliban: yeah - the prob is when the task lives, but the thing(s) it created are deleted
<dkliban> would we use a database trigger or soemthing?
<ggainey> and right now, we just say 'null' for a missing resource
<ggainey> dkliban: it would have to be on every table
<dkliban> dalley: it wouldn't really be a problem ... the created resources field is a list of strings
<dalley> I don't think we should be messing w/ that, especially in combination with generic foreign key where it's all quite abstracted to begin with
<ggainey> dkliban: the change I had in mind is output-only - the table and model stay as they are, the task-output just says "<deleted resource>" where it currently says "null"
<dkliban> ggainey: cool 
<dkliban> that makes more sense than what i was saying
<dkliban> the bindings should have no issue with this
<ggainey> dalley: dkliban : tell you what - I'm'a submit a PR, so we have the conrete code in front of us, and if we decide to do someting else (or nothing!), I can just close it
<dalley> does katello try to do anything w/ created_resources that needs to take this into account?
<dkliban> i don't know
<ggainey> same
<ggainey> it's def an obscure-ish edge case
<ggainey> (which migration steps into all the time, of course :) )
<jsherrill> we do record created resources from some pulp tasks
<jsherrill> like after creating a new version
<dkliban> ggainey: earlier today in #pulp, wibbit was asking about such a scenario. however, it was not about the migration plugin but the repository version 'base_version' field
<jsherrill> so we could add some logic to check for nil values rather than an empty array
<ggainey> dkliban: yeah, I took a stab at answering him, but it was pre-coffee, yours was better
<dalley> it's definitely easier to process null than a random string
<jsherrill> yeah, i'd rather not have a random string
<ggainey> jsherrill: yeah, that's the crux of it - right now, you get an array which may have null entries, which is not create for humans reading it but better for code to rely on
<jsherrill> especially when an href is also a string
<ggainey> yeah
<jsherrill> its not possible to remove those nil's from the response?
<jsherrill> or is that not desired
<ggainey> ^not create^not great
<ggainey> jsherrill: that's exactly the problem
<dalley> I kinda think we should just leave it as-is until we can fix this API in an eventual Pulp 4
<ggainey> the CreatedResource modle-entries *still exist* - it's the things they point to that are gone
<ggainey> dalley: I am def leaning that way myself, aye
<jsherrill> ggainey: yeah i guess i'm asking if its not possible to easily strip/ignore ones in that situation
<jsherrill> and its fine if its not, i don't know the code ;)
<ggainey> jsherrill: as it turns out, the answer is "argh" :)
<jsherrill> hehe
<ggainey> I spent some time inside serializers trying to figure out the right thing, and if there is a way, it's def nonobvious
<jsherrill> gotcha
<ggainey> the conversation is essentially "Hey, how many CreatedResource entities do we have here?" "One." "Great! Give me the Thing it points to!"  "It's gone."  <sadness ensues>
<jsherrill> well nil is def. preferred to random string from me!
<ggainey> ok
<ggainey> dalley: close this issue as WONTFIX?
<ggainey> I mean, I will, I'm asking if you're ok w/that
<dalley> ggainey, if there's no way to filter out the dead entries, then yes I think so
<ggainey> not that I could find, the CreatedResource serializer is called per-row, and whether it returns something or not, "an entry" is going to show up in the many-to-many list because the actual model does in fact have a row.
<ggainey> I can/we do change what the row displays-as, but it's too late to hide that it exists
<ggainey> dalley: anyway - I will close as WONTFIX. I think I'll throw this irc discussion in there as well, so someone else doesn't have to go thru the same confusion I did trying to figure out what was up

Also available in: Atom PDF