Project

Profile

Help

Actions
Send by e-mail Copy link

Story #2059

closed

As a user, I want pulp-manage-db to show a basic progress indicator with each migration, so I can see that the migration is in progress and not hung up

Added by ashbyj@imsweb.com almost 8 years ago. Updated over 3 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

When upgrading from pulp 2.7 to 2.8, DB migrations took much longer than expected. The pulp-manage-db script appeared to hang on a few migrations, but in fact these were "known big" migrations so they finally completed successfully after I let it continue.

One suggestion is to have pulp-manage-db output dots as a simple progress or "still working" indicator. So you'll see a dot "." every second or a dot for say, every item processed in that migration. The progress indicator should reset at each migration.

This is a simple solution that will let users know that the script is still doing some sort of work and that the script or their ssh session hasn't totally crashed and burned.

Background from the mailing list:

-----Original Message-----

From: Brian Bouterse []
Sent: Tuesday, July 05, 2016 11:27 AM
To: Ashby, Jason (IMS) <>
Subject: Re: [Pulp-list] Long upgrade times from 2.6 -> 2.8

Jason, these are great ideas. Would you want to file each of them as an issue [0]? One being the printing of warnings about specific ones taking a long time and the other being periodic output of '.' to show progress?

[0]: https://pulp.plan.io/projects/pulp/issues/new

-Brian

On 07/03/2016 08:25 AM, Ashby, Jason (IMS) wrote:

Yeah our content is on NFS mounts backed by regular non-SSD disks, so
IO is pretty crappy. But its cheap and perfect for serving yum content.

My recommendation would be having pulp-manage-db spit out a message
before each big migration that says “this next migration is a big deal!”
that would be a nice fyi. And perhaps a URL to see migration details.
This doesn’t prep people ahead of time to schedule downtime, but at
least keeps them from thinking something is wrong.

From:*
[] *On Behalf Of *Michael Hrivnak
*Sent: Friday, July 01, 2016 4:19 PM
To: Eric Helms <>
Cc: pulp-list <>
Subject: Re: [Pulp-list] Long upgrade times from 2.6 -> 2.8

I expect the performance will be highly dependent on disk access speed.
Latency is far more important than throughput for those two migrations.
Deployments using NFS for example may benefit from finding ways to
reduce latency during the migration; there may be mount options that
can be adjusted only while the migration runs, or perhaps the code
itself can be run on a machine closer to the data than where pulp usually runs.

Michael

On Fri, Jul 1, 2016 at 11:53 AM, Eric Helms <
<>> wrote:

My main worry or first check I wanted to put to rest was that it was
an actual issue. Given that everything is sane, I think warning
users that upgrades will scale with the amount of content and giving
them some rough estimates based on other users data is the best
option. That way they can at least prepare themselves with knowledge
for planning their upgrade outages.

Thanks for looking into this so quickly,

Eric

On Fri, Jul 1, 2016 at 11:50 AM, Brian Bouterse <
<>> wrote:

With 2.8 specifically there is a lot of work done by the
migrations and the runtime should scale with the content size.
In other words I don't think it's doing useless work, there is
just a lot of it.

I just did a quick audit of the two major offending migrations
and they are implemented sanely so I don't think there is a
quick fix to improve the runtime.

I believe to get a shorter runtime we would have to parallelize
the migrations.

Regarding things going wrong, we do assume that migrations
should be re-entrant meaning that if for some reason something
strange occurs it should always be safe to run pulp-manage-db
and it will pick up where it left off. Just a behavioral FYI.

These migration with 2.8 should not be norm, so I don't expect
this to be very common in the future until maybe a major release
(3.0).

Given all ^, what do you think we should do?

-Brian

On 07/01/2016 11:23 AM, Eric Helms wrote:

I think there are a couple of considerations.

1) The first issue is that a 6-18 hour upgrade window is
not something
users expect and we've not been warning them of such so they
can plan an
outage accordingly. Lengthy upgrades also have that tendency
to make
users feel something is wrong or increase the risk that
something can go
wrong in between.
2) The fundamental question of - is this a bug or does this
make
perfect sense and how it has to work?
3) Applying the upgrade on an existing 2.6 if it changed
nothing of the
environment could work, the tough part is having to
distribute that
backwards. Pulp would have to distribute it back to 2.6, and
Katello
would have to push out patches to our 2.4 release channel.

Eric

On Fri, Jul 1, 2016 at 11:14 AM, Brian Bouterse
< <>
< <>>>
wrote:

I'm trying to understand if the pain point is related to
downtime or
total runtime.

For instance, what if these migration could be run as a
pre-migration step, ahead of time while Pulp was still
online? The
upgrade would still take just as long but you could use
your (in
this case) 2.6 install normally while the migrations are
applying.
Once they are done then the actual upgrade of the
codebase could be
very short.

-Brian

On 07/01/2016 09:20 AM, Eric Helms wrote:

On Fri, Jul 1, 2016 at 8:52 AM, Ashby, Jason (IMS)
< <>
< <>>
< <>
< <>>>> wrote:

FWIW I just upgraded from 2.7 -> 2.8 and it was
approx. 1-2 hr
upgrade to get through the migrations in
pulp-manage-db.____

_ _

290 GB /var/lib/pulp____

16 GB MongoDB____

_ _

From:*
<>
<
<>>
<
<>
<
<>>>
[
<>
<
<>>
<
<>
<
<>>>] *On Behalf Of *Michael
Hrivnak
*Sent: Friday, July 01, 2016 8:31 AM
To: Eric Helms <
<>
<
<>> <
<>
< <>>>>
Cc: pulp-list <
<>
<
<>> <
<>
<
<>>>>
Subject: Re: [Pulp-list] Long upgrade times
from 2.6 ->
2.8____

_ _

Did you get any feedback on whether one
particular migration
seemed
to be running for a lot of that time?

For the 1.5TB/100GB MongoDB scenario here is what I
am able to glean
from user logs (which I can share privately with
anyone debugging):

~5 hours: Applying pulp_puppet.plugins.migrations
version 4
~10 hours: Applying pulp_rpm.plugins.migrations
version 28

Use reports "lots of stating, unlinking, and linking
of all the
symlinks
in /var/lib/pulb" if that helps.

Another user reports ~6 hours on 176G of data.

Eric

__

_ _

Michael____

_ _

On Fri, Jul 1, 2016 at 7:23 AM, Eric Helms
< <>
< <>>
<
<> <
<>>>>
wrote:____

Howdy,____

_ _

We (Katello) have had users reporting
incredibly long
upgrade
times when upgrading from 2.6 to 2.8. This
occurs during the
pulp-manage-db step that is run as the
beginning of our
installers upgrade process. Based on the
numbers below, does
this make sense at all?____

_ _

Some numbers:____

_ _

18 hour upgrade____

1.5 TB /var/lib/pulp____

100GB MongoDB____

_ _

_ _

Thanks,____

Eric____

_**________________________________________
Pulp-list mailing list

<> <
<>>
<
<> <
<>>>

https://www.redhat.com/mailman/listinfo/pulp-list____

_ _

----------------------------------------------------------------------
--

Information in this e-mail may be confidential.
It is
intended only
for the addressee(s) identified above. If you
are not the
addressee(s), or an employee or agent of the
addressee(s),
please
note that any dissemination, distribution, or
copying of this
communication is strictly prohibited. If you
have received this
e-mail in error, please notify the sender of the
error.

_**________________________________________
Pulp-list mailing list
<>
< <>>
https://www.redhat.com/mailman/listinfo/pulp-list

_**________________________________________
Pulp-list mailing list
<>
< <>>
https://www.redhat.com/mailman/listinfo/pulp-list

_**________________________________________
Pulp-list mailing list
<>
https://www.redhat.com/mailman/listinfo/pulp-list

----------------------------------------------------------------------
--

Information in this e-mail may be confidential. It is intended only
for the addressee(s) identified above. If you are not the
addressee(s), or an employee or agent of the addressee(s), please note
that any dissemination, distribution, or copying of this communication
is strictly prohibited. If you have received this e-mail in error,
please notify the sender of the error.

_**________________________________________
Pulp-list mailing list

https://www.redhat.com/mailman/listinfo/pulp-list

Actions
Copy link
 #1

Updated by bmbouter almost 8 years ago

After some discussion there are two additional points that should be included with this bugfix:

1. It should be just for these really long migrations. We may want a more general mechanism elsewhere, but for now we should fix the immediate problem.
2. In addition to stdout, we should send steady, update statements to logs for monitoring there. Katello runs pulp-manage-db for it's user's and doesn't show the stdout.

Actions
Copy link
 #2

Updated by dkliban@redhat.com almost 8 years ago

  • Tracker changed from Issue to Story

Changing this to a story because this is a feature request.

Actions
Copy link
 #4

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Copy link
 #5

Updated by dkliban@redhat.com almost 5 years ago

This is not going to be addressed in Pulp 2.

Actions
Copy link
 #6

Updated by dalley over 3 years ago

  • Status changed from NEW to CLOSED - WONTFIX
Actions
Send by e-mail Copy link

Also available in: Atom PDF