Pulp

I was not able to reproduce when upgrading from 2.8.7 to 2.10.2.

why do you think that symlinks are wrong?
storage path is different if unit was synced or uploaded

ll /var/lib/pulp/published/https/isos/iso-test/
total 20
-rw-r--r--. 1 apache apache 350 Nov 30 16:13 PULP_MANIFEST
lrwxrwxrwx. 1 apache apache  97 Nov 30 16:13 python-nectar-1.3.1.tar.gz -> /var/lib/pulp/content/units/iso/30/d7df0dea57549d23720b2af4a399ca8802b0af1f2240742f00fc9fe7c4385f  -----------> uploaded 

lrwxrwxrwx. 1 apache apache 106 Nov 30 16:13 test.iso -> /var/lib/pulp/content/units/iso/06/87471ad1bb04c3757fb52e3bf466644f49cfceafdcfcb88de7f771ca3f6484/test.iso ------------------> synced

Please make sure that everything is correct in pulp_content.conf, especially if /var/lib/pulp/ is symlinked.

My pulp_content.conf is correct and /var/lib/pulp is NOT symlinked.

It's very reproducible in my current setup. I have content (which is listed in the PULP_MANIFEST) which is no longer accessible because the links created when publishing are wrong. The content was uploaded when the server was under version 2.8.0. New content uploaded after upgrading to 2.10 is linked correctly.

If someone can tell me a workaround, that would be fine. But my 2.8 uploaded files are in limbo now.

I am more than willing to do whatever testing is needed since my system I can reproduce the problem with a simple "publish"

Can you verify a few things for me?

First:¶

Based on the Description, It's my understanding that the symlink target: /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7/zookeeper-3.4.8.tar.gz does not exist. Correct?

Does any part of that path after units/iso/ exist?

Can you post the output of tree or find starting with what does exist after units/iso/f2/?

Second:¶

Can you get me the DB record for this ISO?

Should be:

$ mongo pulp_database
> db.units_iso.find({name:"zookeeper-3.4.8.tar.gz"})
> exit

Thanks!

You are correct, the symlink does not exist:
$ ls -l /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7/zookeeper-3.4.8.tar.gz
ls: cannot access /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7/zookeeper-3.4.8.tar.gz: Not a directory

However...if you drop the filename from the end, the tar file is actually named: 2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7

$ ls l /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7
-rw-r--r- 1 apache apache 22261552 Apr 21 2016 /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7

$ file /var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7
/var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7: gzip compressed data, from Unix, last modified: Fri Feb 5 19:51:25 2016

I verified that it is actually the zookeeper tar file.

I do not have the mongo app loaded. The pulp install only installed mongodb-server. Do you know what package contains the mongo client binary? (I am wary to install the full mongodb package on the chance it might overwrite my existing database.)

Thanks for the information.

I understand your concern. The mongo packages are poorly named and even the documentation is misleading. Contrary to its name, mongodb package only installs the client shell, tools and documentation.

[jortel@f23d pulp]$ rpm -ql mongodb
/usr/bin/mongo
/usr/bin/mongoperf
/usr/bin/mongosniff
/usr/share/doc/mongodb
/usr/share/doc/mongodb/README
/usr/share/licenses/mongodb
/usr/share/licenses/mongodb/APACHE-2.0.txt
/usr/share/licenses/mongodb/GNU-AGPL-3.0.txt
/usr/share/man/man1/mongo.1.gz
/usr/share/man/man1/mongoperf.1.gz
/usr/share/man/man1/mongosniff.1.gz

You can also install the client package, mongodb, on a separate host and connect to your database using the following command. This command assumes the database server has IP address 1.2.3.4.

mongo 1.2.3.4/pulp_database

Thanks for clarifying what mongodb installs.

Here is the output you were asking for:

$ mongo pulp_database
MongoDB shell version: 2.6.12
connecting to: pulp_database
Welcome to the MongoDB shell.
> db.units_iso.find({name:"zookeeper-3.4.8.tar.gz"})
{ "_id" : "b9bfc4ba-35df-4b95-8620-987bbd339ebf", "pulp_user_metadata" : {  }, "_last_updated" : 1461278671, "_storage_path" : "/var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7/zookeeper-3.4.8.tar.gz", "downloaded" : true, "name" : "zookeeper-3.4.8.tar.gz", "checksum" : "f10a0b51f45c4f64c1fe69ef713abf9eb9571bc7385a82da892e83bb6c965e90", "size" : 22261552, "_ns" : "units_iso", "_content_type_id" : "iso" }
> exit
bye

Interesting. The unit._storage_path does not match where the file is actually stored. Really having a hard time explaining this but will keep digging. Regardless, let's focus on getting your data fixed. I'm not sure how may ISO type content units you have but will suggest the following in hopes that it will be manageable.

Run this:

> db.units_iso.find({}, {_storage_path:1})
{ "_id" : "fa987e12-49cc-479c-8e04-d2f60c364747", "_storage_path" : "/var/lib/pulp/content/units/iso/79/bda7e1c3fed6ed2a2f723d2fff2d39cdb161cd3288f5637949cfb8f4239d1a" }
> 

to get a list of content units and their storage paths in the DB. Then, use that information to validate and adjust where the files are stored to match what is in the DB.

Does this sound reasonable or do the numbers make this impractical?

It looks like I only have 18 units...but my fear is that they will disappear again if I add another unit and republish.

But anything to get me going again would be GREATLY appreciated!

Here is the list:
(You can see the new ones that were added...they do not include the unit name in the storage path)

> db.units_iso.find({}, {_storage_path:1})
{ "_id" : "f6e69408-66c2-47d1-ab2c-fdadf7015e89", "_storage_path" : "/var/lib/pulp/content/units/iso/8f/169411dead97ae30aa3722cedd5630013e290a5f5238d8d47dd9677aea4c82/CentOS-6.7-x86_64-minimal.iso" }
{ "_id" : "93da2b69-a94e-41cc-ab56-448f188ce4d5", "_storage_path" : "/var/lib/pulp/content/units/iso/1e/21cd3d18f3c0d90d0941ed2842a01d665536bcde7b2930536679c8be7919e6/CentOS-7-x86_64-Minimal-1511.iso" }
{ "_id" : "b9bfc4ba-35df-4b95-8620-987bbd339ebf", "_storage_path" : "/var/lib/pulp/content/units/iso/f2/2a909ec15770cbffc478d5ea344205f2c5ace7036e6d0dd65681ed87fff8d7/zookeeper-3.4.8.tar.gz" }
{ "_id" : "7078969f-351b-4a2f-96d1-98a05e40f692", "_storage_path" : "/var/lib/pulp/content/units/iso/1c/22f0d20afbbe72c3f78c1b916f02c7c37433b72dac7aac8319a49cca314d38/kafka_2.11-0.9.0.1.tgz" }
{ "_id" : "1ca6e67d-665a-433d-b373-fa9b84f73352", "_storage_path" : "/var/lib/pulp/content/units/iso/1c/249b541bbb1d3589123b823c97fb63990b909d2217e94a30c3f43913c721f4/chef-1-lv-701.admin.purple.us.crt" }
{ "_id" : "b634b53b-950a-498f-b2cc-fc867972fd9f", "_storage_path" : "/var/lib/pulp/content/units/iso/cb/1c3148e9086edde7b5fcfb1158e7394e7d6204cc205220db7e53d59fac9120/PURPLE-validator.pem" }
{ "_id" : "5f6eb892-0a39-4a9b-8ed5-1561af241283", "_storage_path" : "/var/lib/pulp/content/units/iso/cf/4c5e646c7d1a2f7fa557ebb87e6e1f2470d4c4f4a2691735a014de472a4292/opscode-push-jobs-server-1.1.6-1.x86_64.rpm" }
{ "_id" : "6d775854-6789-4822-89fe-850a08420be7", "_storage_path" : "/var/lib/pulp/content/units/iso/b8/5bf66fe40148d2badebc604e9d8dd1318b332d5c3bb441c48fa2ccfc9987e7/nagiosna-2.2.0.tar.gz" }
{ "_id" : "eec97a55-70e6-43ea-b7f8-cc781650be9d", "_storage_path" : "/var/lib/pulp/content/units/iso/45/8a2b17a46807a042a03c3cbaebed27361bd3f200adab79838f80fbf597f766/jre-8u101-linux-x64.tar.gz" }
{ "_id" : "27920af9-3d98-4685-af92-1882c16df2cb", "_storage_path" : "/var/lib/pulp/content/units/iso/26/a981b6d20b0842df62638f210c9cd1d6f60690f6076afc220b2ae7833a00ae/ipa.key" }
{ "_id" : "5a39b62a-4d2b-4fac-84b4-2819d2870d2a", "_storage_path" : "/var/lib/pulp/content/units/iso/f7/fcebdfe7224c1f3b72e1ca937e2f001a3f29eec3ed54f69fe6742fa6960015/marathon-1.1.1.tgz" }
{ "_id" : "8c40239e-f68b-446e-ad59-20caf2164a2c", "_storage_path" : "/var/lib/pulp/content/units/iso/b4/8ab17dad066b0ff660b14f1d8945249a9e431bf6e2a66794692655e8ce24c1/purple_broker-1.0.tar.gz" }
{ "_id" : "2a22fd3f-2f14-4ec9-bba3-7f055227980d", "_storage_path" : "/var/lib/pulp/content/units/iso/a1/5c8dafabff1fbdc05ef58a65ee1530bdf52d21dc5bf116321525a8ba2d573a/purple_broker-1.0.1.tar.gz" }
{ "_id" : "4213ce83-caf2-4083-868b-62dd1a05f53e", "_storage_path" : "/var/lib/pulp/content/units/iso/71/8b22aeccdb7b37b202e462d5db6f0e4b8aac6387150529ba39d741ba115504" }
{ "_id" : "22e9ea32-2270-4b77-9faf-a690dc991729", "_storage_path" : "/var/lib/pulp/content/units/iso/d8/4bac67eb58d1270eed22b7e0a65f361d252d417bb47f4c4456ee4fb95e239d" }
{ "_id" : "1f58f684-2d95-4e80-8faa-8945770706c6", "_storage_path" : "/var/lib/pulp/content/units/iso/b3/10486cc51eb711ccbd866eab0bcabc7e1829dad2dbfe8aa4cad96750481113/purple_broker-1.0.tar.gz" }
{ "_id" : "1ac91129-f410-4669-b755-3444d5c928c0", "_storage_path" : "/var/lib/pulp/content/units/iso/1e/739c486f36ab929f004449afa4d4f04c617126f64e3d640b2138f6002ca713" }
{ "_id" : "2b99a53c-8d7d-4c6f-af13-e347f9033969", "_storage_path" : "/var/lib/pulp/content/units/iso/61/55e55fd6647b25978769666e05003caddfc385b251b0e0b947055f099fb7a1" }

I understand.

Although, I'm still digging into how your data got this way, I really don't think it could be caused by publishing. That said, you could fix one; publish; and then validate that the one you fixed is .. still fixed.

I recommend proceeding.

Did you initially install pulp 2.8.0 or was it upgraded from older version? If upgraded, were any of these units uploaded before the upgrade to 2.8.0?

Oh...I 100% agree with proceeding. :-)

I started with 2.8.0.

Thanks for your help.

Fixing this manually is good for the reporter, but not good for other Pulp users. Before the reporter who can provide more info resolves all the issues, I hope a root cause is identified.

I am happy to provide whatever info the community needs.

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.