mergerepo weird behavior

Discussion:

Jon Wallace

2011-08-09 14:56:18 UTC

Hello,

I'm seeing some weird mergerepo behavior I can't make sense of. I have
three repositories, PRODA, PRODB, PRODC, and each has a single package
within it. I'm trying to merge various combinations of these together.
The example below tries to merge PRODB and PRODC into PROD-BC.

#mergerepo --repo=file:///<path-to-repo>/PRODB
--repo=file:///<path-to-repo>/PRODC -o PROD-BC
Loaded plugins: refresh-packagekit
Not using downloaded repomd.xml because it is older than what we have:
Current : Tue Aug 9 09:36:40 2011
Downloaded: Mon Aug 8 17:50:36 2011
3/3 - hp-PRODC-1.0-1.x86_64
Saving Primary metadata
Saving file lists metadata
Saving other metadata

The "current" timestamp of Aug 9 9:36 corresponds to PRODA.
The "downloaded" timestamp of Aug 8 17:50 corresponds to PRODB.
Though not listed, PRODC repomd.xml has a timestamp of Aug 8 17:54.

The result is that I end up with a merge of PRODA and PRODC even though
I asked for PRODB and PRODC.

Here is the merged group file.
#cat groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>

<group>
<id>PRODA</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-A Test Group</name>
<description>This is PROD-A #1111.</description>
<packagelist>
<packagereq type="default">PRODA</packagereq>
</packagelist>
</group>
<group>
<id>PRODC</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-C Test Group</name>
<description>This is PROD-C #1111.</description>
<packagelist>
<packagereq type="default">PRODC</packagereq>
</packagelist>
</group>
<category>
<id>producta</id>
<display_order>10</display_order>
<name>Product-A Test Software</name>
<description>Product-A Test Software - Hello </description>
<grouplist>
<groupid>PRODA</groupid>
</grouplist>
</category>

<category>
<id>productc</id>
<display_order>10</display_order>
<name>Product-C Test Software</name>
<description>Product-C Test Software - Hello</description>
<grouplist>
<groupid>PRODC</groupid>
</grouplist>
</category>

</comps>

What am I doing wrong, or what have I set up wrong with my repositories?
Jon

seth vidal

2011-08-09 15:23:44 UTC

Permalink

Post by Jon Wallace
Hello,
I'm seeing some weird mergerepo behavior I can't make sense of. I have
three repositories, PRODA, PRODB, PRODC, and each has a single package
within it. I'm trying to merge various combinations of these together.
The example below tries to merge PRODB and PRODC into PROD-BC.
#mergerepo --repo=file:///<path-to-repo>/PRODB
--repo=file:///<path-to-repo>/PRODC -o PROD-BC
Loaded plugins: refresh-packagekit
Current : Tue Aug 9 09:36:40 2011
Downloaded: Mon Aug 8 17:50:36 2011
3/3 - hp-PRODC-1.0-1.x86_64
Saving Primary metadata
Saving file lists metadata
Saving other metadata
The "current" timestamp of Aug 9 9:36 corresponds to PRODA.
The "downloaded" timestamp of Aug 8 17:50 corresponds to PRODB.
Though not listed, PRODC repomd.xml has a timestamp of Aug 8 17:54.
The result is that I end up with a merge of PRODA and PRODC even though
I asked for PRODB and PRODC.
Here is the merged group file.
#cat groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
<group>
<id>PRODA</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-A Test Group</name>
<description>This is PROD-A #1111.</description>
<packagelist>
<packagereq type="default">PRODA</packagereq>
</packagelist>
</group>
<group>
<id>PRODC</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-C Test Group</name>
<description>This is PROD-C #1111.</description>
<packagelist>
<packagereq type="default">PRODC</packagereq>
</packagelist>
</group>
<category>
<id>producta</id>
<display_order>10</display_order>
<name>Product-A Test Software</name>
<description>Product-A Test Software - Hello </description>
<grouplist>
<groupid>PRODA</groupid>
</grouplist>
</category>
<category>
<id>productc</id>
<display_order>10</display_order>
<name>Product-C Test Software</name>
<description>Product-C Test Software - Hello</description>
<grouplist>
<groupid>PRODC</groupid>
</grouplist>
</category>
</comps>
What am I doing wrong, or what have I set up wrong with my repositories?
Jon

what ver of createrepo/mergerepo is this?

Can you go into /var/tmp/ and look for a tmpdir with your username in it
and clear it out? maybe something like /var/tmp/yum-$username-blah

-sv

Jon Wallace

2011-08-10 21:39:00 UTC

Permalink

Post by seth vidal

Post by Jon Wallace
Hello,
I'm seeing some weird mergerepo behavior I can't make sense of. I have
three repositories, PRODA, PRODB, PRODC, and each has a single package
within it. I'm trying to merge various combinations of these together.
The example below tries to merge PRODB and PRODC into PROD-BC.
#mergerepo --repo=file:///<path-to-repo>/PRODB
--repo=file:///<path-to-repo>/PRODC -o PROD-BC
Loaded plugins: refresh-packagekit
Current : Tue Aug 9 09:36:40 2011
Downloaded: Mon Aug 8 17:50:36 2011
3/3 - hp-PRODC-1.0-1.x86_64
Saving Primary metadata
Saving file lists metadata
Saving other metadata
The "current" timestamp of Aug 9 9:36 corresponds to PRODA.
The "downloaded" timestamp of Aug 8 17:50 corresponds to PRODB.
Though not listed, PRODC repomd.xml has a timestamp of Aug 8 17:54.
The result is that I end up with a merge of PRODA and PRODC even though
I asked for PRODB and PRODC.
Here is the merged group file.
#cat groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>
<group>
<id>PRODA</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-A Test Group</name>
<description>This is PROD-A #1111.</description>
<packagelist>
<packagereq type="default">PRODA</packagereq>
</packagelist>
</group>
<group>
<id>PRODC</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-C Test Group</name>
<description>This is PROD-C #1111.</description>
<packagelist>
<packagereq type="default">PRODC</packagereq>
</packagelist>
</group>
<category>
<id>producta</id>
<display_order>10</display_order>
<name>Product-A Test Software</name>
<description>Product-A Test Software - Hello</description>
<grouplist>
<groupid>PRODA</groupid>
</grouplist>
</category>
<category>
<id>productc</id>
<display_order>10</display_order>
<name>Product-C Test Software</name>
<description>Product-C Test Software - Hello</description>
<grouplist>
<groupid>PRODC</groupid>
</grouplist>
</category>
</comps>
What am I doing wrong, or what have I set up wrong with my repositories?
Jon

what ver of createrepo/mergerepo is this?
Can you go into /var/tmp/ and look for a tmpdir with your username in it
and clear it out? maybe something like /var/tmp/yum-$username-blah
-sv

Sorry for the delay and length of this email, I had to rebuild my test
environment but at least I've been able to duplicate the problem again.
Also trying to provide as much info as possible.

OS = Fedora Core 11
createrepo = 0.9.7-7

I cleared /var/tmp/yum-<myname>-blah and confirmed that solves the
problem, but only if I clear it prior to each time I run mergerepo. I
suppose I can do that but it seems a bit of a pain and should probably
be documented if that's the fix. Here's a synopsis of the problem
again, and I've included the contents of some files.

REPOSITORY CONTENTS
./PRODA
./PRODA/repodata
./PRODA/repodata/repomd.xml
./PRODA/repodata/groups.xml.gz
./PRODA/repodata/other.xml.gz
./PRODA/repodata/groups.xml
./PRODA/repodata/primary.xml.gz
./PRODA/repodata/filelists.xml.gz
./PRODA/PRODA-1.0-1.x86_64.rpm
./PRODA/groups.xml
./PRODB
./PRODB/repodata
./PRODB/repodata/repomd.xml
./PRODB/repodata/groups.xml.gz
./PRODB/repodata/other.xml.gz
./PRODB/repodata/groups.xml
./PRODB/repodata/primary.xml.gz
./PRODB/repodata/filelists.xml.gz
./PRODB/PRODB-1.0-1.x86_64.rpm
./PRODB/groups.xml
./PRODC
./PRODC/repodata
./PRODC/repodata/repomd.xml
./PRODC/repodata/groups.xml.gz
./PRODC/repodata/other.xml.gz
./PRODC/repodata/groups.xml
./PRODC/repodata/primary.xml.gz
./PRODC/repodata/filelists.xml.gz
./PRODC/PRODC-1.0-1.x86_64.rpm
./PRODC/groups.xml

Note that each package contains only a single file trying to make the
contents of the repository as simple as possible.

$ cat PRODA/groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>

<group>
<id>PRODA</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-A Test Group</name>
<description>This is PROD-A #1111.</description>
<packagelist>
<packagereq type="default">hp-PRODA</packagereq>
</packagelist>
</group>

<category>
<id>producta</id>
<display_order>10</display_order>
<name>Product-A Test Software</name>
<description>Product-A Test Software Prototype Evaluation</description>
<grouplist>
<groupid>PRODA</groupid>
</grouplist>
</category>

</comps>

$ cat PRODB/groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>

<group>
<id>PRODB</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-B Test Group</name>
<description>This is PROD-B #1111.</description>
<packagelist>
<packagereq type="default">hp-PRODB</packagereq>
</packagelist>
</group>

<category>
<id>productb</id>
<display_order>10</display_order>
<name>Product-B Test Software</name>
<description>Product-B Test Software Prototype Evaluation</description>
<grouplist>
<groupid>PRODB</groupid>
</grouplist>
</category>

</comps>

$ cat PRODC/groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>

<group>
<id>PRODC</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-C Test Group</name>
<description>This is PROD-C #1111.</description>
<packagelist>
<packagereq type="default">hp-PRODC</packagereq>
</packagelist>
</group>

<category>
<id>productc</id>
<display_order>10</display_order>
<name>Product-C Test Software</name>
<description>Product-C Test Software Prototype Evaluation</description>
<grouplist>
<groupid>PRODC</groupid>
</grouplist>
</category>

</comps>

1) Start with a clean /var/tmp/yum-<myname>, then merge PRODA, PRODB,
PRODC into PROD-ABC. This completes as expected with no errors.

2) DO NOT clear /var/tmp/yum-<myname>, then merge PRODB and PRODC into
PROD-BC, and I get the following results.

$ mergerepo --repo=file:///<absoute-path>/PRODB
--repo=file:///<absolute-path>/PRODC -o PROD-BC
Loaded plugins: refresh-packagekit
Not using downloaded repomd.xml because it is older than what we have:
Current : Wed Aug 10 16:34:51 2011
Downloaded: Wed Aug 10 16:14:11 2011
2/2 - hp-PRODC-1.0-1.x86_64
Saving Primary metadata
Saving file lists metadata
Saving other metadata

$ ls PROD-BC
total 4
drwxr-xr-x. 2 jon jon 4096 2011-08-10 16:41 repodata

$ ls PROD-BC/repodata
total 28
-rw-r--r--. 1 jon jon 374 2011-08-10 16:41 filelists.xml.gz
-rw-r--r--. 1 jon jon 1328 2011-08-10 16:41 groups.xml
-rw-r--r--. 1 jon jon 457 2011-08-10 16:41 groups.xml.gz
-rw-r--r--. 1 jon jon 344 2011-08-10 16:41 other.xml.gz
-rw-r--r--. 1 jon jon 824 2011-08-10 16:41 primary.xml.gz
-rw-r--r--. 1 jon jon 2091 2011-08-10 16:41 repomd.xml
-rw-r--r--. 1 jon jon 116 2011-08-10 16:41 updateinfo.xml.gz

HERE IS THE MERGED GROUPS FILE.
Note that even though I merged PRODB and PRODC, the groups file has an
entry for PRODA in both the group and category tags.

$ cat groups.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps PUBLIC "-//Red Hat, Inc.//DTD Comps info//EN" "comps.dtd">
<comps>

<group>
<id>PRODA</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-A Test Group</name>
<description>This is PROD-A #1111.</description>
<packagelist>
<packagereq type="default">hp-PRODA</packagereq>
</packagelist>
</group>
<group>
<id>PRODC</id>
<default>true</default>
<uservisible>true</uservisible>
<display_order>1024</display_order>
<name>PROD-C Test Group</name>
<description>This is PROD-C #1111.</description>
<packagelist>
<packagereq type="default">hp-PRODC</packagereq>
</packagelist>
</group>
<category>
<id>producta</id>
<display_order>10</display_order>
<name>Product-A Test Software</name>
<description>Product-A Test Software Prototype Evaluation</description>
<grouplist>
<groupid>PRODA</groupid>
</grouplist>
</category>

<category>
<id>productc</id>
<display_order>10</display_order>
<name>Product-C Test Software</name>
<description>Product-C Test Software Prototype Evaluation</description>
<grouplist>
<groupid>PRODC</groupid>
</grouplist>
</category>

</comps>

HERE IS THE MERGED filelists.xml FILE.
Note it contains the PRODA package even though I merged PRODB and PRODC.

$ cat filelists.xml.gz | gunzip
<?xml version="1.0" encoding="UTF-8"?>
<filelists xmlns="http://linux.duke.edu/metadata/filelists" packages="2">
<package
pkgid="85fca2cee6628eea8f6aa67001606e3585380b1ff6ba8dce2dfef57e38d1d6d1"
name="hp-PRODA" arch="x86_64">
<version epoch="0" ver="1.0" rel="1"/>
<file>/opt/HP/test1/fileA-1</file>
</package>

<package
pkgid="b681e126c401780913c2878c253f8e513520f23194117e77c82bb5d4ecbeaa3c"
name="hp-PRODC" arch="x86_64">
<version epoch="0" ver="1.0" rel="1"/>
<file>/opt/HP/test1/fileC-1</file>
</package>

HERE IS THE MERGED primary.xml FILE.
Note that the <location> tag has a reference to the PRODB repository but
lists the package as PRODA. The only thing I have changed here is the
path to the repositories with <absolute-path> to protect the
innocent...(me).

$ cat primary.xml.gz | gunzip
<?xml version="1.0" encoding="UTF-8"?>
<metadata xmlns="http://linux.duke.edu/metadata/common"
xmlns:rpm="http://linux.duke.edu/metadata/rpm" packages="2">
<package type="rpm">
<name>hp-PRODA</name>
<arch>x86_64</arch>
<version epoch="0" ver="1.0" rel="1"/>
<checksum type="sha256"
pkgid="YES">85fca2cee6628eea8f6aa67001606e3585380b1ff6ba8dce2dfef57e38d1d6d1</checksum>
<summary>HP PROD-A Test RPM x86_64</summary>
<description>This is PROD-A test RPM ID# 1296569852</description>
<packager></packager>
<url></url>
<time file="1313007190" build="1313006939"/>
<size package="2023" installed="2023" archive="316"/>
<location xml:base="file:///<absolute-path>/PRODB"
href="hp-PRODA-1.0-1.x86_64.rpm"/>
<format>
<rpm:license>GPLv2+</rpm:license>
<rpm:vendor/>
<rpm:group>HPTEST</rpm:group>
<rpm:buildhost>FC11</rpm:buildhost>
<rpm:sourcerpm>hp-PRODA-1.0-1.src.rpm</rpm:sourcerpm>
<rpm:header-range start="280" end="1856"/>
<rpm:provides>
<rpm:entry name="hp-PRODA(x86-64)" flags="EQ" epoch="0" ver="1.0" rel="1"/>
<rpm:entry name="hp-PRODA" flags="EQ" epoch="0" ver="1.0" rel="1"/>
</rpm:provides>
<rpm:requires>
<rpm:entry name="/bin/sh" pre="0"/>
</rpm:requires>
</format>
</package>
<package type="rpm">
<name>hp-PRODC</name>
<arch>x86_64</arch>
<version epoch="0" ver="1.0" rel="1"/>
<checksum type="sha256"
pkgid="YES">b681e126c401780913c2878c253f8e513520f23194117e77c82bb5d4ecbeaa3c</checksum>
<summary>HP PROD-C Test RPM x86_64</summary>
<description>This is PROD-C test RPM ID# 1296569852</description>
<packager></packager>
<url></url>
<time file="1313007214" build="1313006978"/>
<size package="2020" installed="2020" archive="316"/>
<location xml:base="file:///<absolute-path>/PRODC"
href="hp-PRODC-1.0-1.x86_64.rpm"/>
<format>
<rpm:license>GPLv2+</rpm:license>
<rpm:vendor/>
<rpm:group>HPTEST</rpm:group>
<rpm:buildhost>FC11</rpm:buildhost>
<rpm:sourcerpm>hp-PRODC-1.0-1.src.rpm</rpm:sourcerpm>
<rpm:header-range start="280" end="1856"/>
<rpm:provides>
<rpm:entry name="hp-PRODC(x86-64)" flags="EQ" epoch="0" ver="1.0" rel="1"/>
<rpm:entry name="hp-PRODC" flags="EQ" epoch="0" ver="1.0" rel="1"/>
</rpm:provides>
<rpm:requires>
<rpm:entry name="/bin/sh" pre="0"/>
</rpm:requires>
</format>
</package>
</metadata>

Jon Wallace

2011-08-16 14:17:09 UTC

Permalink

Hello,

I sent some file contents a few days ago to show the results I was
getting in reference to sequential merges of repositories in different
combinations. I haven't heard back from anyone on this...just curious
if I should open a ticket on it, if everyone is just busy, or if this
just wasn't found to be an interesting problem. Not sure of protocol, I
guess is my question here. I usually try to find out if something is a
known issue before submitting a bug ticket. Any constructive
suggestions? Should I submit a fix for review or is there a fixed set
of developers working on createrepo?

Jon

seth vidal

2011-08-16 19:12:30 UTC

Permalink

Post by Jon Wallace
Hello,
I sent some file contents a few days ago to show the results I was
getting in reference to sequential merges of repositories in different
combinations. I haven't heard back from anyone on this...just curious
if I should open a ticket on it, if everyone is just busy, or if this
just wasn't found to be an interesting problem. Not sure of protocol, I
guess is my question here. I usually try to find out if something is a
known issue before submitting a bug ticket. Any constructive
suggestions? Should I submit a fix for review or is there a fixed set
of developers working on createrepo?

1. F11 is quite old
2. the createrepo in there has seen a fair number of changes, too.
3. Can you verify in createrepo/merge.py in your createrepo pkg that you
see something that says:

metadata_expire=0,

anywhere in there?

I doubt it is - it was added in jan of 2011.

Seems likely that a newer createrepo will solve your problem.

-sv

Jon Wallace

2011-08-17 21:59:17 UTC

Permalink

Post by seth vidal
1. F11 is quite old
2. the createrepo in there has seen a fair number of changes, too.
3. Can you verify in createrepo/merge.py in your createrepo pkg that you
metadata_expire=0,
anywhere in there?
I doubt it is - it was added in jan of 2011.
Seems likely that a newer createrepo will solve your problem.

I updated to FC-15 and createrepo-9.9-4. Problem persists. The
merge.py file does have "metadata_expire=0" in it.

Jon

James Antill

2011-08-24 21:07:13 UTC

Permalink

Post by Jon Wallace
I updated to FC-15 and createrepo-9.9-4. Problem persists. The
merge.py file does have "metadata_expire=0" in it.

Ok ... got some free time to test this and it turned out to be a bug in
yum (in theory it affects more just groups/pkgtags in mergerepo, but in
practice probably not). Fix is simple enough:

commit 581d161bbefc9b940289f15a93c3fdef6ad8c843
Author: James Antill <***@and.org>
Date: Wed Aug 24 17:03:58 2011 -0400

Fix mergerepo on generated data (groups/pkgtag/etc.)

diff --git a/yum/yumRepo.py b/yum/yumRepo.py
index 91b7dde..a681370 100644
--- a/yum/yumRepo.py
+++ b/yum/yumRepo.py
@@ -1256,6 +1256,11 @@ Insufficient space in download directory %s

# if we have a 'fresh' repomd.xml then update the cookie
self.setMetadataCookie()
+ # Also remove any files that are generated ... this can be slightl
+ # non-optimal, in the case where we have to revert, but it's just
+ # sooo much easier to deal with it here.
+ for fname in misc.getFileList(self.cachedir + '/gen', '', []):
+ misc.unlink_f(fname)

if self._repoXML is None:
self._repoXML = self._parseRepoXML(result)

...feel free to confirm.

Jon Wallace

2011-09-02 06:04:34 UTC

Permalink

Post by James Antill
Ok ... got some free time to test this and it turned out to be a bug in
yum (in theory it affects more just groups/pkgtags in mergerepo, but in
commit 581d161bbefc9b940289f15a93c3fdef6ad8c843
Date: Wed Aug 24 17:03:58 2011 -0400
Fix mergerepo on generated data (groups/pkgtag/etc.)
diff --git a/yum/yumRepo.py b/yum/yumRepo.py
index 91b7dde..a681370 100644
--- a/yum/yumRepo.py
+++ b/yum/yumRepo.py
@@ -1256,6 +1256,11 @@ Insufficient space in download directory %s
# if we have a 'fresh' repomd.xml then update the cookie
self.setMetadataCookie()
+ # Also remove any files that are generated ... this can be slightl
+ # non-optimal, in the case where we have to revert, but it's just
+ # sooo much easier to deal with it here.
+ misc.unlink_f(fname)
self._repoXML = self._parseRepoXML(result)
...feel free to confirm.

Thanks,
I've confirmed this modification fixes the cache problem with createrepo
9.9-4. Any projected date for a new package that includes this fix?

Jon

James Antill

2011-08-09 15:27:09 UTC

Permalink

What versions of createrepo and yum? If it's RHEL-6, then you need this
createrepo patch:

commit 4047f09e0f1b41427d6cf943489e2a3992320365
Author: James Antill <***@and.org>
Date: Fri Jan 21 11:14:34 2011 -0500

Override timestamp check on repos. for mergerepo (like repodiff).

We fixed one "gotcha" recently with timestamp_check, in yum, but that
was when the baseurl didn't work IIRC.

You can work around it by doing:

rm -rf /var/cache/yum/.../repo?

...as the repos are created as "repo1", "repo2" etc.