Rpm-metadata Digest, Vol 75, Issue 2

Matthew Dawkins

2010-08-03 16:30:00 UTC

Send Rpm-metadata mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Rpm-metadata digest..."
1. Re: adding different compression types to createrepo
(Anders F Bj?rklund)
2. Re: adding different compression types to createrepo
(Duncan Mac-Vicar P.)
3. Re: adding different compression types to createrepo (seth vidal)
4. Re: adding different compression types to createrepo (seth vidal)
5. Re: Rpm-metadata Digest, Vol 75, Issue 1 (Matthew Dawkins)
6. Re: adding different compression types to createrepo
(James Antill)
----------------------------------------------------------------------
Message: 1
Date: Tue, 3 Aug 2010 16:55:55 +0200
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

I think the extended opensuse patch should be acceptable, where it
uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
and "other_lzma", it causes less breakage than changing "primary"...

/usr/bin/repo2solv.sh which is the tool that parses the repos in
openSUSE, already supports lzma (look at the source), we just don't
have any tool generating lzma.

And Smart also supports lzma/xz, since it just uses the file suffix.
I was under the impression that you had patched createrepo for lzma ?
And it should be simple to support both of .lzma and .xz compressions
by using PylibLZMA in yum, or liblzma directly for yum-metadata-parser.

I disagree the type should be changed. It would add more confusing
undocumented stuff to the format.

The concern was that if the type wasn't changed, then older versions
of yum would try to parse the lzma/xz as uncompressed and just get
checksum errors. So it was more for "compatibility" than anything ?
But the "primary_lzma" wasn't my idea, since it came from openSUSE.
--anders
------------------------------
Message: 2
Date: Tue, 03 Aug 2010 16:45:31 +0200
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

/usr/bin/repo2solv.sh which is the tool that parses the repos in
openSUSE, already supports lzma (look at the source), we just don't have
any tool generating lzma.
I disagree the type should be changed. It would add more confusing
undocumented stuff to the format.
Duncan
------------------------------
Message: 3
Date: Tue, 03 Aug 2010 11:07:05 -0400
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset="UTF-8"

Hello, I'm new to the list, but not new to using createrepo and yum
for repo creation.
I'm looking to find if there are not patches out there to add an
option to calling for lzma or xz compression of the metafiles instead
of gzip. I have seen what opensuse uses to compress the primary.xml
in conjunction with the gzip compression and I have extended on it to
compress the other and filelists files too, but they don't really add
clean functionality.

there aren't any upstream, yet. I just got a python liblzma pkg into
fedora to support that compression natively in python. So that's half of
it. The biggest trick is having to name the datatype in repomd.xml to
reflect the compression format since older yums (and apts and smarts,
etc) won't necessarily be able to handle lzma compressed files.
-sv
------------------------------
Message: 4
Date: Tue, 03 Aug 2010 11:14:28 -0400
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset="UTF-8"

I'm looking to find if there are not patches out there to add an
option to calling for lzma or xz compression of the metafiles
instead of gzip. I have seen what opensuse uses to compress the
primary.xml in conjunction with the gzip compression and I have
extended on it to compress the other and filelists files too, but
they don't really add clean functionality.

YAML? I don't think anyone has discussed YAML and was taken seriously.
More likely than not most of the metadata will be transitioned to
sqlite-db ONLY and no xml at all. Yum already supports repos with only
sqlite dbs and no xml (other than repomd.xml).

[primary|filelists|other][_db]_xz in the repomd.xml, in addition to the
gzip/bzip2 compressed alternatives and then gradually phase out the
older ones.
My biggest reticence in adding the xz support now is that if we move to
changing the repodata format to the layout that's been discussed on here
and on yum-devel list that we'll be able to move to xz support(and some
future proofing for compression formatting) w/o having to kludge a bunch
of xz compression crap into createrepo.
What do y'all think?
-sv
------------------------------
Message: 5
Date: Tue, 3 Aug 2010 09:41:23 -0600
Subject: Re: [Rpm-metadata] Rpm-metadata Digest, Vol 75, Issue 1
Content-Type: text/plain; charset="iso-8859-1"

Send Rpm-metadata mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Rpm-metadata digest..."
1. adding different compression types to createrepo (Matthew Dawkins)
2. Re: adding different compression types to createrepo
(Anders F Bj?rklund)
----------------------------------------------------------------------
Message: 1
Date: Mon, 2 Aug 2010 17:39:45 -0600
Subject: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset="iso-8859-1"
Hello, I'm new to the list, but not new to using createrepo and yum for
repo
creation.
I'm looking to find if there are not patches out there to add an option

calling for lzma or xz compression of the metafiles instead of gzip. I

have

seen what opensuse uses to compress the primary.xml in conjunction with
the
gzip compression and I have extended on it to compress the other and
filelists files too, but they don't really add clean functionality.
Regards,
Matthew Dawkins
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <

http://lists.baseurl.org/pipermail/rpm-metadata/attachments/20100802/dcf602ee/attachment-0001.html

------------------------------
Message: 2
Date: Tue, 3 Aug 2010 10:46:07 +0200
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

That would be https://features.opensuse.org/309167 I think,
but see also http://duncan.mac-vicar.com/blog/archives/537
/and http://lists.baseurl.org/pipermail/yum/2009-April/022560.html)
and some other discussion about using YAML and/or LZMA for repodata.
I think the extended opensuse patch should be acceptable, where it
uses "primary" for .xml or .xml.gz and the new "primary_lzma" key
for .xml.lzma or .xml.xz - plus the same added for "filelists_lzma"
and "other_lzma", it causes less breakage than changing "primary"...
--anders
------------------------------
_______________________________________________
Rpm-metadata mailing list
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
End of Rpm-metadata Digest, Vol 75, Issue 1
*******************************************

Well, I'll add the URL to my rediffed patch. It would be nice to drop the
hybrid from making both compression types files and just use one or the
other.
http://dev.unity-linux.org/projects/unitylinux/repository/entry/packages/createrepo/P/createrepo-0.9.8-add-lzma-option-to-generate-primary.xml.lzma.patch
I tried to do it with little luck. I believe the creation of the checksum
for other than gzip files present is needed and then not appending the the
primary,other,filelists_lzma, but rather just that info instead inside
repomd.xml.
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.baseurl.org/pipermail/rpm-metadata/attachments/20100803/af6cdff4/attachment-0001.html
------------------------------
Message: 6
Date: Tue, 03 Aug 2010 12:02:45 -0400
Subject: Re: [Rpm-metadata] adding different compression types to
createrepo
Content-Type: text/plain; charset="UTF-8"

[primary|filelists|other][_db]_xz in the repomd.xml, in addition to the
gzip/bzip2 compressed alternatives and then gradually phase out the
older ones.

*shrug*, I'm not sure why we'd want to add support for lzma on the
obsolete .xml files. Smart and apt both support .sqlite now, and if
zypper doesn't they can always use modifyrepo to add them for SuSE.
Also calling them "primary_db.xz" would fit the convention with
"groups.gz".

My biggest reticence in adding the xz support now is that if we move to
changing the repodata format to the layout that's been discussed on here
and on yum-devel list that we'll be able to move to xz support(and some
future proofing for compression formatting) w/o having to kludge a bunch
of xz compression crap into createrepo.

Well it's going to be much easier to just change the compression, but
it's not the end of the world if you want to put it off a year or so
(IMO).
------------------------------
_______________________________________________
Rpm-metadata mailing list
http://lists.baseurl.org/mailman/listinfo/rpm-metadata
End of Rpm-metadata Digest, Vol 75, Issue 2
*******************************************

No time like the present if you ask me, and why force adoption? Shouldn't
this be all optional? If you want to use lzma give the option of --lzma,
same for xz or bz2 and gz. Personally I'd like to see xz as that's what we
use for all our compression. I just said lzma b/c I saw it kinda being
worked on.

The same with forcing the adoption of the sqlitedb, why? Just because
fedora/yum use it??? That's really weak logic. Again, I would say it's all
about options. Let the repo provider decide.

Regards,
Matt