metadata layout problems and some history

[seth vidal]

Hi folks,
The discussion of future changes to the repodata may be a bit sideways,
so let's talk about it from another direction - that of the history of
the rpm-md format:

In 2003 there were many different formats for repositories of pkgs. Yum
had one, Apt had two (apt-deb and apt-rpm+bloat for the filelists),
up2date had rhn's server doing depsolving/listing of pkgs, ximian had
one for red carpet, there were a bunch. The goal of the rpm-metadata
discussions were to have a single format everyone could use so that all
tools could access the other repos. I headed this up and tried to corral
people to talk about it and had some amount of success. Out of it came
roughly what you know today as the format for the xml files in a repo.

There have been some small changes over the years but I believe none
that have broken the parsers of any of tools.

Then what happened was all of the folks using/maintaining the tools
found issues they had to work around and so we did:

- Yum eventually started generating sqlite dbs of the same data that was
in the xml files to speed up access to the data.

- Zypper uses .solv files to do much the same thing for the satsolver
code.

- Smart runs an indexer over the xml files to generate the byte-offset
locations of pkgs in the xml file so it can seek to those locations
quickly.


We all worked around the problems but kept the basic compat with the
repo format. So, we achieved the original goal, right? Seems like we
did.



So, with that in mind - I was hoping we could take this opportunity
(since I saw a bunch of people just subscribed to the list) to discuss
the problems we've had to work around and if we can make a new format
that can:

1. cover new changes to rpm packages

2. make our workarounds less necessary/less painful

3. work with new realities of interoperating with MANY MANY repos

4. handle things that were never included in the original specification
we've later found out would be useful.


Think of this as compiling a list of issues to be solved with a Version2
of things.


Ideas?

Thanks,
-sv

[seth vidal]

Post by seth vidal
So, with that in mind - I was hoping we could take this opportunity
(since I saw a bunch of people just subscribed to the list) to discuss
the problems we've had to work around and if we can make a new format
1. cover new changes to rpm packages
2. make our workarounds less necessary/less painful
3. work with new realities of interoperating with MANY MANY repos
4. handle things that were never included in the original specification
we've later found out would be useful.
Think of this as compiling a list of issues to be solved with a Version2
of things.

Problems yum/fedora have had:
1. filelists are important to us b/c we have filedeps on pkgs - with
this in mind we need to further break out the filelists so a person who
is installing a pkg which requires a file outside of the ones included
in primary ends up downloading a big file from a lot of locations

2. translation information needed for summary and description.

3. potential for explosion in the provides/requires listing as per-pkg
find-prov/find-reqs become more common - essentially dealing with A LOT
more provides/reqs than we're used to - maybe as much as 1 or 2 orders
of magnitude.

4. allowing users to get partial updates to metadata easily

5. updating metadata in place w/pkgs removed/added (part of this is the
fault of createrepo)

6. softdeps and where they should be stored.

7. An occasionally-requested feature is to have the repodata also
contain (somewhere) the scriptlets (%pre, %post, %triggers, etc) of all
the pkgs - package collections as has recently been discussed on
rpm-maint will come into play there, too.

8. repository 'expiration'

9. fast random access to the metadata for searches from a variety of
tools - not just yum based ones.

10. repo labeling and repo 'dependencies' to 'help' in determining if
two repos have pkgs which can interoperate work with one another.


-sv

[Matt Barringer]

Post by seth vidal
7. An occasionally-requested feature is to have the repodata also
contain (somewhere) the scriptlets (%pre, %post, %triggers, etc) of all
the pkgs - package collections as has recently been discussed on
rpm-maint will come into play there, too.
8. repository 'expiration'
10. repo labeling and repo 'dependencies' to 'help' in determining if
two repos have pkgs which can interoperate work with one another.

I would love to see these three features. But isn't repo labeling already
covered by the <repo> tag? Something simple like <base_system>repo
tag</basesystem> would be nice for giving an idea of whether or not a
repository can work with a certain platform.

--
Matt Barringer, Software Engineer
SUSE Linux Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
http://susestudio.com http://www.novell.com/linux

[seth vidal]

Post by Matt Barringer

Post by seth vidal
7. An occasionally-requested feature is to have the repodata also
contain (somewhere) the scriptlets (%pre, %post, %triggers, etc) of all
the pkgs - package collections as has recently been discussed on
rpm-maint will come into play there, too.
8. repository 'expiration'
10. repo labeling and repo 'dependencies' to 'help' in determining if
two repos have pkgs which can interoperate work with one another.

I would love to see these three features. But isn't repo labeling already
covered by the <repo> tag? Something simple like <base_system>repo
tag</basesystem> would be nice for giving an idea of whether or not a
repository can work with a certain platform.

you can munge repo labeling with tags, yes - but it's very freeform.

We have architectures in there so you can glance at that list and see
whether or not you even need to go on. If it's an all-ppc repo and
you're on i686 it's kinda pointless. :)

-sv

[Matt Barringer]

Post by seth vidal

Post by Matt Barringer
I would love to see these three features. But isn't repo labeling already
covered by the <repo> tag? Something simple like <base_system>repo
tag</basesystem> would be nice for giving an idea of whether or not a
repository can work with a certain platform.

you can munge repo labeling with tags, yes - but it's very freeform.
We have architectures in there so you can glance at that list and see
whether or not you even need to go on. If it's an all-ppc repo and
you're on i686 it's kinda pointless. :)

Sorry, by "platform" I meant the base distribution the packages were built
for, not the architecture. In SUSE Studio, we have users that try to
mix-and-match packages from creative repository combinations like Fedora,
Packman, and multiple versions of openSUSE, which doesn't really work out
in the dependency resolution stage. We try to guess at what the base
distro is based on the URL, but it's an inexact method, so having a more
explicit dependency tag in the metadata would be great.

--
Matt Barringer, Software Engineer
SUSE Linux Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
http://susestudio.com http://www.novell.com/linux

[seth vidal]

On Fri, 2010-08-06 at 16:46 -0400, seth vidal wrote:

As a reminder - something I wrote up a couple of months ago with regards
to how yum might benefit:

http://yum.baseurl.org/wiki/dev/NewRepoDataIdeas

-sv

[Michael Schroeder]

Post by seth vidal
Then what happened was all of the folks using/maintaining the tools
- Yum eventually started generating sqlite dbs of the same data that was
in the xml files to speed up access to the data.
- Zypper uses .solv files to do much the same thing for the satsolver
code.

Yes, but we use them internally and don't download them from the
server. It works for us because xml -> solv conversion doesn't take
much time:

$ ls -l fedora12upprimary.xml.gz
-rw-r--r-- 1 mls suse 2843370 Aug 7 17:25 fedora12upprimary.xml.gz
$ gunzip < fedora12upprimary.xml.gz | (time repomdxml2solv > /dev/null)
0.896u 0.062s 0:00.96 98.9% 0+0k 0+0io 0pf+0w

So it takes about a second to convert the 2.8 Mbytes xml file to
solv. I guess creating the sqlite database is a bit slower, so you
chose to do it on the server and not on the client.

As parsing the data file is not the bottleneck for us, anything that
reduces the transfer size is a good thing. A switch to sqlite
would hurt us, as the sqlite database is currently bigger than the
compressed xml. (This is probably worse if lzma is used
instead of gzip.)

(I have no e-mail access for the next two weeks. Klaus and Duncan,
could you please answer any questions?)

Cheers,
Michael.

[seth vidal]

Post by Michael Schroeder
So it takes about a second to convert the 2.8 Mbytes xml file to
solv. I guess creating the sqlite database is a bit slower, so you
chose to do it on the server and not on the client.

As the size of this file increases how much time is eaten up? You showed
a f12 updates repo, it seems.

Fedora 13 GA for x86_64 - the compressed primary.xml.gz is 7.9M.

So does that mean the solv-creation is going to take ~3s?

I don't see the number of pkgs decreasing at any point in time and since
we were pushing fedora onto some REALLY underpowered boxes we found the
xml->sqlite conversion can SUCK on some machines (OLPC XO-1's for
example were horrendous)

Post by Michael Schroeder
As parsing the data file is not the bottleneck for us, anything that
reduces the transfer size is a good thing. A switch to sqlite
would hurt us, as the sqlite database is currently bigger than the
compressed xml. (This is probably worse if lzma is used
instead of gzip.)

Do you end up storing the xml AND the solv files in the cache dir? I ask
b/c this was another reason for doing server-side-generated sqlite b/c
we were keeping both in the cache and, again, on the XO-1 and other
quasi-small-disk systems (and most importantly SSDs) we were trying to
keep the disk writes and disk use as low as possible.

Post by Michael Schroeder
(I have no e-mail access for the next two weeks. Klaus and Duncan,
could you please answer any questions?)

Ahh - a wish to have no email for two weeks. :)

Enjoy your vacation.

-sv

[Klaus Kaempf]

Post by seth vidal

Post by Michael Schroeder
So it takes about a second to convert the 2.8 Mbytes xml file to
solv. I guess creating the sqlite database is a bit slower, so you
chose to do it on the server and not on the client.

As the size of this file increases how much time is eaten up? You showed
a f12 updates repo, it seems.
Fedora 13 GA for x86_64 - the compressed primary.xml.gz is 7.9M.
So does that mean the solv-creation is going to take ~3s?

Yes. Compared to all the other stuff (xml download, reading the rpm
db), solv-creation is almost neglectable.

Post by seth vidal
I don't see the number of pkgs decreasing at any point in time and since
we were pushing fedora onto some REALLY underpowered boxes we found the
xml->sqlite conversion can SUCK on some machines (OLPC XO-1's for
example were horrendous)

Agreed. Thats why we were also thinking about making cached metadata
(solv files) available in the repository in addition to the 'pristine'
xml data.

Post by seth vidal

Post by Michael Schroeder
As parsing the data file is not the bottleneck for us, anything that
reduces the transfer size is a good thing. A switch to sqlite
would hurt us, as the sqlite database is currently bigger than the
compressed xml. (This is probably worse if lzma is used
instead of gzip.)

Do you end up storing the xml AND the solv files in the cache dir?

Yes.

Post by seth vidal
I ask b/c this was another reason for doing server-side-generated
sqlite b/c we were keeping both in the cache and, again, on the XO-1
and other quasi-small-disk systems (and most importantly SSDs) we were
trying to keep the disk writes and disk use as low as possible.

I see.
But wouldn't operating on a data base actually require more disk i/o ?

Parsing xml to solv is one streamed read (of the xml) and one streamed
write (of the solv). Reading the solv is similar, only a couple of
mallocs and reading in big chunks.


Klaus
---
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)

[Duncan Mac-Vicar P.]

Post by seth vidal

Post by Michael Schroeder
So it takes about a second to convert the 2.8 Mbytes xml file to
solv. I guess creating the sqlite database is a bit slower, so you
chose to do it on the server and not on the client.

As the size of this file increases how much time is eaten up? You showed
a f12 updates repo, it seems.

Our updates repo also increases its size. I haven't noticed that the
performance goes in any annoying way. At least you can't notice it
because download is what takes most time.

Post by seth vidal
Fedora 13 GA for x86_64 - the compressed primary.xml.gz is 7.9M.
So does that mean the solv-creation is going to take ~3s?
I don't see the number of pkgs decreasing at any point in time and since
we were pushing fedora onto some REALLY underpowered boxes we found the
xml->sqlite conversion can SUCK on some machines (OLPC XO-1's for
example were horrendous)

As Michael said, for xml->solv you can't almost notice it. I have used
it on netbooks and the performance is still very good.

Post by seth vidal
Do you end up storing the xml AND the solv files in the cache dir? I ask
b/c this was another reason for doing server-side-generated sqlite b/c
we were keeping both in the cache and, again, on the XO-1 and other
quasi-small-disk systems (and most importantly SSDs) we were trying to
keep the disk writes and disk use as low as possible.

Yes, we have /var/cache/zypp/raw and /var/cache/zypp/solv

Duncan

[Anders F Björklund]

Post by seth vidal
There have been some small changes over the years but I believe none
that have broken the parsers of any of tools.

The biggest changes for smart was the change to the sha256 checksums
and unique-md-filenames which meant old cache files weren't deleted.

Post by seth vidal
Then what happened was all of the folks using/maintaining the tools
- Smart runs an indexer over the xml files to generate the byte-offset
locations of pkgs in the xml file so it can seek to those locations
quickly.

Except that quick-er doesn't mean that it still wasn't dog slow...

"Well, at least having an index cuts list time from
30 minutes to 30 seconds. On the other hand, that is
still ridiculously slow compared to the usual 3...
So I don't think the cache size/startup time matters ?"

So neither metadata nor sqlite has been merged into the trunk.
It's all being stored in the cache, instead of using XML/SQL.

--anders

[seth vidal]

Post by Anders F Björklund

Post by seth vidal
There have been some small changes over the years but I believe none
that have broken the parsers of any of tools.

The biggest changes for smart was the change to the sha256 checksums
and unique-md-filenames which meant old cache files weren't deleted.

Post by seth vidal
Then what happened was all of the folks using/maintaining the tools
- Smart runs an indexer over the xml files to generate the byte-offset
locations of pkgs in the xml file so it can seek to those locations
quickly.

Except that quick-er doesn't mean that it still wasn't dog slow...
"Well, at least having an index cuts list time from
30 minutes to 30 seconds. On the other hand, that is
still ridiculously slow compared to the usual 3...
So I don't think the cache size/startup time matters ?"
So neither metadata nor sqlite has been merged into the trunk.
It's all being stored in the cache, instead of using XML/SQL.

So help me understand what would make your situation better? What would
make the rpm-md less problematic for smart?

-sv

[Anders F Björklund]

Post by seth vidal

Post by Anders F Björklund

Post by seth vidal
- Smart runs an indexer over the xml files to generate the byte-
offset
locations of pkgs in the xml file so it can seek to those locations
quickly.

Except that quick-er doesn't mean that it still wasn't dog slow...

So help me understand what would make your situation better? What would
make the rpm-md less problematic for smart?

It seems like the best is to use a custom second-level cache.

The parsing of the XML, or the connection to the SQL, is still
more troublesome to maintain. Especially if it might change...

Currently "info" contains:
location
build_time
installed_size
size
md5
sha
sha256
description
summary
url
sourcerpm
group
license

That's the fields that aren't needed in the first-level cache.

For rpm headerlists or deb tagfiles we keep the file offsets,
but this doesn't translate to repodata so using the yum pkgid.


But I wouldn't say it's "problematic". Missing rpm groups or
required yum comps might be, but not all rpm distros have it.

Only other changes were the requests from distros using smart,
such as missing fields in the markup or better compression...

--anders

[Panu Matilainen]

Post by seth vidal

Post by Anders F Björklund

Post by seth vidal
There have been some small changes over the years but I believe none
that have broken the parsers of any of tools.

The biggest changes for smart was the change to the sha256 checksums
and unique-md-filenames which meant old cache files weren't deleted.

Post by seth vidal
Then what happened was all of the folks using/maintaining the tools
- Smart runs an indexer over the xml files to generate the byte-offset
locations of pkgs in the xml file so it can seek to those locations
quickly.

Except that quick-er doesn't mean that it still wasn't dog slow...
"Well, at least having an index cuts list time from
30 minutes to 30 seconds. On the other hand, that is
still ridiculously slow compared to the usual 3...
So I don't think the cache size/startup time matters ?"
So neither metadata nor sqlite has been merged into the trunk.
It's all being stored in the cache, instead of using XML/SQL.

So help me understand what would make your situation better? What would
make the rpm-md less problematic for smart?

AFAICT smart, zypp an apt all would just want a raw dump of the relevant
data in a format that's lightweight to read in once to generate their own
native data, and at least in case of smart and apt, can be indexed (eg
jump to file offset X to access the details that aren't included in the
package cache). Yes, dejavu on yum-metadata-parser, which was the yum
equivalent of "internal cache creation", only the sqlite format it creates
got added (as an extension) to repomd.

XML works fine for the raw dump part, except that it's hideously bloated
format for what it's used for and expensive to parse (dejavu
yum-metadata-parser again), and isn't good for indexing or searching.
Heck, a dumb tagged plain-text file would be much better suited for apt
(and I assume smart). I've vague memories of Suse actually having such a
format at some point before switching to repodata.

- Panu -

[Klaus Kaempf]

Hi Panu,

long time no see :-)

* Panu Matilainen <***@laiskiainen.org> [Aug 12. 2010 12:39]:
[...]

Post by Panu Matilainen
XML works fine for the raw dump part, except that it's hideously
bloated format for what it's used for and expensive to parse (dejavu
yum-metadata-parser again), and isn't good for indexing or
searching.
Heck, a dumb tagged plain-text file would be much better suited for
apt (and I assume smart).

+1

Post by Panu Matilainen
I've vague memories of Suse actually having such a format at some point
before switching to repodata.

SUSE had 'common.pkd' years ago and still uses 'susetags' for the
on-media metadata.

Both are plain text dumps with the possibility to include translations.
Nothing to gain here from a parsing effort perspective.

I'd rather see a future 'createrepo' format to put some computing
effort at the creation of the metadata in order to relief all clients
from repeating the work. String indexing would be the prime example.

Klaus
---
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)

[seth vidal]

Post by Panu Matilainen

Post by seth vidal
So help me understand what would make your situation better? What would
make the rpm-md less problematic for smart?

XML works fine for the raw dump part, except that it's hideously bloated
format for what it's used for and expensive to parse (dejavu
yum-metadata-parser again), and isn't good for indexing or searching.
Heck, a dumb tagged plain-text file would be much better suited for apt
(and I assume smart). I've vague memories of Suse actually having such a
format at some point before switching to repodata.

Yes, that was my point in starting this discussion.

But what I've been hearing so far is "don't change anything, just let it
be".

-sv

[Klaus Kaempf]

Post by seth vidal

Post by Panu Matilainen

Post by seth vidal
So help me understand what would make your situation better? What would
make the rpm-md less problematic for smart?

XML works fine for the raw dump part, except that it's hideously bloated
format for what it's used for and expensive to parse (dejavu
yum-metadata-parser again), and isn't good for indexing or searching.
Heck, a dumb tagged plain-text file would be much better suited for apt
(and I assume smart). I've vague memories of Suse actually having such a
format at some point before switching to repodata.

Yes, that was my point in starting this discussion.
But what I've been hearing so far is "don't change anything, just let it
be".

Thats not what you should hear, it should rather be "Don't hurry the
change".

Klaus
---
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)

[Duncan Mac-Vicar P.]

Post by seth vidal
Yes, that was my point in starting this discussion.
But what I've been hearing so far is "don't change anything, just let it
be".
-sv

No way. I fully support creating a better format.

I only oppose using a specific implementation as the "standard" because
it makes one tool life easier. Panu described the situation very well.

For example, even if for ZYpp searching is not an issue (as all our
search is implemented on top of the solv attributes API, which provides
query, and it is fast enough), I fully support a metadata format that
allows easier searching. (old SUSE format had offsets for package
descriptions for example, so they culd go and grab them direclty from
disk at runtime).

My point is. A standard should have as a goal to make every tool life
good enough. At the same time, make the rpm world interoperable (I am
pretty sure we have *SUSE users using yum), while still allowing for
innovation and tool-specific techniques (satsolver/solv files in our case).

Having something "common" means more work too. It means you can't just
add a tag or name something without asking for some input about how it
would affect other tools. But I see as an investment. (example: I am
pretty sure if a more "standard" model for rpmmd would have been in
place, the "sha" tag would have been named "sha1" and the pkgid=YES tag
would not had been agreed upon)

At the same time, standards add bloat and reduce agility, so it is
important to allow for extensions. We have one extension file for repo
information (repomd.xml) and another for package data (primary.xml). We
add unofficial tags there, and do not move them unless agreement has
been made. Until now, only example is the tags information and the
expiration of metadata. But a place for incubation allows to quickly try
new concepts before those long discussions about naming and how to solve
a problem, end.

Duncan

[Anders F Björklund]

Post by Panu Matilainen
AFAICT smart, zypp an apt all would just want a raw dump of the
relevant data in a format that's lightweight to read in once to
generate their own native data, and at least in case of smart and
apt, can be indexed (eg jump to file offset X to access the details
that aren't included in the package cache). Yes, dejavu on yum-
metadata-parser, which was the yum equivalent of "internal cache
creation", only the sqlite format it creates got added (as an
extension) to repomd.

For smart the "easiest" is to use shelve, which is pickle + anydbm.
It is indexed and loadable, but it is also specific to python...

So the repodata is parsed, XML or SQL doesn't matter all that much,
and an internal cache is created. (Ultimately using Python objects.)

Indexed rpm metadata isn't needed for the package dependency solving,
but needed later when requiring info or pathlists or changelog data.

Both indexing XML and querying SQL "worked" in that they saved having
to do yet another copy of the repodata. (Smart doesn't create repos.)

Post by Panu Matilainen
XML works fine for the raw dump part, except that it's hideously
bloated format for what it's used for and expensive to parse
(dejavu yum-metadata-parser again), and isn't good for indexing or
searching.
Heck, a dumb tagged plain-text file would be much better suited for
apt (and I assume smart). I've vague memories of Suse actually
having such a format at some point before switching to repodata.

Yes, the deb tagfiles are doing better than rpm metadata, for smart.
And the rpm headerlists (like apt had) were "better" for indexing etc.

I think urpmi currently has a mix of RPM *and* XML for the media_info,
but don't have all the details as it hasn't been merged in smart yet:

MD5SUM
hdlist.cz
synthesis.hdlist.cz
info.xml.lzma
files.xml.lzma
changelog.xml.lzma

Except for the difference in key used (yum "pkgid" vs. urpmi "fn"),
I think non-primary metadata is rather similar. The primary differ.

But for Smart it all starts with the packages, except when talking
about add-on data such as comps/tasks or update_info/descriptions.

--anders