repository isolation

[Jon Wallace]

Hello,

Are repositories isolated entities? I'm trying to determine if there is
any way that createrepo allows dependencies at the repository level. In
other words, using Repository-B would have a dependency on the existance
of Repository-A and the content of Repository-A would be available as well.

Jon

[Anders F Björklund]

Are repositories isolated entities? I'm trying to determine if there is any way that createrepo allows dependencies at the repository level. In other words, using Repository-B would have a dependency on the existance of Repository-A and the content of Repository-A would be available as well.

If you are using "release" RPMS to distribute your .repo and keys, you could have repository-b-release require repository-a-release. But otherwise that would normally happen between the packages ?

For example the "updates" repository packages usually have dependencies from "base" repository, but there is no explicit dependency between the repositories. (and they both ship as one "release")

--anders

[James Antill]

Post by Jon Wallace
Hello,
Are repositories isolated entities? I'm trying to determine if there is
any way that createrepo allows dependencies at the repository level. In
other words, using Repository-B would have a dependency on the existance
of Repository-A and the content of Repository-A would be available as well.

There has been talk of doing dependency/grouping information within yum
using repomd.xml tags, see Eg.

https://fedorahosted.org/rel-eng/ticket/3759

...and I believe that zypper does something similar. So you could
require a feature like that, from the package manager.

Apart from that you could use requires/conflicts on the -release
packages, as Anders suggested ... but that doesn't guarantee much.

[seth vidal]

Post by Jon Wallace
Hello,
Are repositories isolated entities? I'm trying to determine if there is
any way that createrepo allows dependencies at the repository level. In
other words, using Repository-B would have a dependency on the existance
of Repository-A and the content of Repository-A would be available as well.

To twist this around - what's the problem with having RepoA and RepoB
available? Why do you need B to pull in A? Why not just have them both
enabled? OR is this a case of wanting an infinite set of interdependent
repos and wanting them to be enabled dynamically based on what repos are
explicitly enabled?

If so - then this is not really something that createrepo can do - it
has to be handled in the pkg manager using the repodata: smart, yum,
zypper, etc.


It might be worth describing the actual use case you're interested in -
including the scale of the repos (how many, how interdependent, how
complex) and maybe we can come up with a solution.

I can think of a few ways off the top of my head using yum plugins.

-sv

[Jon Wallace]

Post by seth vidal

Post by Jon Wallace
Hello,
Are repositories isolated entities? I'm trying to determine if there is
any way that createrepo allows dependencies at the repository level. In
other words, using Repository-B would have a dependency on the existance
of Repository-A and the content of Repository-A would be available as well.

To twist this around - what's the problem with having RepoA and RepoB
available? Why do you need B to pull in A? Why not just have them both
enabled? OR is this a case of wanting an infinite set of interdependent
repos and wanting them to be enabled dynamically based on what repos are
explicitly enabled?
If so - then this is not really something that createrepo can do - it
has to be handled in the pkg manager using the repodata: smart, yum,
zypper, etc.
It might be worth describing the actual use case you're interested in -
including the scale of the repos (how many, how interdependent, how
complex) and maybe we can come up with a solution.
I can think of a few ways off the top of my head using yum plugins.
-sv

The intent is to understand limitations at a repository level and
whether we can think of the repository as part of a "bundle". So one
example might be that you have one repository called PRODUCT_A and
another repository called PRODUCT_B. Now for customer purposes you
might create three "bundles". One would be PRODUCT_A by itself, another
would be PRODUCT_B by itself, and you might have another bundle called
"EVERYTHING" which would provide you software from both PRODUCT_A and
PRODUCT_B.

A different example would be trying to limit the size of repositories
that have similar packages. So if PRODUCT_A and PRODUCT_B both need the
same rpm package (xyz.rpm) it would be nice if the physical rpm file
only had to live in one place and could be referenced from any other
number of repositories, similar to how symbolic links work.

Jon

[James Antill]

Post by Jon Wallace
The intent is to understand limitations at a repository level and
whether we can think of the repository as part of a "bundle".

You can _think_ of them that way, and a number of them work that
way ... but there is no std. way to enforce that. As I said, I know of a
few conversations that have been had about adding some functionality
like that to yum ... but the details are non-trivial, esp. as you try to
solve for all the different reasons people want "bundles".

Eg.

blah <ver 2.1>
blah-updates <ver 2.1> (requires blah = $ver)
blah-updates-testing <ver 2.1> (requires blah-updates = $ver)

blah-clustering <ver 1.4> (requires blah >= 1.0)
blah-clustering-updates (requires blah-clustering = $ver)

blah-ha-db <ver 0.1> (requires blah >= 2.1 _and_
requires blah-clustering >= 1.4)

blah-source (relates to blah = $ver)

blah-debuginfo (requires blah/blah-updates = $ver)

...as you can see this quickly leads to needing a "mini. depsolver" to
work out which repos. you want/need (and after thinking about it for a
non-trivial amount of time, I'm still not sure what to do when
blah-ha-db ups it's requirement to blah 2.2 ... but you can't get blah
2.2 after a cache expire).

Then there's the problem of where to put the data. The obvious
candidate is within the repomd.xml (Eg. as tags) ... but that means you
have to download the repomd.xml to even see the data, which causes
problems (as you can't just randomly refresh repos. ... and even
randomly turning them on/off can be confusing).
The next obvious choice is the .repo files, but that means the data is
fixed basically "forever". Something in the middle using "yum vars"
might be doable, but while that is updateable it requires package
updates ... which is "not-good" at best.

And that's just the yum POV.

The general solution of: Have it all be implicit, and don't screw it up
too much ... still seems to be working, mostly.

Post by Jon Wallace
A different example would be trying to limit the size of repositories
that have similar packages. So if PRODUCT_A and PRODUCT_B both need the
same rpm package (xyz.rpm) it would be nice if the physical rpm file
only had to live in one place and could be referenced from any other
number of repositories, similar to how symbolic links work.

This is already possible:

mergerepo --repo=http://.../Fedora/releases/13/Everything/x86_64/os \
--repo=http://.../Fedora/updates/13/x86_64/

...will produce a new repo. which just has repodata in it (all packages
served from the specified mirrors). Obviously you lose the full mirror
capability, which isn't nice ... and for your case you'd need to create
5 repos., 3 "internal" and 2 "external".
Adding features to createrepo, or mergerepo is possible though.

[Jon Wallace]

Thanks for the tip about mergerepo which sounds like it is something I
can use. However, testing it just now I am getting errors and don't
know where I am going wrong.

1) I copied all the rpms from Fedora 14 to a directory called Package_1
2) Made a copy of Package_1 and called it Package_2.
3) Deleted all xorg rpm packages from Package_1.
4) Deleted everything BUT xorg rpm packages from Package_2.
5) Ran createrepo on Package_1 and Package_2.

When I run mergerepo I get the following errors.

mergerepo --repo=file:///a/b/c/d/Packages_1
--repo=file:///a/b/c/d/Packages_2 --nogroups -o merged1
Loaded plugins: refresh-packagekit
3/2766 - chkconfig-1.3.47-1.fc14.x86_64Traceback (most recent call last):
File "/usr/share/createrepo/mergerepo.py", line 84, in <module>
main(sys.argv[1:])
File "/usr/share/createrepo/mergerepo.py", line 81, in main
rmbase.write_metadata()
File "/usr/lib/python2.6/site-packages/createrepo/merge.py", line
135, in write_metadata
mdgen.doPkgMetadata()
File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line
364, in doPkgMetadata
self.writeMetadataDocs(packages)
File "/usr/lib/python2.6/site-packages/createrepo/__init__.py", line
527, in writeMetadataDocs
self.primaryfile.write(po.xml_dump_primary_metadata())
File "/usr/lib/python2.6/site-packages/yum/packages.py", line 1015,
in xml_dump_primary_metadata
msg += misc.to_unicode(self._dump_format_items())
File "/usr/lib/python2.6/site-packages/yum/packages.py", line 894, in
_dump_format_items
msg += self._dump_pco('provides')
File "/usr/lib/python2.6/site-packages/yum/packages.py", line 919, in
_dump_pco
msg += pcostring
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 32:
ordinal not in range(128)


Any suggestions?
Thanks,
Jon