[Yum-devel] Implementing delta metadata

[James Antill]

Hi!
I frequently face this problem with yum where i need to install a
package, and it starts updating all the repo data at once. Since I am
mostly behind a slow internet connection, this is all the more
frustrating. I read up http://yum.baseurl.org/wiki/YumFuture , and it
says here that there are plans to serve metadata delta to reduce the
download. Also, the large databases can be intelligently broken up
into smaller pieces (easier to download), so that when i try to
install a package starting with 'a', only the repodata for all the
packages starting with 'a' gets downloaded and synced [this was a
crude example, but conveys the idea].

Right, the main page for the new repodata ideas was/is:

http://yum.baseurl.org/wiki/dev/NewRepoDataIdeas

...there was also some discussion on this list (6+ months ago). You
probably want to come by IRC and speak to people there.

[Tirtha Chatterjee]

Hi!

I frequently face this problem with yum where i need to install a
package, and it starts updating all the repo data at once. Since I am
mostly behind a slow internet connection, this is all the more
frustrating. I read up http://yum.baseurl.org/wiki/YumFuture , and it
says here that there are plans to serve metadata delta to reduce the
download. Also, the large databases can be intelligently broken up
into smaller pieces (easier to download), so that when i try to
install a package starting with 'a', only the repodata for all the
packages starting with 'a' gets downloaded and synced [this was a
crude example, but conveys the idea].

I am willing to work on both of these features. I have read the source
code of createrepo and yum roughly, and am confident I'll be able to
make this happen. I'd be glad if someone with ample experience in the
codebase steps forward to guide me.

Also, comments / feedback is invited. Please reply.

[Michael Schroeder]

I frequently face this problem with yum where i need to install a
package, and it starts updating all the repo data at once. Since I am
mostly behind a slow internet connection, this is all the more
frustrating. I read up http://yum.baseurl.org/wiki/YumFuture , and it
says here that there are plans to serve metadata delta to reduce the
download. Also, the large databases can be intelligently broken up
into smaller pieces (easier to download), so that when i try to
install a package starting with 'a', only the repodata for all the
packages starting with 'a' gets downloaded and synced [this was a
crude example, but conveys the idea].

Uh, that won't work, as the package you want to install may require
something a different package provides.

I am willing to work on both of these features. I have read the source
code of createrepo and yum roughly, and am confident I'll be able to
make this happen. I'd be glad if someone with ample experience in the
codebase steps forward to guide me.

openSUSE uses a zsync algorithm to download only the "changed" parts
of the repodata. (The repodata needs to be compressed with
--rsyncable in that case, of course).

I don't know if this is what you're aiming for, but createrepo
could be changed to also create the needed .zsync files. (For
openSUSE, the files aren't needed as the data is provided in
the metalink files download.opensuse.org offers.)

Cheers,
Michael.