Zypper data probably corrupted, how to re-create?

On Tue, Apr 20, 2010 at 9:05 PM, David C. Rankin

On Tue, Apr 20, 2010 at 7:54 PM, David C. Rankin

Mark, I noticed the exact same problem with KDE:KDE3 here, been through
cache cleanup et al., and problem persists.
While I have no solution for this issue, here are some more more or less
educated guesses, hence the crosspost to buildservice.
Mark and me have problems with certain repos from build service for 11.1,
where zypper doesn't see any packages from that repo.
# zyp se -s -r KDE_KDE3
Verbosity: 1
Initializing Target
Checking whether to refresh metadata for Latest KDE 3 environment for
released SUSE distributions (openSUSE_11.1)
Retrieving: repomd.xml [done]
Retrieving: repomd.xml [done]
Retrieving: repomd.xml.asc [done]
Retrieving: repomd.xml.key [done]
Retrieving: repomd.xml [done]
Retrieving: primary.xml.gz [done]
Retrieving repository 'Latest KDE 3 environment for released SUSE
distributions (openSUSE_11.1)' metadata [done]
Building repository 'Latest KDE 3 environment for released SUSE
distributions (openSUSE_11.1)' cache [done]
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256
- [|] Failed to cache repo (4).
Warning: Disabling repository 'Latest KDE 3 environment for released SUSE
distributions (openSUSE_11.1)' because of the above error.
Loading repository data...
Reading installed packages...
Force resolution: No
No packages found.
Ahh, zypper chokes on sha256.
If I compare these repos with those that operate correctly, there are
significant differences in the repomd.xml. Here I'm going to compare the
<data type="primary">
<location href="repodata/primary.xml.gz"/>
<checksum
type="sha">57ac66c34bdf5c44fd2cdb00fb07620706f0c4f8</checksum>
<timestamp>1271759831</timestamp>
<open-checksum
type="sha">2cb77a5d202f6572e86a2027637bb682b832b20f</open-checksum>
</data>
<data type="primary">
<checksum
type="sha256">1805495e28cdff0b638fb19db31091045be88eae2f48ba2ab8ee845bfe320
3bb</checksum> <timestamp>1271737040</timestamp>
<size>408803</size>
<open-size>3136333</open-size>
<open-checksum
type="sha256">3561980446cb496f46c9b88d6fd0af34de6154060c7ff81c8b87467abd58e
08c</open-checksum> <location
href="repodata/1805495e28cdff0b638fb19db31091045be88eae2f48ba2ab8ee845bfe32
03bb-primary.xml.gz"/> </data>
Indeed, checksum is sha256 now.
# grep 'type="sha256"' $(find /var/cache/zypp/raw -name repomd.xml)
results in KDE_KDE3, devel_tools, openSUSE_Factory_Contrib, and
openSUSE_Tools_Unstable repos with sha256 checksums.
# zyp se -s -r openSUSE_Tools_Unstable
Verbosity: 1
Initializing Target
Checking whether to refresh metadata for Stabilizing version of the
openSUSE Tools (openSUSE_11.1)
Loading repository data...
Reading installed packages...
Force resolution: No
No packages found.
Well, it should.
# zyp se -s -r KDE4_11.1
Verbosity: 1
Initializing Target
Checking whether to refresh metadata for KDE 4.3.5 (openSUSE_11.1)
Loading repository data...
Reading installed packages...
Force resolution: No
S | Name | Type |
Version | Arch | Repository
--+--------------------------------------+------------+--------------------
---+--------+--------------------------
| KDE 4 Devel |
pattern | | noarch | KDE 4.3.5 (openSUSE_11.1)
| KDE 4 Games |
pattern | | noarch | KDE 4.3.5 (openSUSE_11.1)
i | KDE 4.1 Basis |
pattern | | noarch | KDE 4.3.5 (openSUSE_11.1)
| NetworkManager-kde4 | package |
0.9.svn1043876-4.10 | i586 | KDE 4.3.5 (openSUSE_11.1)
[...]
Expected output.
The question is, how do we trick BS, namely publisher to generate palatable
repo descriptions for our systems?
Of course, we could mirror the offending repos locally, and run createrepo
on them :-( , but there ought to be an easier way, doesn't it?
TIA,
Pete

FWIW, I am seeing the same issues on 11.1 i586 here: e.g:

'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

amd:/home/mrdocs # zypper in kde3
Building repository 'Tools' cache [done]
Error building the cache:
'repo2solv.sh' '-o' '/var/cache/zypp/solv/20090103-003115/solv'
'/var/cache/zypp/raw/20090103-003115'
Unknown checksum type: 8: sha256

Warning: Disabling repository 'Tools' because of the above error.
Retrieving repository 'KDE3' metadata [done]
Building repository 'KDE3' cache [done]
Error building the cache:
'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE3/solv'
'/var/cache/zypp/raw/KDE3'
Unknown checksum type: 8: sha256

Warning: Disabling repository 'KDE3' because of the above error.
Retrieving repository 'Qt45-home' metadata [done]
Building repository 'Qt45-home' cache [done]
Error building the cache:
'repo2solv.sh' '-o' '/var/cache/zypp/solv/Qt45-home/solv'
'/var/cache/zypp/raw/Qt45-home'
Unknown checksum type: 8: sha256

HTH,
Peter

--
To unsubscribe, e-mail: opensuse-buildservice+***@opensuse.org
For additional commands, e-mail: opensuse-buildservice+***@opensuse.org

Marcus Rueckert

2010-04-20 22:18:58 UTC

someone needs to trigger publishing for the affect repositories.
adrian could we do that on our side to get them all fixed at once?

darix

--
openSUSE - SUSE Linux is my linux
openSUSE is good for you
www.opensuse.org

Hans-Peter Jansen

2010-04-20 22:50:36 UTC

Post by Marcus Rueckert
someone needs to trigger publishing for the affect repositories.
adrian could we do that on our side to get them all fixed at once?

Marcus, yeah, that would be fine. But what does prevent the regeneration of
the issues again and again...

Did somebody fix this issue meanwhile?

Thanks,
Pete

--
To unsubscribe, e-mail: opensuse-buildservice+***@opensuse.org
For additional commands, e-mail: opensuse-buildservice+***@opensuse.org

Michael Schroeder

2010-04-21 08:48:50 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv' '/var/cache/zypp/raw/KDE_KDE3'
Unknown checksum type: 8: sha256

Oops, we updated the createrepo package on the build service host
to use some new features, and now all of the repos use sha256 checksums
instead of sha1. Unfortunatelly libzypp in 11.1 and older can't
handle that checksum (yet).
We now use a createrepo option to use sha1 again for older repos,
so things should work again for all freshly created build service
repositories.

Sorry for the inconvenience.

Cheers,
Michael.

--
Michael Schroeder ***@suse.de
SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
--
To unsubscribe, e-mail: opensuse-buildservice+***@opensuse.org
For additional commands, e-mail: opensuse-buildservice+***@opensuse.org

Hans-Peter Jansen

2010-04-21 21:12:07 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

Michael, it looks, like it's not enough. Unfortunately, it damaged more
repos than expected.

From a cursory lookup, _after_ resolving the repos I maintain myself, I
still suffer from these damaged repos for now:

http://download.opensuse.org/repositories/KDE:/KDE4:/STABLE:/Desktop/openSUSE_11.1/
http://download.opensuse.org/repositories/KDE:/KDE4:/Factory:/Desktop/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/tools/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.1/ |
http://download.opensuse.org/repositories/openSUSE:/Tools:/Unstable/openSUSE_11.1/
http://download.opensuse.org/repositories/server:/monitoring/openSUSE_11.1/
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.1/
http://download.opensuse.org/repositories/system:/packagemanager/openSUSE_11.1/

Post by Michael Schroeder
Sorry for the inconvenience.

This isn't an simple inconvenience. It's grown to a large scale denial of
update service thread, and those of us who suffer from this might have to
wait for weeks, until some packages _with differences_¹ are build by chance
in all these damaged repos.

Those of us, who are using the build service heavily, are even punished
harder: we need to do "zypper dup" all the time to get really all updated
packages, and it's not unusual, that the update list is huge. Now if you're
not _really_ careful, you're going to shot yourself in both knees: all
packages from damaged repos get downgraded to the basic repos. Needless to
say, that this will damage the build service reputation as a whole for
quite some time..

Hence, fix this as soon as possible please or give me a shell account with
access to all repos and I'm going to fix it with an ad hoc script this
night.

Thanks,
Pete

¹) which is a feature, as Adrian noted, but one that's biting us now..

Adrian Schröter

2010-04-22 07:34:39 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

Michael, it looks, like it's not enough. Unfortunately, it damaged more
repos than expected.
From a cursory lookup, _after_ resolving the repos I maintain myself, I
http://download.opensuse.org/repositories/KDE:/KDE4:/STABLE:/Desktop/openSUSE_11.1/
http://download.opensuse.org/repositories/KDE:/KDE4:/Factory:/Desktop/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/tools/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_11.1/ |
http://download.opensuse.org/repositories/openSUSE:/Tools:/Unstable/openSUSE_11.1/
http://download.opensuse.org/repositories/server:/monitoring/openSUSE_11.1/
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.1/
http://download.opensuse.org/repositories/system:/packagemanager/openSUSE_11.1/

They look good to me. what error do you see ?

Post by Michael Schroeder
Sorry for the inconvenience.

Please discuss that with upstream yum repo people and people who wrote package
managers not alligning the specification. We are just the users here.

bye
adrian

--
Adrian Schroeter
SUSE Linux Products GmbH
email: ***@suse.de
--
To unsubscribe, e-mail: opensuse-buildservice+***@opensuse.org
For additional commands, e-mail: opensuse-buildservice+***@opensuse.org

Hans-Peter Jansen

2010-04-22 09:25:54 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

Oops, we updated the createrepo package on the build service host
to use some new features, and now all of the repos use sha256
checksums instead of sha1. Unfortunatelly libzypp in 11.1 and older
can't handle that checksum (yet).
We now use a createrepo option to use sha1 again for older repos,
so things should work again for all freshly created build service
repositories.

Michael, it looks, like it's not enough. Unfortunately, it damaged more
repos than expected.
From a cursory lookup, _after_ resolving the repos I maintain myself, I
http://download.opensuse.org/repositories/KDE:/KDE4:/STABLE:/Desktop/op
enSUSE_11.1/
http://download.opensuse.org/repositories/KDE:/KDE4:/Factory:/Desktop/o
penSUSE_11.1/
http://download.opensuse.org/repositories/devel:/tools/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/languages:/python/open
SUSE_11.1/ |
http://download.opensuse.org/repositories/openSUSE:/Tools:/Unstable/ope
nSUSE_11.1/
http://download.opensuse.org/repositories/server:/monitoring/openSUSE_1
1.1/
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.1/
http://download.opensuse.org/repositories/system:/packagemanager/openSU
SE_11.1/

They look good to me. what error do you see ?

Adrian, sorry for being too dense: please look into repodata/repomd.xml of
the respective repos: if checksum type="sha256" appears, as it is the still
the fact from the last three of the above list, that repo cannot be used
from 11.1 and derivates at all, since libzypp doesn't support that yet.

In the light of this fact, reread my last mail, please.

My point is, that waiting for repos to be cured by chance isn't an option
here, as zypper may damage perfectly fine systems by using "zypper dup", as
it just downgrades the lost packages to the lowest common denominator:
mostly the distribution.

If you now want to discuss the usage of "zypper dup", then just try to "one
click" install any rpm from an affected repo with 11.1: it won't work:
there's no way around fixing the damaged repos.

Post by Michael Schroeder
Sorry for the inconvenience.

package managers not alligning the specification. We are just the users
here.

This is not an upstream problem, propably the decision of using a new
default for checksumming is questionable, but that is not the matter here.

Pete

--
To unsubscribe, e-mail: opensuse-buildservice+***@opensuse.org
For additional commands, e-mail: opensuse-buildservice+***@opensuse.org

Adrian Schröter

2010-04-22 09:31:37 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

Oops, we updated the createrepo package on the build service host
to use some new features, and now all of the repos use sha256
checksums instead of sha1. Unfortunatelly libzypp in 11.1 and older
can't handle that checksum (yet).
We now use a createrepo option to use sha1 again for older repos,
so things should work again for all freshly created build service
repositories.

Michael, it looks, like it's not enough. Unfortunately, it damaged more
repos than expected.
From a cursory lookup, _after_ resolving the repos I maintain myself, I
http://download.opensuse.org/repositories/KDE:/KDE4:/STABLE:/Desktop/op
enSUSE_11.1/
http://download.opensuse.org/repositories/KDE:/KDE4:/Factory:/Desktop/o
penSUSE_11.1/
http://download.opensuse.org/repositories/devel:/tools/openSUSE_11.1/
http://download.opensuse.org/repositories/devel:/languages:/python/open
SUSE_11.1/ |
http://download.opensuse.org/repositories/openSUSE:/Tools:/Unstable/ope
nSUSE_11.1/
http://download.opensuse.org/repositories/server:/monitoring/openSUSE_1
1.1/
http://download.opensuse.org/repositories/server:/mail/openSUSE_11.1/
http://download.opensuse.org/repositories/system:/packagemanager/openSU
SE_11.1/

They look good to me. what error do you see ?

Adrian, sorry for being too dense: please look into repodata/repomd.xml of
the respective repos: if checksum type="sha256" appears, as it is the still
the fact from the last three of the above list

yes, they were not generated again yet.

Post by Michael Schroeder
Sorry for the inconvenience.

package managers not alligning the specification. We are just the users
here.

This is not an upstream problem, propably the decision of using a new
default for checksumming is questionable, but that is not the matter here.

sure, but blocking all new features for OBS in general since some old code
base could have a bug is not an option either.

Andrew Daugherity

2010-04-22 17:23:49 UTC

Post by Hans-Peter Jansen
- 'repo2solv.sh' '-o' '/var/cache/zypp/solv/KDE_KDE3/solv'
'/var/cache/zypp/raw/KDE_KDE3' Unknown checksum type: 8: sha256

Oops, we updated the createrepo package on the build service host
to use some new features, and now all of the repos use sha256
checksums instead of sha1. Unfortunatelly libzypp in 11.1 and older
can't handle that checksum (yet).
We now use a createrepo option to use sha1 again for older repos,
so things should work again for all freshly created build service
repositories.

[snip]