yum update / python problem, yum stopped working after an update

[log69]

Hi Everyone,

I'm having an issue with my system, specifically the yum package after a recent update. Yum stopped working and its python is missing a module.

Does anyone have the same issue? I wrote to the ML, would you guys take a look at that?

http://www.mail-archive.com/scientific-linux-users@listserv.fnal.gov/msg11641.html

I greatly welcome every idea!

Thanks.

[AndrewSerk]

Hi log69,

I see you manually downloaded and installed the new python packages. You may want to try to do the same but with the older packages:

python-2.6.6-29.el6. instead of python-2.6.6-29.el6_2.2

python-libs-2.6.6-29.el6 instead of python-libs-2.6.6-29.el6_2.2

and so on for the packages that where updated with python.

Hope this helps,

[zxq9]

Is there anything unique running on your systems? Like non-standard setup, anything installed via pip or pipy or something you got that needed a "python install.py setup" thing to happen? By "unique" I really mean external repos or anything that isn't canonical SL. Also, are you on SL 6.2 or upgrading from another point release to 6.2 in one big jump?

Since I haven't seen widespread reports of Python or Yum failures I feel like something (probably tiny) is unique about your setup that is messing you around. Yum or Python failing or even acting a little sick are two things I would notice immediately on my own systems.

[log69]

QUOTE (zxq9 @ Jun 19 2012, 10:16 PM)

Is there anything unique running on your systems? Like non-standard setup, anything installed via pip or pipy or something you got that needed a "python install.py setup" thing to happen? By "unique" I really mean external repos or anything that isn't canonical SL. Also, are you on SL 6.2 or upgrading from another point release to 6.2 in one big jump? Since I haven't seen widespread reports of Python or Yum failures I feel like something (probably tiny) is unique about your setup that is messing you around. Yum or Python failing or even acting a little sick are two things I would notice immediately on my own systems.

There's nothing extra, couple of media packages from rpmforge with keeping the least minimum, and only enabled repos are epel. And what surprised me is that the problem occurred on python update and failing since then.

Can't seem to find the problem since the files are all ok.

Edit: This error message is weird too while running "yum update": Plugin "refresh-packagekit" can't be imported

[log69]

If I restore "/usr/lib64/python2.6" directory from my backup, then yum works again just fine. I can run "yum update" too. But when I run "yum reinstall python", then after the package install everything goes wrong again.

Any idea what could cause this? This is a simple case installing a package.

[zxq9]

From the symptoms sounds like the Python runtime or library in the new Python you're installing is borked.

That's pretty weird, though, because that would mean the last Python patch should have screwed everybody, but that's not the case.

You don't happen to still have rpmforge enabled or maybe have a Python library package coming from a repo other than SL?

If these packages were unsigned then I'd ask what mirror you were getting your updates from, but since they are all signed any mistake in the copy should invalidate the package, so this shouldn't be a problem.

[log69]

I checked which files change and how they change after reinstalling the python package. So I copied my dir as "python2.6.good", then ran "yum reinstall python", and then ran:

CODE

cd /usr/lib64/python2.6.good/ find -iname "*.py" ! -type d | xargs md5sum | sort > ~/o1 cd /usr/lib64/python2.6/ find -iname "*.py" ! -type d | xargs md5sum | sort > ~/o2 diff ~/o1 ~/o2 109d108 < 0f258f8d6a25ac9ccf0495a1a10c1878  ./json/__init__.py 126a126 > 11fedf8661c4edcaf4a06ee41c8c141f  ./json/__init__.py 283d282 < 25995f9376917f9b68e02a454561ae1c  ./distutils/config.py 1219d1217 < 92d7906c1d7309f40d9449f6ad879156  ./SimpleXMLRPCServer.py 1299a1298 > 9d7cedcba70fe132ca6bc01d84290302  ./distutils/config.py 1307d1305 < 9e339f546320fa9a7edc4728fe236608  ./os.py 1326a1325 > a09c3c7476c2fdcc85621ac795d44b51  ./os.py 1405d1403 < a921195b9bfe76a4e71328cb2d1a0c3d  ./SimpleHTTPServer.py 1557a1556 > bc1f3e34012531075e052588c9380e30  ./SimpleHTTPServer.py 1586a1586 > bf0be0b4137e25dfc3274c4efb6e36ea  ./SimpleXMLRPCServer.py

Now I see which files were modified, so I checked their diff too. From what I saw I think only the "./distutils/config.py" file may cause the problem. The diff looks like this:

CODE

diff python2.6.good/distutils/config.py python2.6/distutils/config.py 46,55c46,47 <         f = open(rc, 'w') <         try: <             f.write(DEFAULT_PYPIRC % (username, password)) <         finally: <             f.close() <         try: <             os.chmod(rc, 0600) <         except OSError: <             # should do something better here <             pass --- >         with os.fdopen(os.open(rc, os.O_CREAT | os.O_WRONLY, 0600), 'w') as fp: >             fp.write(DEFAULT_PYPIRC % (username, password))

[log69]

QUOTE (zxq9 @ Jun 20 2012, 08:02 AM)

From the symptoms sounds like the Python runtime or library in the new Python you're installing is borked. That's pretty weird, though, because that would mean the last Python patch should have screwed everybody, but that's not the case. You don't happen to still have rpmforge enabled or maybe have a Python library package coming from a repo other than SL? If these packages were unsigned then I'd ask what mirror you were getting your updates from, but since they are all signed any mistake in the copy should invalidate the package (and the odds of a random has collision are astronomical), so this shouldn't be a problem.

No rpmforge is enabled. Also, I always update manually running "yum update" and see which repos the packages are coming from (to avoid any mistake). And they come only from SL.

See more details in my other reply.

[zxq9]

Two thoughts:

1- Reading your original traceback from the ML post more critically, I'm not sure that config.py has anything to do with this. From the traceback it looks like there is an import not being made from /usr/lib64/python2.6/xml/etree/cElementTree.py for some reason. But the problem could be way more complicated than that. Check and see if that's there, at least, and if your updating did anything to it or altered the path to it.

Speaking of path, I don't think the Python runtime's own path is borked. If it was then none of this should spin up properly in the first place.

2- It seems like someone has an overactive desire to use low-ish file operations instead of just leaving yum with a few extra (very short) lines that don't halt execution when file permissions prevent the writing of .pypirc. I'm not certain, but it seems like the os.fdopen(blahblah... stuff in the patch might not work in all situations (like maybe if you're using networked storage and the SELinux permissions are set on mount, for example) but the original try...finally version would still permits failure of this non-essential write.

That said, maybe could try something like:

CODE

try:   with os.fdopen(os.open(rc, os.O_CREAT | os.O_WRONLY, 0600), 'w') as fp:     fp.write(DEFAULT_PYPIRC % (username, password)) except:   print "DEBUG ME"

to see if this is the troubled section or not. I don't think it is, though. Also, I haven't tried using a with statement inside of a try, but it should work the same. If not you could tack the .write() call on to the os.fdopen thing directly. The author this seems to use the fp tag only to shorten the length of the line for whatever reason (granted, that line would be like 109 characters which is 29 too long for some people's taste and 4 short of the limit of some other standards -- but a necessity for gigantic lines tends to indicate the presence of other issues...).

CODE

try:   os.fdopen(os.open(rc, os.O_CREAT | os.O_WRONLY, 0600), 'w').write(DEFAULT_PYPIRC % (username, password)) except:   print "DEBUG ME"

should be an equivalent statement. But I'm a Python noob, so don't expect this to work.

Alternately you could just jam the old one block back in there and see if you get the same problem.

I don't think this will fix it, though. I think the problem probably lies somewhere in a change to the xml module or something else yum is depending on that's not this.

---
UPDATE:
I just learned something interesting about exception handling in Python and how to use os.open() as a context manager. I'd have to go back and read the function again to be sure that the code I extended by tacking the write() call directly onto the fdopen() bit was actually doing this, but in any case I understand where the idiom/habit might come from, whether or not the author was actually doing this or not.
This is what I read that happened to touch on this obliquely

[log69]

@zxq9:

Adding the "try" and "except" lines don't change anything, I get the same error traceback as in my original post.

What's weird is that the only difference between a working and non-working configuration is the result of a "yum reinstall python" command. I don't get it. What else could change after running this?

CODE

# find /usr -mmin -1 /usr/include/python2.6 /usr/bin /usr/bin/python2 /usr/share/systemtap/tapset /usr/share/man/man1 /usr/share/doc/python-2.6.6 /usr/lib64/python2.6 /usr/lib64/python2.6/email /usr/lib64/python2.6/email/mime /usr/lib64/python2.6/idlelib /usr/lib64/python2.6/idlelib/Icons /usr/lib64/python2.6/bsddb /usr/lib64/python2.6/wsgiref /usr/lib64/python2.6/test /usr/lib64/python2.6/multiprocessing /usr/lib64/python2.6/multiprocessing/dummy /usr/lib64/python2.6/lib2to3 /usr/lib64/python2.6/lib2to3/pgen2 /usr/lib64/python2.6/lib2to3/fixes /usr/lib64/python2.6/site-packages /usr/lib64/python2.6/json /usr/lib64/python2.6/sqlite3 /usr/lib64/python2.6/hotshot /usr/lib64/python2.6/config /usr/lib64/python2.6/xml /usr/lib64/python2.6/xml/sax /usr/lib64/python2.6/xml/parsers /usr/lib64/python2.6/xml/dom /usr/lib64/python2.6/xml/etree /usr/lib64/python2.6/plat-linux2 /usr/lib64/python2.6/encodings /usr/lib64/python2.6/compiler /usr/lib64/python2.6/ctypes /usr/lib64/python2.6/ctypes/macholib /usr/lib64/python2.6/lib-dynload /usr/lib64/python2.6/curses /usr/lib64/python2.6/distutils /usr/lib64/python2.6/distutils/command /usr/lib64/python2.6/logging

After the reinstall, the only files modified are the ones above in my other post. I include here the full diff:
http://pastebin.com/GGkDQxhN

Files are:

CODE

./json/__init__.py ./distutils/config.py ./SimpleXMLRPCServer.py ./os.py ./SimpleHTTPServer.py

[zxq9]

Where did you get your yum plugins from? This bit here interests me a little:

QUOTE

Plugin "refresh-packagekit" can't be imported Loaded plugins: priorities, protectbase, remove-with-leaves, security

My plugins include only these by default: fastestmirror, refresh-packagekit, security

I wonder if something about the protectbase or remove-with-leaves plugin (or the way it has been used, rather) made your update get funny. That could definitely cause some unpredictable weirdness depending on what you've declared as "protected".

Can you roll back to the previous Python version again, and then try updating with

CODE

yum --disableplugin=protectbase,remove-with-leaves update

and see if you get a proper xml/etree installed?

Also, if you roll back to the old Python, when you run yum can refresh-packagekit load, or have you always gotten that message?

[log69]

I rolled back my /usr/lib64/python2.6 dir and ran this:

CODE

yum --disableplugin=protectbase,remove-with-leaves update yum --disableplugin=protectbase,remove-with-leaves reinstall python

But it brought back the same problem.

Also, this is interesting too:
http://www.mail-archive.com/scientific-linux-users@listserv.fnal.gov/msg11658.html

(thanks for your help)

[zxq9]

What is the output of this?

CODE

import sys for i in sys.path:   print i

Lefty and Righty just aren't looking at the same thing, I think.

[log69]

QUOTE (zxq9 @ Jun 20 2012, 02:33 PM)

What is the output of this? CODE import sys for i in sys.path:   print i Lefty and Righty just aren't looking at the same thing, I think.

/home/$USER/Desktop
/usr/lib64/python26.zip
/usr/lib64/python2.6
/usr/lib64/python2.6/plat-linux2
/usr/lib64/python2.6/lib-tk
/usr/lib64/python2.6/lib-old
/usr/lib64/python2.6/lib-dynload
/usr/lib64/python2.6/site-packages
/usr/lib64/python2.6/site-packages/gst-0.10
/usr/lib64/python2.6/site-packages/gtk-2.0
/usr/lib64/python2.6/site-packages/webkit-1.0
/usr/lib/python2.6/site-packages
/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info

[zxq9]

Hrm... you might want to put your /home/$USER/Desktop bit at the end, not the beginning. Do you know what the reason is for it being up there at the beginning or rather, what package or code set it that way? Is it in your .bashrc or someplace similar?

[log69]

I have no idea why that's at the first line. I don't have any settings for that in my .bashrc or anywhere else.

[zxq9]

Can you try this:

CODE

import sys for i in sys.path:   print i from xml.etree import cElementTree sys.path.pop(1) for i in sys.path:   print i from xml.etree import cElementTree

and paste the output?

[log69]

CODE

$ python test.py /home/andras/Desktop Traceback (most recent call last):  File "test.py", line 4, in <module>    from xml.etree import cElementTree  File "/usr/lib64/python2.6/xml/etree/cElementTree.py", line 3, in <module>    from _elementtree import * ImportError: /usr/lib64/python2.6/lib-dynload/pyexpat.so: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference

[zxq9]

Yeah, that's trying to pull it from the wrong place.

I just saw the new posts on your ML thread about this.

The system simply missing a lib like that is pretty weird -- actually really weird. Also (and probably unrelated) having something prepend instead of append your Python runtime path is also sort of troublesome. My guess is that you installed a naughty package that thought it would go off and do smart things with your Python installation.

[log69]

About my recent mail to the ML: This file is the root of all problems: /usr/lib64/python2.6/lib-dynload/pyexpat.so

What's wierd is that my updated python from SL repo works good with the older version of pyexpat.so, but doesn't work with the one that the python package ships with itself. What's really astonishing is that all my files are identical to other compared systems of mine (servers, chroot, vm), and changing this very file solves everything.

[log69]

Finally, Success has come!

Oleg's last answer on ML:

CODE

yum install expat.i686

Wow, thanks for your great help and time, I appreciate that!