Mining the Fedora Infrastructure Bus

If you listen to the Fedora Infrastructure Bus, you will hear things like:
u'i': 1, u'timestamp': 1353031870.694725, u'topic':
u'', u'signature':
u'msg': {u'user': {u'username': u'odemia'}, u'agent': {u'username':

which in plain text means that a new FAS username was created which goes by the name ‘odemia’.

Or, for example:
1Evcit2TFFZZWoxN1cwCi0tLS0tRU5EIENFUlRJRklDQVRF\nLS0tLS0K\n', u'i': 1, 
u'timestamp': 1353038721.092251, 
u'topic': u'', u'signature': 
btBMYUabi3X0mOmz3dj0rzGeIJVrwVdz+u5ePeBhR6rTDAvrmknrw6lgnr\nyVG7eFk6mQUGXk+MM9o=\n', u'msg':
{u'comment': {u'group': u'proventesters', u'author': u'adamwill', u'text': u'mostly working okay here', 
u'karma': 1, u'anonymous': False, u'timestamp': 1353038719.0, u'update_title': u'anaconda-18.29-1.fc18'}, 
u'agent': u'adamwill'}}

which corresponds to adamwill’s comment on bodhi [1].

If you start storing these messages emitted, you can mine for some interesting statistics. (I am storing all the messages in a file ‘messages’ using a particular format. See the Python source code at the end)

For example, in the last 20 hours or so, 16 new users registered with FAS:

$ cat messages | grep '' | wc -l

And 175 edits was made to the various wiki pages:

$ cat messages | grep | wc -l

You can also look for rawhide and branch compose tree messages:

$ cat messages | grep

And as you can imagine, the possibilities are well, endless. Check out the documentation here at [2]. The consumer script I used is at [3]. Thanks to threebean for his help.

[2] Fedmsg Documentation
[3] Naive Consumer Script

Conflicting libdb dependencies: rpm-libs + httpd 2.2

I ran into a segfault-ing web application while attempting to get Beaker server working on Fedora 17. Beaker uses TurboGears for the web application using mod_wsgi to talk to Apache (httpd-2.2). (I am a little new to the web application world, so ignore any misuse of terms).

The particular use case that triggered the crash was uploading a task RPM to the Beaker server. That was some hint into where to look for the faulting code. The task RPM was being saved fine on the disk, so there was no problem in writing (permissions, etc). So, there had to be a problem in reading the RPM file (which is what Beaker does to retrieve the files and task specifications).

To investigate whether it was a problem reading the RPM file, I simply copied the relevant code from Beaker sources to a standalone Python script. No problems there. Hence, there had to be something wrong with the combination of using the rpm libraries (rpm-libs) together with the web application (httpd).

Debugging httpd with gdb

After futile efforts of inserting exception handling codes combined with logging in the code to bracket the code which was causing the crash, I decided to bite the bullet and just go the gdb way. Turns out it was simpler than I thought. Run ‘httpd’ in single process mode, and  then monitor the error_log file for the wsgi daemon process ID and then attach to it using gdb in another terminal.

Start httpd in single process mode:

# gdb /usr/sbin/httpd
(gdb) run -X

Then attach gdb to the wsgi daemon process (You should ensure via your wsgi configuration to create only one mod_wsgi daemon process with a single thread):

# gdb /usr/sbin/httpd attach 
(gdb) cont.

Once I had done this, I simply performed the action that triggered the crash, and I had a good stack trace, first few lines of which were:

#0 0x0000000100000001 in ?? ()
#1 0x00007fffd7fe1097 in db_init (dbhome=0x7fffdf7a9e70 "/var/lib/rpm", rdb=0x7fffdd052970)
at backend/db3.c:151
#2 dbiOpen (rdb=rdb@entry=0x7fffdd052970, rpmtag=rpmtag@entry=0, dbip=dbip@entry=0x7fffe399ff38,
flags=flags@entry=0) at backend/db3.c:551
#3 0x00007fffd7fe8e53 in rpmdbOpenIndex (db=db@entry=0x7fffdd052970, rpmtag=rpmtag@entry=0, flags=0)
at rpmdb.c:149
#4 0x00007fffd7fe93ef in openDatabase (prefix=, dbpath=dbpath@entry=0x0, dbp=dbp@entry=
0x7fffdf797648, mode=mode@entry=0, perms=perms@entry=420, 

So that pretty much confirmed that it was something that httpd did not like about the rpm-libs which caused it to crash the application. Discussing with Dan, this was indeed a case of conflicting shared libraries and bit more of looking around we found that this was the Berkeley DB database library (libdb) that was the culprit. httpd had both libdb-4.8 and libdb-5.2 loaded in its process maps, also verified with ‘lsof‘ (Thanks to StackOverflow for the lsof tip):

# lsof /lib64/

httpd    9722 apache mem    REG  253,1  1555128 1189531 /usr/lib64/
httpd   10578 apache mem    REG  253,1  1555128 1189531 /usr/lib64/
httpd   18828 apache mem    REG  253,1  1555128 1189531 /usr/lib64/
httpd   18832 apache mem    REG  253,1  1555128 1189531 /usr/lib64/
gdb     18863   root mem    REG  253,1  1555128 1189531 /usr/lib64/

[root@asaha temp]# lsof /lib64/ 
gdb      18824   root mem    REG  253,1  1756808 1204315 /usr/lib64/
httpd    18832 apache mem    REG  253,1  1756808 1204315 /usr/lib64/
gdb      18863   root mem    REG  253,1  1756808 1204315 /usr/lib64/
qemu-kvm 30866   qemu mem    REG  253,1  1756808 1204315 /usr/lib64/

As you can see, both the libdb versions is mapped to httpd’s process space. Since, httpd itself depends only on libdb-4.8, there is something else in the web application which is bringing in libdb-5.2. That something turned out to be rpm-libs:

# yum deplist rpm-libs |  grep libdb
   provider: libdb.i686 5.2.36-5.fc17

So, that’s the problem and the reason for the crash.


There is no solution on Fedora 17 at this point of time to this other than trying to get httpd-2.4 which is linked against However, the Fedora 18 release, currently in development ships with httpd-2.4 which is linked against, same as the rpm-libs version it is shipped with. And indeed, the above crash did not occur there.


To reproduce this with a minimal application, I taught myself how to integrate Flask with mod_wsgi and wrote this simple Flask application, which you can check out here. Follow the steps on the Flask docs for help.


Thanks to Dan on #fedora-qa and Dan in real life who helped me with debugging the crash and thanks to Graham Dumpleton for the mod_wsgi help on #pocoo. Here are some of the docs I referred to:

Related bug reports

self in Python

self keyword is a reference to the invoking class instance/object. Consider the following code:

''' Self really refers to the instance called with '''
class Point:
    def __init__(self):
        print 'Self identifier:: ',id(self)

if __name__ == '__main__':
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)

As you can see, a class (that does nothing) called Point is defined and then two objects/instances are created. To verify that self is really a reference to the invoking object, we print the identifier for the calling object and self. Unsurprisingly, both are the same.

Soon after posting this, I realized that self is not a keyword. In fact its just a convention to use self as a reference to the invoking object. You can use anything else, for example, this:

class Point:
    def __init__(this):
        print 'This identifier:: ',id(this)

if __name__ == '__main__':
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)
    # create an instance of the class
    p = Point()

    print 'Point object identifier', id(p)

GSoC 2012: On-Demand Fedora Build Service: Update #8

I spent some time cleaning up the code, mainly using pylint as an indicator. Also rewrote the fabfile to use fabric commands rather than native Linux commands wherever possible and is cleaner now. The HOWTO is also updated now.

This code refactoring has brought home (quite strongly!) the need for having my unit tests up and running soon. I plan to use py.test for my testing, and hopefully will have some tests ready in a week.

The code is available at

GSoC 2012: On-Demand Fedora Build Service: Update #7

In my last update, I reported that I had a functional build service, capable of harnessing multiple build nodes and a functional Web UI as well. The process to deploy the build service was however quite manual and tedious. I was looking for a way to help do this without as much manual intervention as possible and fabric (Thanks, Tim) was the answer. I wrote a fabfile for the whole task, called ‘’.  As outlined in my earlier post, I am using celery to distribute the build tasks, so the celeryd process needs to be running on the build daemons. I found zdaemon (Thanks Jan on fabric mailing list) to be the easiest way to run the celeryd process as daemons.  The updated code is available in the repository now [1].

There is a HOWTO [2] document, which should help you deploy the service and run your own home based build service. I recently used it to build myself a Fedora Scientific ISO. Please don’t abuse it. There is hardly any error handling now. I am however accepting bug reports now. Thanks for all your help.


GSoC 2012: On-Demand Fedora Build Service: Update #5

The current github code [1] does quite a few things as of now. Let me try to explain to the changes since last update and my rationale behind them.

Support for building Live images: I attempted to use livemedia-creator (which is going to be THE tool from F18+), but unfortunately ran into issues which prevented me from building images. So for now, I have implemented this feature using ‘livecd-creator’. The Kickstart file (flattened) needs to be specified and other details such as architecture, any extra packages to be pulled from Koji, etc. The specifications are specified via the config/live.conf file.

The User specifications are now completely via .conf files: Myrationale behind that in the first place was that since this code is really going to serve as the ‘backend’, command line arguments could be done away with. But, even if we want to use this as standalone, specifying .conf files is fine as well. (We will see what happens with this after discussing with my mentors). Here is a brief description of the config files in config/

  • imagebuild.conf: type of image, architecture, staging area (to be explained later) and email (for notification)
  • boot.conf: configuration for boot.iso images
  • repoinfo.conf: repository configuration required for the above
  • pungi.conf: configuration for DVD images
  • live.conf: configuration for live images

The kickstart files if needed are to be placed in the kickstarts/ sub-directory. To use it, you will need to ‘cd’ into ‘image_builder’ directory and run ‘$python’ after setting up the appropriate config files in image_builder/config and kickstart files in kickstarts/
if any.

Support for copying images: I have also now enabled support for copying the images to a ‘staging area’ as mentioned earlier. I assume a passwordless login setup and hence do a ‘scp’ once the desired image has been created. This is how you would use it standalone.

Now, as a first step towards being able to distribute build jobs to different node, I have also now added simple support for carrying out the build process on a different host. This is done by the file

Here is what it does:

  • Assumes that the config/ and kickstart/ files have been correctly setup by the web-form handler or manually.
  • Then copies these files to the image_builder/ directory
  • It creates a .tar archive of the image_builder
  • Then it reads the appropriate node (architecture) from the nodes.conf file and also retrieves the working location specified.
  • The .tar file is then ‘scp’-ed to the appropriate location specified
  • Then runs the script on the build node by ‘SSHing’
  • The image is then automatically transferred to the staging area
    specified as earlier.

To try this feature, simply setup nodes.conf file correctly and the config/ and kickstarts/ in data/ and run $sudo python

Note that the specified nodes should have all the dependencies installed, such as lorax, koji (setup correctly), pykickstart and livecd-creator.


ownCloud 4 on Fedora 16

ownCloud 4 installation is pretty much similar to ownCloud 3 installation and this shell script here will do the job for you on Fedora. One nit: I ran into error messages as this:


As you can see (on opening the screenshot), there is missing header error from ‘/var/www/html/owncloud/apps/files_odfviewer/appinfo/app.php’. This post here told me that it could be due to a empty line. So that;s it. I went to the file and removed the line no. 8 which was an empty line! And it works fine.