Programming and writing about it.

echo $RANDOM

Category: gsoc2012

GSoC 2012: The Experience

The evaluations are in and I have formally passed the Google Summer of Code, 2012 as part of the Fedora project. It was a huge learning experience. I was introduced to a number of new tools and techniques which I hadn’t heard of before.  I do not yet know how to literally “handcraft” Fedora ISO’s, however I at least know the tools for the job and have got familiar with working with Koji, the various repositories and the learnt about the “lifecycle” of a package.  In the process I hope my project will be useful to the Fedora QA, which is where we will be working towards.

Takeaways

  1. This was my first major open source project with a deadline to complete. And since I used a number of third party Open Source software, when I was stuck with a problem, the only avenue for help I had was the mailing list/project lead. That had its own shortcomings. I think there is an important lesson to learn. If you are using Open Source software in Business, you better buy support. In the same vein, if you are a Open Source software expecting to be used in anything serious, provide commercial support.
  2. I had multiple priorities to deal with during the past few months – professional and personal and I wanted to do them all equally well, because all were dear to me. I am glad I managed everything pretty well in the end – Just decide what you want to do, and work towards it. Things have a strange way of coming together at the end and leave you smiling.

Thanks

Any successful endeavour is usually not achieved alone. First and foremost, thanks to Tim Flink – my project mentor for seeing this through. Thankfully, my proposed idea of a web-based build service resonated with something which he had in mind and had already worked on for a bit and that was the beginning of it all. He also had to write extra long emails explaining to me what exactly we would be attempting to achieve. Thanks, Tim for your co-operation and guidance.

Thanks are due to the community of the following projects: celery (Thanks Ask Solem and others),  Flask (Thanks guys for making me believe even I can write web applications), Fabric, Zdaemon, Read-the-docs, Sphinx and of course the Fedora community in various ways. Python made working on the project a delight. Finally, thanks Google for organizing the Summer of Code. It definitely helps in pooling a dedicated task force working on Open Source projects.

And all the non-technical factors that makes a person take up such endeavours in the first place – when that little push is needed there is always someone behind the scene giving that push- explicitly or implicitly. Thanks to Protyusha for that.

And last but not the least, sometimes you need people against you to always spur you on. So, thanks to those people who knows only how to try to pull someone down.

For those of you kids who are thinking – hey this guy is writing like he has scaled Everest, I would like to tell you: Learn to celebrate the smallest of things, it just makes it that more precious.

GSoC 2012: On-Demand Fedora Build Service: Final Update

The Google Summer of Code’12 working deadline was yesterday. So that brings me to this last project update.  In a last addition, I added Fedora Account System (FAS) authentication to the Web application. This was really easy, thanks to the Flask-FAS plugin. The latest documentation can be referred to here and gives you a fair idea of how you could setup and use this service.

However, this is just the end of the Google Summer of Code. The original target of this project was to make it useful to Fedora QA and that’s what we will be working towards next. So, feel free to read the docs and try it out and file issues.

GSoC 2012: On-Demand Fedora Build Service: Update #11

Its been a while since I posted an update, and this will probably be the last before the final update as part of the Google Summer of Code.

The big update is that I have got pretty decent documentation up now. You may browse it here [1]. Its up to date with the current code base [2]. Besides the user documentation, it also contains some of the internal details about the tools/libraries/design decisions that have been taken in the project.

If you have got some free cycles to spare (human + computer), you may want to give it a shot to try building  images for yourself. As a tip, the local mode, involves least amount of setup and is easiest to try out what is in there. If you see anything broken, feel free to file an issue.

As an aside, I was literally blown away by the awesomeness of Sphinx and Read the docs. Really great tools and services.

Links:

[1] Documentation: http://on-demand-fedora-build-service.readthedocs.org/en/latest/

[2] GitHub: https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #10

In my latest commit [1], I added a number of features to the code which I had in mind:

  1. Application wide Logging
  2. Basic Build Monitoring
  3. Email notification

These features are most useful when the service is deployed with multiple worker nodes. For local image building, these are not so important. I also added support for specifying local filesystem as staging – useful for local image building.

Next up, I have a few things to work on. Test the Web UI and Rest API interface again. Update the HOWTO (its outdated already!). And few other things remain.

Demo of Email Notification

When you submit a build request (in multiple worker nodes mode), you get an email upon notification:

Your Image Building Request have been submitted. You may monitor the progress by going to http://192.168.1.4:5100/log/tmp/imagebuild_134292350407.log. You will also recieve an email upon completion.

And then when the job completes, another email:

The build was completed by worker:: 192.168.1.4. Detailed log:

2012-07-22 12:18:24,074 – Registered a new Image Build request fromamitsaha.in@gmail.com
2012-07-22 12:18:24,075 – Image type:: dvd
2012-07-22 12:18:29,074 – Image Build notification sent
2012-07-22 12:18:29,383 – Starting the Image Build Process
2012-07-22 12:18:29,385 – DVD image arch requested should be the same as the build arch
2012-07-22 12:18:29,386 – Image building process complete
2012-07-22 12:18:29,387 – Error creating image. Transferring Logs.
2012-07-22 12:18:29,388 – Initiating local transfer of logs to /tmp/staging
2012-07-22 12:18:29,401 – Logs available at file:///tmp/staging

At this point of time, the best place to understand what is happening is to begin with the command line clients in cli/ : build_cli_basic.py and build_cli.py.I Should have an updated HOWTO soon.

Links:

[1] Latest commit: https://github.com/amitsaha/gsoc2012_fbs/commit/cf23dcc17d38924296eb099f6d1c2a440efb3d82

GSoC 2012: On-Demand Fedora Build Service: Update #9

The code saw a few additions/enhancements since the last update:

  • There is now a command line client directly submitting build jobs and another client using the basic REST interface exposed via the Web application [1].
  • Remote Kickstart files specified via a http:// or ftp:// are now supported. Once again, this has to be ‘ksflattened’.
  • I added some validation on the web based interface. However it happens on the server side using Flask’s pre_validate( ) method. Ideally I would like to familiarize myself with enough Java Script to do this on the client side
  • I started writing some Unit tests using py.test and Mock. But, I am currently waiting for some guidance from Tim to really write some good tests.
With the Mid-term evaluation due in less than a week, I am quite happy with the way things are progressing at this moment.  The TODO and GOOD-TO-HAVE notes down the things I would like to finish before the project deadline is over [2].
Links

GSoC 2012: On-Demand Fedora Build Service: Update #8

I spent some time cleaning up the code, mainly using pylint as an indicator. Also rewrote the fabfile to use fabric commands rather than native Linux commands wherever possible and is cleaner now. The HOWTO is also updated now.

This code refactoring has brought home (quite strongly!) the need for having my unit tests up and running soon. I plan to use py.test for my testing, and hopefully will have some tests ready in a week.

The code is available at https://github.com/amitsaha/gsoc2012_fbs

GSoC 2012: On-Demand Fedora Build Service: Update #7

In my last update, I reported that I had a functional build service, capable of harnessing multiple build nodes and a functional Web UI as well. The process to deploy the build service was however quite manual and tedious. I was looking for a way to help do this without as much manual intervention as possible and fabric (Thanks, Tim) was the answer. I wrote a fabfile for the whole task, called ‘deploy.py’.  As outlined in my earlier post, I am using celery to distribute the build tasks, so the celeryd process needs to be running on the build daemons. I found zdaemon (Thanks Jan on fabric mailing list) to be the easiest way to run the celeryd process as daemons.  The updated code is available in the repository now [1].

There is a HOWTO [2] document, which should help you deploy the service and run your own home based build service. I recently used it to build myself a Fedora Scientific ISO. Please don’t abuse it. There is hardly any error handling now. I am however accepting bug reports now. Thanks for all your help.

[1] https://github.com/amitsaha/gsoc2012_fbs/
[2] https://github.com/amitsaha/gsoc2012_fbs/blob/master/HOWTO

Follow

Get every new post delivered to your Inbox.

Join 66 other followers