News on debimg

Well, you may have noticed that debimg 0.1 is still not released. But a lot of work happened over the weekend in my local branch.

First of all, debimg’s set support is almost finished. I uploaded a tarball containing the differences between the official lenny weekly build from yesterday and a build created today by debimg, using the tasks of debian-cd 3.0.4 (after manual conversion to a format supported by debimg). Look at http://jak-linux.org/cdimage/tests/ for the tarball.

Secondly, the dependency resolver has been rewritten. It’s a bit slower now (0.72 seconds for main), but creates much better results. Resolving the dependencies of all packages in Debian Lenny i386 in alphabetical order, debimg 0.0.X resolved 206 dependencies differently than apt. Now, these have been decreased to 15 dependencies, whereas 13 dependencies are false-positive (some packages were not installed because they were already installed). This means that only two ones were different, in this caseachims-guestbook and chdrv, which both depend on virtual-only packages (achims-guestbook: apache | httpd, chdrv: console-utilities).

The third big change is the addition of the hooks module. This module allows you to hook in custom functions, which have access to the Configuration object (ConfigObj) and the MediaSet. There are currently three types of hooks: pre_hooks (run before fetching packages, adding files to the disk), mid_hooks (run after the packages have been fetched) and post_hooks (run after the image has been built). Hooks can be added based on project and architecture, using a simple syntax which support shell patterns. (It’s ‘project/arch’). The hooks module uses python decorators to register functions. debimg 0.2 will switch to hooks for internal functions, too, like bootloaders and other stuff.

The code has not been merged into the master branch, but I will hopefully be able to merge it tomorrow. The release of debimg 0.1 is now planned for this weekend.

What will be in debimg 0.1

debimg 0.1 will be the first major milestone in the development of debimg. I will now list some changes compared to the current release, debimg 0.0.3 (Please note that the following text is from an internal NEWS file and may not be correct in some aspects, as it is already some days old)

Support for disk splitting

This release of debimg adds support for creating media sets, i.e. splitting the packages over multiple disks. This is achieved by providing a new class called MediaSet, which passes all calls to methods to all medias.

When a package is added to a MediaSet, MediaSet tries to find the first disk where it can be added, by checking if all dependencies of the package are provided on the disk (or previous ones). This dependency checker does no recursive dependency checks, and can be disabled via OptimizedDiskSplitting = False (or no, n, 0…) in the configuration file.

The MediaSet classes are also lists, so you can access disk one via the index 0. BaseMedia has been modified to also support being accessed via index, using disknumber-1 in order to have both classes share the same API.

New data lists

This release of debimg brings users a more powerful way to include packages. Instead of various keys to include packages, debimg now uses the Include key.

To include packages, you have to use the ‘Include’ field. The value is a list of items, separated by commas. An item may either be the name of a package or a special form written as key:value. In this case, the following keys are supported:

Task/Priority:
This includes all packages belonging to the specified task or priority. In
case of tasks, debimg first includes all Key packages, then the other
packages.

Configuration files are easier

This release of debimg makes it possible to build multiple architectures using the Architectures option. This option replaces the previous Architecture option, which is now set automatically by the script for each architecture.

It also enables the Projects option, in case you want to build multiple projects. (All need to have the same MediaType and same NumberOfDisks). The project may also be cd-set, dvd-set, dvd9-set, netinst or businesscard. In these cases, MediaType is set automatically. [MediaType may also be moved in to data files].

This release also allows you to prefix any configuration value with path: in order to automatically convert the value to an absolute path. This is needed for many file options, but should not be used for some other stuff.

Improved Jigdo Support

debimg 0.1 allows you to define the public path (name) to (of) the image file and the template file that is written to the Jigdo file.

Use urlgrabber for file downloading

Starting with this release, debimg uses urlgrabber to download all kinds of files (except packages). A urlgrab call has been added to BaseMedia.addFile() which now understands http:// and ftp:// urls.

Plugins (maybe)

debimg 0.1 may introduce support for registering custom functions. This feature is low-priority, although required for easy development of Ubuntu-related code.

About debimg

debimg is a GPL-3 licensed software designed to replace debian-cd, a tool to create Debian images. For further information about debimg, visit the Wiki page. For more information about Debian, visit the website at www.debian.org.

Python Speed: ‘x in list’ vs ‘x in set’

Well, this is my second post about speed in Python. Today, I noticed that debimg’s dependency resolver was much much slower than before. I thought what the problem could be and finally realized that the problem was that I switched from sets to list. This is fixed now in commit d0fd700080de5c19cb5fd66918d14c5ffa26e805

Now, some benchmarks (using IPython):

In [1]: a = range(10**6)

In [2]: b = set(a)

In [3]: %timeit 10**6 in a
10 loops, best of 3: 31.8 ms per loop

In [4]: %timeit 10**6 in b
10000000 loops, best of 3: 98.6 ns per loop

1ms are 1 million ns. Therefore, using sets is about 322515 times faster than using lists (or tuples).

debimg can now calculate dependencies in 0.5 seconds again, instead of 1 minute with lists.

Responding to e-mails with Evolution

Every time I receive an email at my ubuntu.com address, and respond to it, Evolution uses my jak@jak-linux.org e-mail address.

It should be noted that the e-mails are fetched with a single account, as they are on an imap server, and because ubuntu.com is a forward address.

Is there any way to make Evolution respond with the e-mail address at which I received an email?

BTW, work on debimg 0.0.4ubuntu1 (previously 0.0.3ubuntu1) has reached the first milestone: Support for selecting packages based on the output of Germanite. This weekend, I will release debimg 0.0.4 and debimg 0.0.4ubuntu1 (or I merge it directly, let’s see).

Update: This was just some mistake on my side. I get emails from an Ubuntu mailing list and want to send emails to it using the ubuntu.com address. It works for normal emails.

Work on ubuimg / debimg for Ubuntu started

The work on the Ubuntu version of debimg has begun. The majority of changes will be the following ones (in the order they will be done):

1. Change debimg to use germinate to calculate dependencies (package lists)
2. Add the additional stuff (live, etc.)

Once we can recreate the Ubuntu hardy i386 and amd64 alternate disks, work starts on the live filesystem and on merging these features back into debimg master, which will also get support for more archs.

debimg uses germinate directly on the Python level.

debimg 0.0.3ubuntu1 is sheduled for this Sunday. This will be more or less really hacks.

• [master] Move the fetching of packages from packages to media, so we can use it for all files
• Add libdebimg.germinate as a wrapper around germinate, providing functions to build the disks
• Modify libdebimg to build all disks in one run, multiple architectures and multiple seeds.

The basic code structure will look like the following:

1. For each architecture:
   1. seeds = Run germinate
   2. For seed in seeds:
      1. Get the packages
      2. Get extra files
      3. Build the disk

The Debian version may switch to seed files too, in version 0.2.

April Updates

This is a summary of most of my activities since end of march. BTW, I’m still at the T&S step in NM since January (I completed P&P in about 3 days). Also, thank you Tolimar for being the second DD signing my key!

GNOME 2.22 (Python) / Updated Packages

At the end of march, I updated some GNOME packages. These packages were gnome-python and gnome-python-desktop. The upload of gnome-python-desktop was really important, because the old version depended on libtotem-plparser7, which was not available anymore, and FTBFS because the metacity API changed. This upload made other packages building and running again!

That time, I also uploaded new releases of dir2ogg and ndisgtk, which fixed some bugs.

On the first of April, I updated aufs to a new upstream snapshot, which fixed linux-modules-extra-2.6’s FTBFS on armel, removed bashism in shell scripts, added a hack for limited splice support, and enabled building on -rt kernel (if the required functions are exported).

Packages To-Do

In the next week, I will upload a new aufs snapshot with support for kernel 2.6.25 and re-added support for kernels < 2.6.23. I will also update app-install-data to the current state of the archive.

Other stuff includes my ITA upload of gimmie and the upload of jockey, a tool to install drivers. Jockey will also be modified to provide the functionality from ndisgtk, which development has been discontinued as it is feature-complete (bug fixes will still be provided). Another package will be, of course, debimg 0.1 once it’s released and of course the python-libisofs bindings.

Ubuntu packages

I requested syncs for dir2ogg and ndisgtk (after I uploaded sync’able versions to Debian) and for aria2, which was not installable before. I have also reported some bugs and used 8.04 for some time.

debimg stuff

I released debimg 0.0.3 on 4th April. This is the first release to require Python 2.5 and also the first release which uses the new media module, which provides a generic interface to disk creation.

I actually have not worked on debimg since that day, mainly because I did not have enough time. In May, I hope to add support for more architectures (at least theoretically, by providing a generic way of handling bootloaders and other non-packages and non-dists files) and release 0.1.

debimg 0.1 will not contain any features related to python-libisofs, because the focus is getting the basic functionality.

I have also not uploaded any new netinst build on jak-linux.org, since March.

debimg and the Debian Project News

Debimg will appear in the second issue of the Debian Project News, I’m currently working on the text for it. (BTW, I have also fixed one link in the first issue of the DPN)

debimg and Ubuntu

I have suggested to switch Ubuntu’s image building to debimg, and offered to do everything needed to do this. Given the speed of debimg and that the current features almost match the requirements of Ubuntu, this seems to be a great idea.

python-libisofs

As you may know, I released a first preview of the python-libisofs bindings some days ago, with almost complete support for creating image files. The next steps will be reading and growing images and of course, the bindings for libburn and libisoburn.

Switching from Ubuntu 8.04 to Debian unstable

I am currently working on migrating my laptop from Ubuntu 8.04 to Debian unstable, in order to be able to work better on my packages. Since most of my packages are sync’able now, this enables more efficient development. Another reason is to help the lenny release. This does not mean that I will be less active on the Ubuntu side, at least not much.

Let’s make Debian Lenny and Ubuntu Intrepid the best releases ever.

Fedora 9 & Kernel Mode Setting

Today, I decided to try out the new kernel mode setting feature in Fedora 9, which moves some stuff about video from userspace into the kernel.

I tested this on my notebook, a HP Compaq 6720s with Intel X3100 (GM965) graphics controller.

I downloaded the preview live image for x86_64 and booted with the i915.modeset=1 option. The boot was almost normal, except that it was flicker-free. After the system booted I switched the virtual terminal from Xorg to tty1 and back, and it was extremely fast. The terminals all had the same resolution.

This does not mean that everything is perfect. There are a lot of things which do not work. For example, running glxgears and moving the window does not “look good”. Also, speed was a bit low. Normally, in Ubuntu Hardy, I get about 1000 FPS, in Fedora I only got 230 FPS. But the difference was not visible.

Another thing which did not work was suspend & resume, because the graphics card seemed to be not correctly re-initialized. This is worse than Ubuntu & Debian. In Ubuntu, suspend & resume works almost every time. In Debian, it works except that the backlight is disabled in X.

If these problems can be fixed, it would be very interesting to get this feature into Ubuntu Intrepid, maybe not as the default, but as an option for people who want to test it.

Fedora 9 seems to be a fast and stable distribution, with experimental features for experienced users and developers. It also provides a nice application for configuring PulseAudio, like per-application volume settings, a functionality I miss in Ubuntu.

python-libisofs 0.0.1 available

A first preview of the python-libisofs bindings is available now. It’s currently located in a git branch at git.debian.org, but this may change at a later point.

The bindings support the creation of ISO Images and (almost) all options libisofs supports, like Rockridge, Joliet, and much more. Reading and Modifying existing images is not supported yet.

The code is written in Cython and you need cython installed for building from the git branch. It can be installed just like any other Python module/extension/package, using a setup.py.

Browse: http://git.debian.org/?p=users/jak-guest/python-libisofs.git
Get: git clone git://git.debian.org/git/users/jak-guest/python-libisofs.git

I’ll package it for Debian within the next weeks after some further tests.

Asia: Phoenix available in Europe

After the reunion of the original Asia lineup in 2006, a tour and the live cd/dvd “Fantasia - Live in Tokyo” in 2007, the super group returns with the first new studio album in 25 years.

For those of you who don’t know about them, Asia is a band formed in 1981, including former members of Yes, King Crimson, and Emerson, Lake & Palmer, Uriah Heep, U.K., and The Buggles.

The most popular song of Asia is “Heat of the Moment”, from their debut album which spent 9 weeks at number 1 U.S. album chart.

This new album is excellent, featuring songs like the rocker “Never Again” and “Extraordinary Life”. A must-have for all Asia fans out there.

The new album will be released in the United States on April, 15 and in Japan on April, 23. It can be bought from Amazon.com for $9.99 and in Germany from Amazon.de for 15.97€. (See your own Amazon for your local price.)

BTW, the Album is already #20 in Amazon.com Music Bestsellers.

Visit the band’s Website, and buy your own one from your favorite shop.

debimg 0.0.3 - “the checksum” - released

Dear readers,

The third release of debimg is available now: 0.0.3

Get the tarball: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz
Verify it: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz.asc
The ChangeLog: http://alioth.debian.org/~jak-guest/ChangeLog-0.0.3

Clone git repo: git://git.debian.org/git/users/jak-guest/debimg.git
Browse the repo: http://git.debian.org/?p=users/jak-guest/debimg.git

More Information: http://wiki.debian.org/DebImg
Daily images: http://jak-linux.org/cdimage/daily-builds/testing/

About debimg

debimg is a software designed to replace debian-cd, written in Python, and
supporting the creation of single disks for the i386 and amd64 architectures.

debimg is of course free software and licensed under the terms of the GNU
General Public License 3 or (at your option) any later version.

About “the checksum”

This release is called “the checksum”, because the software knows
the MD5SUM, SHA1SUM and SHA256SUM of every file on the disk.

News

• Introduction of the media module
  • Add md5sum.txt and sha1sum.txt to the image
  • NEW OPTION: JigdoMap, see the config file
  • Lot of code cleanup
• Support for custom installer images
  • Modify InstallerImages to an url supported by apt
• Renamed most options in the configuration file for better
  readability
• debimg requires Python 2.5, as it uses the with statement.

Description of the Release

This release of debimg introduces the media module with its classes
MediaFile, BaseMedia and DebianMedia.

The MediaFile class contains information about a file.
It contains the absolute path to the file on the filesystem,
the path on the media, its size, md5sum, sha1sum and sha256sum.

The BaseMedia class provides methods to add files to the
image, open files on it, and creating the final image. It also
provides methods to create the files md5sum.txt and sha1sum.txt,
and methods to support Jigdo file creation.

The DebianMedia class provides methods to add packages to
the image, creating Release and Packages files for dists.

Quick start

To get started with debimg, get the tarball and extract it to
some directory.

Now, open the file debimg.cfg and change the option Mirror to the URL
of your preferred mirror. This mirror may be any kind of mirror supported by
apt, but if you use file:/ they have to be on the same mountpoint, as the files
are hardlinked. (use copy:/ to get them copied).

Now, run ./debimg debimg.cfg. This will create an ISO image named
debian-lenny-i386-netinst.iso. This image is a normal netinst (except for
missing documentation and some other small things), and contains the Lenny d-i.

To change more settings, take a look at debimg.cfg.

Future

• Cleanup of the configuration format (almost finished)
• Support for splitting disks
  • Introduction of MediaSet class
  • Changes to the lists required
• Add documentation to the disks
• Support for PowerPC
• Create Debian package (almost finished)

Another interesting feature will be the libisofs [0] support provided by
the python-libisofs extension, which is currently under development. [1]

Previous release announcements

• 0.0.2: http://lists.debian.org/debian-cd/2008/03/msg00114.html
• 0.0.1: http://lists.debian.org/debian-cd/2008/03/msg00021.html

Links

• [0] libburnia project: http://libburnia-project.org/
• [1] http://juliank.wordpress.com/2008/04/02/python-extension-for-libisofs/