News on debimg

Well, you may have noticed that debimg 0.1 is still not released. But a lot of work happened over the weekend in my local branch.

First of all, debimg’s set support is almost finished. I uploaded a tarball containing the differences between the official lenny weekly build from yesterday and a build created today by debimg, using the tasks of debian-cd 3.0.4 (after manual conversion to a format supported by debimg). Look at http://jak-linux.org/cdimage/tests/ for the tarball.

Secondly, the dependency resolver has been rewritten. It’s a bit slower now (0.72 seconds for main), but creates much better results. Resolving the dependencies of all packages in Debian Lenny i386 in alphabetical order, debimg 0.0.X resolved 206 dependencies differently than apt. Now, these have been decreased to 15 dependencies, whereas 13 dependencies are false-positive (some packages were not installed because they were already installed). This means that only two ones were different, in this caseachims-guestbook and chdrv, which both depend on virtual-only packages (achims-guestbook: apache | httpd, chdrv: console-utilities).

The third big change is the addition of the hooks module. This module allows you to hook in custom functions, which have access to the Configuration object (ConfigObj) and the MediaSet. There are currently three types of hooks: pre_hooks (run before fetching packages, adding files to the disk), mid_hooks (run after the packages have been fetched) and post_hooks (run after the image has been built). Hooks can be added based on project and architecture, using a simple syntax which support shell patterns. (It’s ‘project/arch’). The hooks module uses python decorators to register functions. debimg 0.2 will switch to hooks for internal functions, too, like bootloaders and other stuff.

The code has not been merged into the master branch, but I will hopefully be able to merge it tomorrow. The release of debimg 0.1 is now planned for this weekend.

What will be in debimg 0.1

debimg 0.1 will be the first major milestone in the development of debimg. I will now list some changes compared to the current release, debimg 0.0.3 (Please note that the following text is from an internal NEWS file and may not be correct in some aspects, as it is already some days old)

Support for disk splitting

This release of debimg adds support for creating media sets, i.e. splitting the packages over multiple disks. This is achieved by providing a new class called MediaSet, which passes all calls to methods to all medias.

When a package is added to a MediaSet, MediaSet tries to find the first disk where it can be added, by checking if all dependencies of the package are provided on the disk (or previous ones). This dependency checker does no recursive dependency checks, and can be disabled via OptimizedDiskSplitting = False (or no, n, 0…) in the configuration file.

The MediaSet classes are also lists, so you can access disk one via the index 0. BaseMedia has been modified to also support being accessed via index, using disknumber-1 in order to have both classes share the same API.

New data lists

This release of debimg brings users a more powerful way to include packages. Instead of various keys to include packages, debimg now uses the Include key.

To include packages, you have to use the ‘Include’ field. The value is a list of items, separated by commas. An item may either be the name of a package or a special form written as key:value. In this case, the following keys are supported:

Task/Priority:
This includes all packages belonging to the specified task or priority. In
case of tasks, debimg first includes all Key packages, then the other
packages.

Configuration files are easier

This release of debimg makes it possible to build multiple architectures using the Architectures option. This option replaces the previous Architecture option, which is now set automatically by the script for each architecture.

It also enables the Projects option, in case you want to build multiple projects. (All need to have the same MediaType and same NumberOfDisks). The project may also be cd-set, dvd-set, dvd9-set, netinst or businesscard. In these cases, MediaType is set automatically. [MediaType may also be moved in to data files].

This release also allows you to prefix any configuration value with path: in order to automatically convert the value to an absolute path. This is needed for many file options, but should not be used for some other stuff.

Improved Jigdo Support

debimg 0.1 allows you to define the public path (name) to (of) the image file and the template file that is written to the Jigdo file.

Use urlgrabber for file downloading

Starting with this release, debimg uses urlgrabber to download all kinds of files (except packages). A urlgrab call has been added to BaseMedia.addFile() which now understands http:// and ftp:// urls.

Plugins (maybe)

debimg 0.1 may introduce support for registering custom functions. This feature is low-priority, although required for easy development of Ubuntu-related code.

About debimg

debimg is a GPL-3 licensed software designed to replace debian-cd, a tool to create Debian images. For further information about debimg, visit the Wiki page. For more information about Debian, visit the website at www.debian.org.

Responding to e-mails with Evolution

Every time I receive an email at my ubuntu.com address, and respond to it, Evolution uses my jak@jak-linux.org e-mail address.

It should be noted that the e-mails are fetched with a single account, as they are on an imap server, and because ubuntu.com is a forward address.

Is there any way to make Evolution respond with the e-mail address at which I received an email?

BTW, work on debimg 0.0.4ubuntu1 (previously 0.0.3ubuntu1) has reached the first milestone: Support for selecting packages based on the output of Germanite. This weekend, I will release debimg 0.0.4 and debimg 0.0.4ubuntu1 (or I merge it directly, let’s see).

Update: This was just some mistake on my side. I get emails from an Ubuntu mailing list and want to send emails to it using the ubuntu.com address. It works for normal emails.

April Updates

This is a summary of most of my activities since end of march. BTW, I’m still at the T&S step in NM since January (I completed P&P in about 3 days). Also, thank you Tolimar for being the second DD signing my key!

GNOME 2.22 (Python) / Updated Packages

At the end of march, I updated some GNOME packages. These packages were gnome-python and gnome-python-desktop. The upload of gnome-python-desktop was really important, because the old version depended on libtotem-plparser7, which was not available anymore, and FTBFS because the metacity API changed. This upload made other packages building and running again!

That time, I also uploaded new releases of dir2ogg and ndisgtk, which fixed some bugs.

On the first of April, I updated aufs to a new upstream snapshot, which fixed linux-modules-extra-2.6’s FTBFS on armel, removed bashism in shell scripts, added a hack for limited splice support, and enabled building on -rt kernel (if the required functions are exported).

Packages To-Do

In the next week, I will upload a new aufs snapshot with support for kernel 2.6.25 and re-added support for kernels < 2.6.23. I will also update app-install-data to the current state of the archive.

Other stuff includes my ITA upload of gimmie and the upload of jockey, a tool to install drivers. Jockey will also be modified to provide the functionality from ndisgtk, which development has been discontinued as it is feature-complete (bug fixes will still be provided). Another package will be, of course, debimg 0.1 once it’s released and of course the python-libisofs bindings.

Ubuntu packages

I requested syncs for dir2ogg and ndisgtk (after I uploaded sync’able versions to Debian) and for aria2, which was not installable before. I have also reported some bugs and used 8.04 for some time.

debimg stuff

I released debimg 0.0.3 on 4th April. This is the first release to require Python 2.5 and also the first release which uses the new media module, which provides a generic interface to disk creation.

I actually have not worked on debimg since that day, mainly because I did not have enough time. In May, I hope to add support for more architectures (at least theoretically, by providing a generic way of handling bootloaders and other non-packages and non-dists files) and release 0.1.

debimg 0.1 will not contain any features related to python-libisofs, because the focus is getting the basic functionality.

I have also not uploaded any new netinst build on jak-linux.org, since March.

debimg and the Debian Project News

Debimg will appear in the second issue of the Debian Project News, I’m currently working on the text for it. (BTW, I have also fixed one link in the first issue of the DPN)

debimg and Ubuntu

I have suggested to switch Ubuntu’s image building to debimg, and offered to do everything needed to do this. Given the speed of debimg and that the current features almost match the requirements of Ubuntu, this seems to be a great idea.

python-libisofs

As you may know, I released a first preview of the python-libisofs bindings some days ago, with almost complete support for creating image files. The next steps will be reading and growing images and of course, the bindings for libburn and libisoburn.

Switching from Ubuntu 8.04 to Debian unstable

I am currently working on migrating my laptop from Ubuntu 8.04 to Debian unstable, in order to be able to work better on my packages. Since most of my packages are sync’able now, this enables more efficient development. Another reason is to help the lenny release. This does not mean that I will be less active on the Ubuntu side, at least not much.

Let’s make Debian Lenny and Ubuntu Intrepid the best releases ever.

python-libisofs 0.0.1 available

A first preview of the python-libisofs bindings is available now. It’s currently located in a git branch at git.debian.org, but this may change at a later point.

The bindings support the creation of ISO Images and (almost) all options libisofs supports, like Rockridge, Joliet, and much more. Reading and Modifying existing images is not supported yet.

The code is written in Cython and you need cython installed for building from the git branch. It can be installed just like any other Python module/extension/package, using a setup.py.

Browse: http://git.debian.org/?p=users/jak-guest/python-libisofs.git
Get: git clone git://git.debian.org/git/users/jak-guest/python-libisofs.git

I’ll package it for Debian within the next weeks after some further tests.

debimg 0.0.3 - “the checksum” - released

Dear readers,

The third release of debimg is available now: 0.0.3

Get the tarball: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz
Verify it: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz.asc
The ChangeLog: http://alioth.debian.org/~jak-guest/ChangeLog-0.0.3

Clone git repo: git://git.debian.org/git/users/jak-guest/debimg.git
Browse the repo: http://git.debian.org/?p=users/jak-guest/debimg.git

More Information: http://wiki.debian.org/DebImg
Daily images: http://jak-linux.org/cdimage/daily-builds/testing/

About debimg

debimg is a software designed to replace debian-cd, written in Python, and
supporting the creation of single disks for the i386 and amd64 architectures.

debimg is of course free software and licensed under the terms of the GNU
General Public License 3 or (at your option) any later version.

About “the checksum”

This release is called “the checksum”, because the software knows
the MD5SUM, SHA1SUM and SHA256SUM of every file on the disk.

News

• Introduction of the media module
  • Add md5sum.txt and sha1sum.txt to the image
  • NEW OPTION: JigdoMap, see the config file
  • Lot of code cleanup
• Support for custom installer images
  • Modify InstallerImages to an url supported by apt
• Renamed most options in the configuration file for better
  readability
• debimg requires Python 2.5, as it uses the with statement.

Description of the Release

This release of debimg introduces the media module with its classes
MediaFile, BaseMedia and DebianMedia.

The MediaFile class contains information about a file.
It contains the absolute path to the file on the filesystem,
the path on the media, its size, md5sum, sha1sum and sha256sum.

The BaseMedia class provides methods to add files to the
image, open files on it, and creating the final image. It also
provides methods to create the files md5sum.txt and sha1sum.txt,
and methods to support Jigdo file creation.

The DebianMedia class provides methods to add packages to
the image, creating Release and Packages files for dists.

Quick start

To get started with debimg, get the tarball and extract it to
some directory.

Now, open the file debimg.cfg and change the option Mirror to the URL
of your preferred mirror. This mirror may be any kind of mirror supported by
apt, but if you use file:/ they have to be on the same mountpoint, as the files
are hardlinked. (use copy:/ to get them copied).

Now, run ./debimg debimg.cfg. This will create an ISO image named
debian-lenny-i386-netinst.iso. This image is a normal netinst (except for
missing documentation and some other small things), and contains the Lenny d-i.

To change more settings, take a look at debimg.cfg.

Future

• Cleanup of the configuration format (almost finished)
• Support for splitting disks
  • Introduction of MediaSet class
  • Changes to the lists required
• Add documentation to the disks
• Support for PowerPC
• Create Debian package (almost finished)

Another interesting feature will be the libisofs [0] support provided by
the python-libisofs extension, which is currently under development. [1]

Previous release announcements

• 0.0.2: http://lists.debian.org/debian-cd/2008/03/msg00114.html
• 0.0.1: http://lists.debian.org/debian-cd/2008/03/msg00021.html

Links

• [0] libburnia project: http://libburnia-project.org/
• [1] http://juliank.wordpress.com/2008/04/02/python-extension-for-libisofs/

Python Extension for libisofs

I am working on a Python extension for the libisofs library. The extension is written in Cython , a Python-like language designed to write Python extensions.

At a later point, this extension will be used to create the ISO Images in debimg. It will be disabled by default, but you can enable it via a configuration option.

debimg news: disk splitting, new data lists, PowerPC

The next release of debimg (0.0.3) will be the first release supporting splitting packages over multiple disks. I worked a lot on the base technology today, and the basics are working.

In order to get support for disk splitting, I introduce three new classes: Media, MediaSet and MediaSetCollection (all in libdebimg.media).

The Media class is responsible for adding files to an image (incl. copying/linking), generating extra files (like MD5SUMS or documentation) and for building the image. It knows the space which is free, the size of each file and the md5sum of each file. It also writes all data inside dists/

The MediaSet class contains one or more Media instances. It has the functionality to split the packages into the disks.

MediaSet knows the dependencies of all packages and tries to add it to the disk with smallest number (e.g. 1) if there is enough space left for the package and all dependencies. If a dependency is on disk2, the package will be added to disk 2 or later. (If one dependency is on the last disk, we don’t check where the others are, instead, we put it on the same disk [faster])

The MediaSetCollection class contains multiple MediaSet instances (e.g. for DVDs and CDs), and calls their functions. This makes it possible to build multiple MediaTypes at the same time in an efficient way, because we don’t need to call debimg multiple times or run in multiple loops.

This means that many methods of libdebimg.packages.MirrorBuilder move into the new module. For example, the writePackagesFiles() and writeReleaseFiles() will be part of media.Media. (and can be called from MediaSet [for every Media] and MediaSetCollection [for every MediaSet], too)

From boot, run_genisoimage() will be removed. It is replaced by the createImage() method of media.Media.

In future, debimg will be able to run in three modes: simple, set and sets. The simple mode only creates one disk and is used for netinst and businesscard images. The set is used when only one MediaType has to be generated. The recommended mode for building e.g. weekly-builds is sets, as it has support for multiple MediaTypes.

Another change is the change to the format of data/*.list. I remove the keys called Priority and Tasks. Instead of these, I will add support for these directly into Depends. This means that you will be able to use “Priority:standard” just like “pkgname”. This makes sorted lists possible, which are needed for disk splitting. I will also rename the package list to match the names used in debian-cd (mostly) and add a new field called Single-Disk: yes, which disables sorting of the list, and is a bit faster than the other one.

This feature code is unreleased at the moment, but will hopefully be ready next week. Please note that I will never build complete sets with debimg. Therefore, I added a mode to create a list of all packages and build the image without them. (I have no local mirror)

Next week will also be the start of the first daily-builds of PowerPC images and businesscard images available from jak-linux.org/cdimage.

If you want to use the current release (0.0.2), a good point to start is a mail I wrote:

http://lists.debian.org/debian-cd/2008/03/msg00116.html

debimg 0.0.2 builds netinst in less than 4 seconds

debimg 0.0.2 has a new high speed dependency resolver and support for local repositories.

With these new features, debimg is now able to produce netinst images in less than 4 seconds.

BTW, you can easily build your own disk. Simply unpack the tarball, change the mirror to your one, and run ./debimg debimg.cfg. This should give you an i386 netinst disk.

Get debimg 0.0.2 from http://users.alioth.debian.org/~jak-guest/debimg_0.0.2.tar.gz

Fetch daily-built images at http://jak-linux.org/cdimage/daily-builds/testing/.

For more information, see http://lists.debian.org/debian-cd/2008/03/msg00114.html

cdimage for debimg is online now!

debimg’s cdimage is now online at http://jak-linux.org/cdimage/, and contains daily-builds of Debian testing netinst disks for i386 and amd64 architectures.

debimg is a new software which aims to replace debian-cd. No code has been released yet, but will be in the next days.

Please note that I can only provide jigdo images, because of bandwith and storage.

I will soon add build logs, and files listing the differences compared to original netinst images. I will also add support for more architectures.

debimg homepage at jak-linux.org