Blog of Julian Andres Klode

Debian, Ubuntu, Linux in general, and other free software

Archive for the ‘debimg’ Category

debian-cd, dependencies, debimg, comments

Looking at the amd64 CD images of Lenny, I just saw that there is the package ‘gnome’ on disk 1, while some of its dependencies are on disk 2. I don’t think that this is good. Same also applies to K3B on disk 3, and openoffice.org-kde on KDE-disk 1.

In my opinion, all packages which are located on a disk X, should only depend on a disk N (N<X). This means that the package gnome would be moved to disk 2, or its dependencies to disk 1.

This is exactly the way debimg works. Debimg uses so-called package groups, which simply represents a set of packages with cyclic dependencies. These package groups are returned in a specific order, so that a group N only requires a group <N. This order is kept when the groups are added to the disk. Furthermore, we treat the groups as one when adding them to the disk, ie. we check whether the whole group fits on the disk (and add it) or not (create a new disk). This way, we ensure that all dependencies can be satisfied.

Today, I decided to try it out and therefore wrote a small script reading the debian-cd task files and comparing the file list (of the packages added to the disk) with the file list of the official KDE disk.

The results look very good, debimg adds 11 packages and removes 26 which is not that much, and mostly caused by wrong size limits, etc. You can look at the results your self, and regenerate them using:

Anyway, debimg still needs real configuration and handling of the debian installer and source packages, but this should give you an idea of how debimg works. And a tarball, for those who don’t want to use git all the time.

BTW, I have now enabled comment threading here, a new WordPress feature. I hope that it works, as I have not tried it out yet.

Written by Julian Andres Klode

March 2, 2009 at 17:54

Posted in Debian, debimg

debimg update – creating images

Much happened since the last time  I wrote about debimg. The project is now registered on Alioth and has a mailing list. On the code side, there have also been several changes.

First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the ‘image’ module.

The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it.

Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. ‘[PATCH 1/2]‘). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations.

Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I’m not uploading the current state as 0.1.0~a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0~b1 then).

What is wanted/planned?

  • Compatibility configuration formats
    • (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
    • (debimg.config.debian_cd) There could be an implementation of debian-cd’s configuration format. This would allow people to easily try out debimg. We can only support a subset, though.
  • New configuration formats
    • (debimg.config.yaml) Debimg’s native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.
  • Graphical Frontends:
    • (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
    • (debimg.frontend.qt4) The graphical frontend written in QT4.
  • Text frontends
    • (debimg.frontend.text) The basic command-line frontend.
    • (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It’s also not flexible enough for building custom images.

Requirements for 0.1 Beta

  • Implement at least one configuration format and the build frontend.
  • Have at least one external contributor.

More

Changes

Julian Andres Klode (6):

  • debimg/core/files: When creating a file object allow filename as source
  • debimg/core/resolver.py: Introduce Package.fullname, Package.component
  • debimg/core/repository.py: Merge the repository module.
  • debimg/core/resolver.py: Improve handling of certain dependency types
  • debimg/core/repository.py: Allow Repository.add_group to take a ‘distro’ parameter
  • debimg/core/image.py: Introduce the image module.

Written by Julian Andres Klode

March 1, 2009 at 20:02

Posted in Debian, debimg, Ubuntu

debimg core: creating a repository

As I wrote Monday in “debimg core example”, I have an extended version which creates a repository. The problem on Monday was that the code was not far enough to be published. Not much has changed yet, but I considered to create a temporary branch temp/repository to be able to show you the example, and to give you an impression of what the final API will look like (it’s not complete yet, trust me).

Now I will present you the example. It is not much longer than the pool example I gave on monday, but it does much more – it creates a repository (and it has some commandline options). The concept of the Repository is very simple, too. We have two classes, Repository and Distribution, and a Repository object contains multiple Distribution objects, which are responsible for packages and related stuff.

#!/usr/bin/python
# Copyright (C) 2009 Julian Andres Klode.
#
# Released under the terms of the GNU General Publice License, version 3
# or (at your option) any later version.
#
"""Example to demonstrate the power of debimg's repository management.

This example creates a directory example.debimg, with two subdirectories:
    - pool: This holds debimg's file pool (see debimg.core.files.Pool)
    - repo: This holds the created repository (debimg.core.repository)

It creates a repository of all required packages, using your local apt sources.
"""
from __future__ import with_statement
import operator
import optparse
import sys

from debimg.core.resolver import Resolver
from debimg.core.files import Pool
from debimg.core.repository import Distribution, Repository

def main():
    """Called when the script is executed."""
    parser = optparse.OptionParser()
    parser.add_option('-u', '--user', help='Same as in GPG')
    parser.add_option('-c', '--contents', action="store_true",
                     help='Create Contents-*.gz files')
    opts, args = parser.parse_args()

    # Create a pool, which manages the access to the files.
    pool = Pool('example.debimg/pool')

    # Create a resolver using your local apt configuration.
    pkgs = Resolver()

    # Create a new repository
    repo = Repository(pool, 'example.debimg/repo')

    # Create a new distribution and store it in dist.
    dist = repo.add_distribution('lenny')

    # Add all packages with priority required to the resolver
    pkgs.add_priority('required')

    # Add the packages from the resolver to the distribution
    for group in pkgs.groups():
        for package in group:
            dist.add_package(package)

    pool.compact()   # Optional step
    pool.fetch()     # Fetch all the packages from the mirror
    dist.finalize_files() # Link the files into the repository
    dist.finalize_packages() # Create Packages files (uncompressed and gzip)

    ##
    # If we want to create Contents-*.gz files, we can do this as well:
    # but we don't want to
    if opts.contents:
        dist.finalize_contents()

    # Now create our Release files, and sign them using the key provided on
    # the commandline. If no key has been provided, do not sign it.
    dist.finalize_release(Suite='testing', key=opts.user)

    for file in sorted(pool._files, key=operator.attrgetter('uri')):
        print file   # Print information about every file, sorted by URI.

if __name__ == '__main__':
    main()

To try the code, checkout the temp/repository branch at git://git.debian.org/users/jak/debimg.git and execute the script named test-repository, which is the same as printed above. If you have questions, please ask. Also read the docstring of debimg.core.repository to see which changes are not yet done (eg. common base class for Distribution and Repository, and of course adding files manually).

Written by Julian Andres Klode

February 12, 2009 at 19:42

Posted in Debian, debimg

debimg core example

The following example demonstrates the features of debimg core, and how it can be used to fetch a some packages. As you will see when you run this example, debimg uses SHA1 filenames for the downloaded files. This may be changed in a future version.

There is also an improved version of this example, which creates a repository, but the needed module (debimg.core.repository) is not public yet, because its far from being finished. I expect to complete repository code on Wednesday.

#!/usr/bin/python
# Copyright (C) 2009 Julian Andres Klode.
#
# Released under the terms of the GNU General Publice License, version 3
# or (at your option) any later version.
#
"""Example to demonstrate the power of debimg's pool

This example creates a directory example.debimg, with two subdirectories:
- pool: This holds debimg's file pool (see debimg.core.files.Pool)
"""
from debimg.core.resolver import Resolver
from debimg.core.files import Pool

def main():
    """Called when the script is executed."""

    # Create a pool, which manages the access to the files.
    pool = Pool('example.debimg/pool')

    # Create a resolver using your local apt configuration.
    pkgs = Resolver()

    # Add all packages with priority required to the resolver
    pkgs.add_priority('required')

    # Add the packages from the resolver to the pool
    for group in pkgs.groups():
        for package in group:
            pool.add_package(package)

    pool.fetch() # Fetch all the packages from the mirror

    for file in sorted(pool._files, key=lambda k: k.uri):
        print file # Print information about every file, sorted by URI.

if __name__ == '__main__':
    main()

Written by Julian Andres Klode

February 9, 2009 at 19:03

Posted in Debian, debimg

debimg reloaded – debimg core started

Today, I have published the first pieces of debimg 0.1. The published code includes the resolver, the compression module and the fetcher module. It is rewritten from scratch, this time with a PEP8-conforming style, and more flexible.

Because debimg.core does not depend on any specific configuration format, but is configured solely via parameters, it is more flexible than debimg 0.0. This new code enables people to write their own programs related to Debian images easily. And if you want to, you can use the facilities provided by debimg.configuration and debimg.frontend to write your own program using debimg configuration files (not implemented yet, core needs to be finished first).

Debimg 0.1 will use YAML configuration files, supports multiple repositories, and much more. We could even implement support for pinning packages.

Debimg 0.1 is more than the others. It is a library (actually a python package, debimg.core), it is a program designed for end-users and developers. The enormous flexibility allows us to create applications for almost everyone.

My vision is that someone who needs a custom Debian image simply fires up the debimg GTK+ frontend, selects the packages he/she wants and clicks build. And when someone needs more flexibility, there will be configuration files which lets you configure most aspects. And if this does not suffice, you can write your own application by importing the modules, replacing some functions, methods, etc. and simply call the main() function of the frontend.

The goals:

  • flexibility, high speed, and cool features.
  • be a show case for the features of the low-level python-apt bindings (apt_pkg, apt_inst)
  • high-quality code, no hacks, every single function/method/class/module documented.
  • be a library, and a program.
  • provide a graphical front-end to assist the unexperienced users.
  • provide powerful file-based configuration for advanced developers.
  • And finally, provide a replacement for debian-cd in the near future.

More:

Written by Julian Andres Klode

February 7, 2009 at 21:43

Posted in Debian, debimg

News on debimg

Well, you may have noticed that debimg 0.1 is still not released. But a lot of work happened over the weekend in my local branch.

First of all, debimg’s set support is almost finished. I uploaded a tarball containing the differences between the official lenny weekly build from yesterday and a build created today by debimg, using the tasks of debian-cd 3.0.4 (after manual conversion to a format supported by debimg). Look at http://jak-linux.org/cdimage/tests/ for the tarball.

Secondly, the dependency resolver has been rewritten. It’s a bit slower now (0.72 seconds for main), but creates much better results. Resolving the dependencies of all packages in Debian Lenny i386 in alphabetical order, debimg 0.0.X resolved 206 dependencies differently than apt. Now, these have been decreased to 15 dependencies, whereas 13 dependencies are false-positive (some packages were not installed because they were already installed). This means that only two ones were different, in this caseachims-guestbook and chdrv, which both depend on virtual-only packages (achims-guestbook: apache | httpd, chdrv: console-utilities).

The third big change is the addition of the hooks module. This module allows you to hook in custom functions, which have access to the Configuration object (ConfigObj) and the MediaSet. There are currently three types of hooks: pre_hooks (run before fetching packages, adding files to the disk), mid_hooks (run after the packages have been fetched) and post_hooks (run after the image has been built). Hooks can be added based on project and architecture, using a simple syntax which support shell patterns. (It’s ‘project/arch’). The hooks module uses python decorators to register functions. debimg 0.2 will switch to hooks for internal functions, too, like bootloaders and other stuff.

The code has not been merged into the master branch, but I will hopefully be able to merge it tomorrow. The release of debimg 0.1 is now planned for this weekend.

Written by Julian Andres Klode

May 6, 2008 at 20:21

Posted in Debian, debimg

What will be in debimg 0.1

debimg 0.1 will be the first major milestone in the development of debimg. I will now list some changes compared to the current release, debimg 0.0.3 (Please note that the following text is from an internal NEWS file and may not be correct in some aspects, as it is already some days old)

Support for disk splitting

This release of debimg adds support for creating media sets, i.e. splitting the packages over multiple disks. This is achieved by providing a new class called MediaSet, which passes all calls to methods to all medias.

When a package is added to a MediaSet, MediaSet tries to find the first disk where it can be added, by checking if all dependencies of the package are provided on the disk (or previous ones). This dependency checker does no recursive dependency checks, and can be disabled via OptimizedDiskSplitting = False (or no, n, 0…) in the configuration file.

The MediaSet classes are also lists, so you can access disk one via the index 0. BaseMedia has been modified to also support being accessed via index, using disknumber-1 in order to have both classes share the same API.

New data lists

This release of debimg brings users a more powerful way to include packages. Instead of various keys to include packages, debimg now uses the Include key.

To include packages, you have to use the ‘Include’ field. The value is a list of items, separated by commas. An item may either be the name of a package or a special form written as key:value. In this case, the following keys are supported:

Task/Priority:
This includes all packages belonging to the specified task or priority. In
case of tasks, debimg first includes all Key packages, then the other
packages.

Configuration files are easier

This release of debimg makes it possible to build multiple architectures using the Architectures option. This option replaces the previous Architecture option, which is now set automatically by the script for each architecture.

It also enables the Projects option, in case you want to build multiple projects. (All need to have the same MediaType and same NumberOfDisks). The project may also be cd-set, dvd-set, dvd9-set, netinst or businesscard. In these cases, MediaType is set automatically. [MediaType may also be moved in to data files].

This release also allows you to prefix any configuration value with path: in order to automatically convert the value to an absolute path. This is needed for many file options, but should not be used for some other stuff.

Improved Jigdo Support

debimg 0.1 allows you to define the public path (name) to (of) the image file and the template file that is written to the Jigdo file.

Use urlgrabber for file downloading

Starting with this release, debimg uses urlgrabber to download all kinds of files (except packages). A urlgrab call has been added to BaseMedia.addFile() which now understands http:// and ftp:// urls.

Plugins (maybe)

debimg 0.1 may introduce support for registering custom functions. This feature is low-priority, although required for easy development of Ubuntu-related code.

About debimg

debimg is a GPL-3 licensed software designed to replace debian-cd, a tool to create Debian images. For further information about debimg, visit the Wiki page. For more information about Debian, visit the website at www.debian.org.

Written by Julian Andres Klode

April 29, 2008 at 21:14

Posted in Debian, debimg

Work on ubuimg / debimg for Ubuntu started

The work on the Ubuntu version of debimg has begun. The majority of changes will be the following ones (in the order they will be done):

  1. Change debimg to use germinate to calculate dependencies (package lists)
  2. Add the additional stuff (live, etc.)

Once we can recreate the Ubuntu hardy i386 and amd64 alternate disks, work starts on the live filesystem and on merging these features back into debimg master, which will also get support for more archs.

debimg uses germinate directly on the Python level.

debimg 0.0.3ubuntu1 is sheduled for this Sunday. This will be more or less really hacks.

  • [master] Move the fetching of packages from packages to media, so we can use it for all files
  • Add libdebimg.germinate as a wrapper around germinate, providing functions to build the disks
  • Modify libdebimg to build all disks in one run, multiple architectures and multiple seeds.

The basic code structure will look like the following:

  1. For each architecture:
    1. seeds = Run germinate
    2. For seed in seeds:
      1. Get the packages
      2. Get extra files
      3. Build the disk

The Debian version may switch to seed files too, in version 0.2.

Written by Julian Andres Klode

April 25, 2008 at 16:28

Posted in debimg, Ubuntu

April Updates

This is a summary of most of my activities since end of march. BTW, I’m still at the T&S step in NM since January (I completed P&P in about 3 days). Also, thank you Tolimar for being the second DD signing my key!

GNOME 2.22 (Python) / Updated Packages

At the end of march, I updated some GNOME packages. These packages were gnome-python and gnome-python-desktop. The upload of gnome-python-desktop was really important, because the old version depended on libtotem-plparser7, which was not available anymore, and FTBFS because the metacity API changed. This upload made other packages building and running again!

That time, I also uploaded new releases of dir2ogg and ndisgtk, which fixed some bugs.

On the first of April, I updated aufs to a new upstream snapshot, which fixed linux-modules-extra-2.6′s FTBFS on armel, removed bashism in shell scripts, added a hack for limited splice support, and enabled building on -rt kernel (if the required functions are exported).

Packages To-Do

In the next week, I will upload a new aufs snapshot with support for kernel 2.6.25 and re-added support for kernels < 2.6.23. I will also update app-install-data to the current state of the archive.

Other stuff includes my ITA upload of gimmie and the upload of jockey, a tool to install drivers. Jockey will also be modified to provide the functionality from ndisgtk, which development has been discontinued as it is feature-complete (bug fixes will still be provided). Another package will be, of course, debimg 0.1 once it’s released and of course the python-libisofs bindings.

Ubuntu packages

I requested syncs for dir2ogg and ndisgtk (after I uploaded sync’able versions to Debian) and for aria2, which was not installable before. I have also reported some bugs and used 8.04 for some time.

debimg stuff

I released debimg 0.0.3 on 4th April. This is the first release to require Python 2.5 and also the first release which uses the new media module, which provides a generic interface to disk creation.

I actually have not worked on debimg since that day, mainly because I did not have enough time. In May, I hope to add support for more architectures (at least theoretically, by providing a generic way of handling bootloaders and other non-packages and non-dists files) and release 0.1.

debimg 0.1 will not contain any features related to python-libisofs, because the focus is getting the basic functionality.

I have also not uploaded any new netinst build on jak-linux.org, since March.

debimg and the Debian Project News

Debimg will appear in the second issue of the Debian Project News, I’m currently working on the text for it. (BTW, I have also fixed one link in the first issue of the DPN)

debimg and Ubuntu

I have suggested to switch Ubuntu’s image building to debimg, and offered to do everything needed to do this. Given the speed of debimg and that the current features almost match the requirements of Ubuntu, this seems to be a great idea.

python-libisofs

As you may know, I released a first preview of the python-libisofs bindings some days ago, with almost complete support for creating image files. The next steps will be reading and growing images and of course, the bindings for libburn and libisoburn.

Switching from Ubuntu 8.04 to Debian unstable

I am currently working on migrating my laptop from Ubuntu 8.04 to Debian unstable, in order to be able to work better on my packages. Since most of my packages are sync’able now, this enables more efficient development. Another reason is to help the lenny release. This does not mean that I will be less active on the Ubuntu side, at least not much.

Let’s make Debian Lenny and Ubuntu Intrepid the best releases ever.

Written by Julian Andres Klode

April 24, 2008 at 20:30

Posted in Debian, debimg, General, Ubuntu

debimg 0.0.3 – “the checksum” – released

Dear readers,

The third release of debimg is available now: 0.0.3

Get the tarball: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz
Verify it: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz.asc
The ChangeLog: http://alioth.debian.org/~jak-guest/ChangeLog-0.0.3

Clone git repo: git://git.debian.org/git/users/jak-guest/debimg.git
Browse the repo: http://git.debian.org/?p=users/jak-guest/debimg.git

More Information: http://wiki.debian.org/DebImg
Daily images: http://jak-linux.org/cdimage/daily-builds/testing/

About debimg

debimg is a software designed to replace debian-cd, written in Python, and
supporting the creation of single disks for the i386 and amd64 architectures.

debimg is of course free software and licensed under the terms of the GNU
General Public License 3 or (at your option) any later version.

About “the checksum”

This release is called “the checksum”, because the software knows
the MD5SUM, SHA1SUM and SHA256SUM of every file on the disk.

News

  • Introduction of the media module
    • Add md5sum.txt and sha1sum.txt to the image
    • NEW OPTION: JigdoMap, see the config file
    • Lot of code cleanup
  • Support for custom installer images
    • Modify InstallerImages to an url supported by apt
  • Renamed most options in the configuration file for better
    readability
  • debimg requires Python 2.5, as it uses the with statement.

Description of the Release

This release of debimg introduces the media module with its classes
MediaFile, BaseMedia and DebianMedia.

The MediaFile class contains information about a file.
It contains the absolute path to the file on the filesystem,
the path on the media, its size, md5sum, sha1sum and sha256sum.

The BaseMedia class provides methods to add files to the
image, open files on it, and creating the final image. It also
provides methods to create the files md5sum.txt and sha1sum.txt,
and methods to support Jigdo file creation.

The DebianMedia class provides methods to add packages to
the image, creating Release and Packages files for dists.

Quick start

To get started with debimg, get the tarball and extract it to
some directory.

Now, open the file debimg.cfg and change the option Mirror to the URL
of your preferred mirror. This mirror may be any kind of mirror supported by
apt, but if you use file:/ they have to be on the same mountpoint, as the files
are hardlinked. (use copy:/ to get them copied).

Now, run ./debimg debimg.cfg. This will create an ISO image named
debian-lenny-i386-netinst.iso. This image is a normal netinst (except for
missing documentation and some other small things), and contains the Lenny d-i.

To change more settings, take a look at debimg.cfg.

Future

  • Cleanup of the configuration format (almost finished)
  • Support for splitting disks
    • Introduction of MediaSet class
    • Changes to the lists required
  • Add documentation to the disks
  • Support for PowerPC
  • Create Debian package (almost finished)

Another interesting feature will be the libisofs [0] support provided by
the python-libisofs extension, which is currently under development. [1]

Previous release announcements

Links

Written by Julian Andres Klode

April 4, 2008 at 20:00

Posted in Debian, debimg

Follow

Get every new post delivered to your Inbox.