Archive for the ‘debimg’ Category
Looking at the amd64 CD images of Lenny, I just saw that there is the package ‘gnome’ on disk 1, while some of its dependencies are on disk 2. I don’t think that this is good. Same also applies to K3B on disk 3, and openoffice.org-kde on KDE-disk 1.
In my opinion, all packages which are located on a disk X, should only depend on a disk N (N<X). This means that the package gnome would be moved to disk 2, or its dependencies to disk 1.
This is exactly the way debimg works. Debimg uses so-called package groups, which simply represents a set of packages with cyclic dependencies. These package groups are returned in a specific order, so that a group N only requires a group <N. This order is kept when the groups are added to the disk. Furthermore, we treat the groups as one when adding them to the disk, ie. we check whether the whole group fits on the disk (and add it) or not (create a new disk). This way, we ensure that all dependencies can be satisfied.
Today, I decided to try it out and therefore wrote a small script reading the debian-cd task files and comparing the file list (of the packages added to the disk) with the file list of the official KDE disk.
The results look very good, debimg adds 11 packages and removes 26 which is not that much, and mostly caused by wrong size limits, etc. You can look at the results your self, and regenerate them using:
- http://debimg.alioth.debian.org/tests/debian-500-amd64-kde-CD-1.diff – The results
- http://debimg.alioth.debian.org/tests/debian-cd.py – The script used.
- http://debimg.alioth.debian.org/tests/debimg.tar.gz – The version of debimg used (current HEAD)
Anyway, debimg still needs real configuration and handling of the debian installer and source packages, but this should give you an idea of how debimg works. And a tarball, for those who don’t want to use git all the time.
BTW, I have now enabled comment threading here, a new WordPress feature. I hope that it works, as I have not tried it out yet.
Much happened since the last time I wrote about debimg. The project is now registered on Alioth and has a mailing list. On the code side, there have also been several changes.
First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the ‘image’ module.
The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it.
Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. ‘[PATCH 1/2]’). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations.
Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I’m not uploading the current state as 0.1.0~a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0~b1 then).
What is wanted/planned?
- Compatibility configuration formats
- (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
- (debimg.config.debian_cd) There could be an implementation of debian-cd’s configuration format. This would allow people to easily try out debimg. We can only support a subset, though.
- New configuration formats
- (debimg.config.yaml) Debimg’s native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.
- Graphical Frontends:
- (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
- (debimg.frontend.qt4) The graphical frontend written in QT4.
- Text frontends
- (debimg.frontend.text) The basic command-line frontend.
- (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It’s also not flexible enough for building custom images.
Requirements for 0.1 Beta
- Implement at least one configuration format and the build frontend.
- Have at least one external contributor.
- Mailing-List: email@example.com (Archive at http://lists.alioth.debian.org/pipermail/debimg-devel/)
- Wiki: http://wiki.debian.org/DebImg
- VCS-Git: git://git.debian.org/debimg/debimg.git
- VCS-Browser: http://git.debian.org/?p=debimg/debimg.git
Julian Andres Klode (6):
- debimg/core/files: When creating a file object allow filename as source
- debimg/core/resolver.py: Introduce Package.fullname, Package.component
- debimg/core/repository.py: Merge the repository module.
- debimg/core/resolver.py: Improve handling of certain dependency types
- debimg/core/repository.py: Allow Repository.add_group to take a ‘distro’ parameter
- debimg/core/image.py: Introduce the image module.
As I wrote Monday in “debimg core example”, I have an extended version which creates a repository. The problem on Monday was that the code was not far enough to be published. Not much has changed yet, but I considered to create a temporary branch temp/repository to be able to show you the example, and to give you an impression of what the final API will look like (it’s not complete yet, trust me).
Now I will present you the example. It is not much longer than the pool example I gave on monday, but it does much more – it creates a repository (and it has some commandline options). The concept of the Repository is very simple, too. We have two classes, Repository and Distribution, and a Repository object contains multiple Distribution objects, which are responsible for packages and related stuff.
#!/usr/bin/python # Copyright (C) 2009 Julian Andres Klode. # # Released under the terms of the GNU General Publice License, version 3 # or (at your option) any later version. # """Example to demonstrate the power of debimg's repository management. This example creates a directory example.debimg, with two subdirectories: - pool: This holds debimg's file pool (see debimg.core.files.Pool) - repo: This holds the created repository (debimg.core.repository) It creates a repository of all required packages, using your local apt sources. """ from __future__ import with_statement import operator import optparse import sys from debimg.core.resolver import Resolver from debimg.core.files import Pool from debimg.core.repository import Distribution, Repository def main(): """Called when the script is executed.""" parser = optparse.OptionParser() parser.add_option('-u', '--user', help='Same as in GPG') parser.add_option('-c', '--contents', action="store_true", help='Create Contents-*.gz files') opts, args = parser.parse_args() # Create a pool, which manages the access to the files. pool = Pool('example.debimg/pool') # Create a resolver using your local apt configuration. pkgs = Resolver() # Create a new repository repo = Repository(pool, 'example.debimg/repo') # Create a new distribution and store it in dist. dist = repo.add_distribution('lenny') # Add all packages with priority required to the resolver pkgs.add_priority('required') # Add the packages from the resolver to the distribution for group in pkgs.groups(): for package in group: dist.add_package(package) pool.compact() # Optional step pool.fetch() # Fetch all the packages from the mirror dist.finalize_files() # Link the files into the repository dist.finalize_packages() # Create Packages files (uncompressed and gzip) ## # If we want to create Contents-*.gz files, we can do this as well: # but we don't want to if opts.contents: dist.finalize_contents() # Now create our Release files, and sign them using the key provided on # the commandline. If no key has been provided, do not sign it. dist.finalize_release(Suite='testing', key=opts.user) for file in sorted(pool._files, key=operator.attrgetter('uri')): print file # Print information about every file, sorted by URI. if __name__ == '__main__': main()
To try the code, checkout the temp/repository branch at git://git.debian.org/users/jak/debimg.git and execute the script named test-repository, which is the same as printed above. If you have questions, please ask. Also read the docstring of debimg.core.repository to see which changes are not yet done (eg. common base class for Distribution and Repository, and of course adding files manually).
The following example demonstrates the features of debimg core, and how it can be used to fetch a some packages. As you will see when you run this example, debimg uses SHA1 filenames for the downloaded files. This may be changed in a future version.
There is also an improved version of this example, which creates a repository, but the needed module (debimg.core.repository) is not public yet, because its far from being finished. I expect to complete repository code on Wednesday.
#!/usr/bin/python # Copyright (C) 2009 Julian Andres Klode. # # Released under the terms of the GNU General Publice License, version 3 # or (at your option) any later version. # """Example to demonstrate the power of debimg's pool This example creates a directory example.debimg, with two subdirectories: - pool: This holds debimg's file pool (see debimg.core.files.Pool) """ from debimg.core.resolver import Resolver from debimg.core.files import Pool def main(): """Called when the script is executed.""" # Create a pool, which manages the access to the files. pool = Pool('example.debimg/pool') # Create a resolver using your local apt configuration. pkgs = Resolver() # Add all packages with priority required to the resolver pkgs.add_priority('required') # Add the packages from the resolver to the pool for group in pkgs.groups(): for package in group: pool.add_package(package) pool.fetch() # Fetch all the packages from the mirror for file in sorted(pool._files, key=lambda k: k.uri): print file # Print information about every file, sorted by URI. if __name__ == '__main__': main()
Today, I have published the first pieces of debimg 0.1. The published code includes the resolver, the compression module and the fetcher module. It is rewritten from scratch, this time with a PEP8-conforming style, and more flexible.
Because debimg.core does not depend on any specific configuration format, but is configured solely via parameters, it is more flexible than debimg 0.0. This new code enables people to write their own programs related to Debian images easily. And if you want to, you can use the facilities provided by debimg.configuration and debimg.frontend to write your own program using debimg configuration files (not implemented yet, core needs to be finished first).
Debimg 0.1 will use YAML configuration files, supports multiple repositories, and much more. We could even implement support for pinning packages.
Debimg 0.1 is more than the others. It is a library (actually a python package, debimg.core), it is a program designed for end-users and developers. The enormous flexibility allows us to create applications for almost everyone.
My vision is that someone who needs a custom Debian image simply fires up the debimg GTK+ frontend, selects the packages he/she wants and clicks build. And when someone needs more flexibility, there will be configuration files which lets you configure most aspects. And if this does not suffice, you can write your own application by importing the modules, replacing some functions, methods, etc. and simply call the main() function of the frontend.
- flexibility, high speed, and cool features.
- be a show case for the features of the low-level python-apt bindings (apt_pkg, apt_inst)
- high-quality code, no hacks, every single function/method/class/module documented.
- be a library, and a program.
- provide a graphical front-end to assist the unexperienced users.
- provide powerful file-based configuration for advanced developers.
- And finally, provide a replacement for debian-cd in the near future.
- Email: http://lists.debian.org/debian-cd/2009/02/msg00034.html
- Vcs-Git: git://git.debian.org/users/jak/debimg.git
- Vcs-Browser: http://git.debian.org/?p=users/jak/debimg.git;a=summary
Well, you may have noticed that debimg 0.1 is still not released. But a lot of work happened over the weekend in my local branch.
First of all, debimg’s set support is almost finished. I uploaded a tarball containing the differences between the official lenny weekly build from yesterday and a build created today by debimg, using the tasks of debian-cd 3.0.4 (after manual conversion to a format supported by debimg). Look at http://jak-linux.org/cdimage/tests/ for the tarball.
Secondly, the dependency resolver has been rewritten. It’s a bit slower now (0.72 seconds for main), but creates much better results. Resolving the dependencies of all packages in Debian Lenny i386 in alphabetical order, debimg 0.0.X resolved 206 dependencies differently than apt. Now, these have been decreased to 15 dependencies, whereas 13 dependencies are false-positive (some packages were not installed because they were already installed). This means that only two ones were different, in this caseachims-guestbook and chdrv, which both depend on virtual-only packages (achims-guestbook: apache | httpd, chdrv: console-utilities).
The third big change is the addition of the hooks module. This module allows you to hook in custom functions, which have access to the Configuration object (ConfigObj) and the MediaSet. There are currently three types of hooks: pre_hooks (run before fetching packages, adding files to the disk), mid_hooks (run after the packages have been fetched) and post_hooks (run after the image has been built). Hooks can be added based on project and architecture, using a simple syntax which support shell patterns. (It’s ‘project/arch’). The hooks module uses python decorators to register functions. debimg 0.2 will switch to hooks for internal functions, too, like bootloaders and other stuff.
The code has not been merged into the master branch, but I will hopefully be able to merge it tomorrow. The release of debimg 0.1 is now planned for this weekend.
debimg 0.1 will be the first major milestone in the development of debimg. I will now list some changes compared to the current release, debimg 0.0.3 (Please note that the following text is from an internal NEWS file and may not be correct in some aspects, as it is already some days old)
Support for disk splitting
This release of debimg adds support for creating media sets, i.e. splitting the packages over multiple disks. This is achieved by providing a new class called MediaSet, which passes all calls to methods to all medias.
When a package is added to a MediaSet, MediaSet tries to find the first disk where it can be added, by checking if all dependencies of the package are provided on the disk (or previous ones). This dependency checker does no recursive dependency checks, and can be disabled via OptimizedDiskSplitting = False (or no, n, 0…) in the configuration file.
The MediaSet classes are also lists, so you can access disk one via the index 0. BaseMedia has been modified to also support being accessed via index, using disknumber-1 in order to have both classes share the same API.
New data lists
This release of debimg brings users a more powerful way to include packages. Instead of various keys to include packages, debimg now uses the Include key.
To include packages, you have to use the ‘Include’ field. The value is a list of items, separated by commas. An item may either be the name of a package or a special form written as key:value. In this case, the following keys are supported:
This includes all packages belonging to the specified task or priority. In
case of tasks, debimg first includes all Key packages, then the other
Configuration files are easier
This release of debimg makes it possible to build multiple architectures using the Architectures option. This option replaces the previous Architecture option, which is now set automatically by the script for each architecture.
It also enables the Projects option, in case you want to build multiple projects. (All need to have the same MediaType and same NumberOfDisks). The project may also be cd-set, dvd-set, dvd9-set, netinst or businesscard. In these cases, MediaType is set automatically. [MediaType may also be moved in to data files].
This release also allows you to prefix any configuration value with path: in order to automatically convert the value to an absolute path. This is needed for many file options, but should not be used for some other stuff.
Improved Jigdo Support
debimg 0.1 allows you to define the public path (name) to (of) the image file and the template file that is written to the Jigdo file.
Use urlgrabber for file downloading
Starting with this release, debimg uses urlgrabber to download all kinds of files (except packages). A urlgrab call has been added to BaseMedia.addFile() which now understands http:// and ftp:// urls.
debimg 0.1 may introduce support for registering custom functions. This feature is low-priority, although required for easy development of Ubuntu-related code.
debimg is a GPL-3 licensed software designed to replace debian-cd, a tool to create Debian images. For further information about debimg, visit the Wiki page. For more information about Debian, visit the website at www.debian.org.