Archive for the ‘Debian’ Category
Yesterday was my 21st birthday, and I received all “Hitchhiker’s Guide to the Galaxy” novels, the five ones in one book, and the sixth one written by Eoin Colfer in another book. Needless to say, the first book weights more than an N900. I did not read them yet, so now is the perfect chance to do so. Yes, I did not know that 25th is towel day, sorry for that.
I also bought a Toshiba AC100 before my birthday, a Tegra 2 based notebook/netbook/”web companion” with 1 GHz dual core ARM Cortex A9 chip and 512 MB RAM. It runs Android by default, and had a price of 160€ which is low compared to anything else with Cortex A9. It currently runs Ubuntu 11.04 with a specialised kernel 2.6.37 from time to time, without sound and accelerated video (and not functioning HDMI). Mostly waiting for Nvidia to release a new binary blob for the video part (And yes, if you just want to build packages, you can probably get happy without those things).
Another thing happening last week is the upload of python-apt 0.8.0 to unstable, marking the beginning (or end) of the API transition I started more than a year ago. Almost all packages not supporting it have proper Breaks in python-apt [most of them already fixed, only 2 packages remaining, one of which is "maintained" (well, not really maintained right now) by me], but there may be some which do not work correctly despite being fixed (or at least thought to be fixed).
If you know any other interesting thing I did last week, leave a comment, I wrote enough now. And yes, WordPress wants to write a multiplication sign instead of an x, so I had to use x instead.
I did not write much code or merge much of my prototype code, but some things happened since the last blog post about APT2 specific things in August and I forgot to write about them.
First of all, I dropped the GVariant-based cache. The format strings were simply getting ugly long and were not very understandable, performance was just much too slow (needing more than a few nanoseconds for a package lookup is obviously too slow for solving dependency problems); furthermore, building the cache was also slow and complicated because we needed all attributes of an object at once to pass them to GVariant, leading to ugly API.
I replaced the GVariant cache with one that can be easily mmap()ed and is described completely in C. It’s derived from APT’s cache design (but more robust, as it includes the size of the cache and we can thus detect to small files, although that’s scheduled for the next ABI break in APT as well), but has fewer duplicate data, and uses arrays where APT uses linked lists. The reason for arrays is simple: They take up less space and can be represented naturally in Python and other languages using array-based lists. The cache also contains a coalesced hash table which does use a linked list, but that one is a bit different, as it is for searching only and not exposed. Everything non-stringy is 64-bit aligned in order to keep things as simple as possible. All integers are fixed size, thus the format is architecture-independent if you fix byte orders. The format is described at http://people.debian.org/~jak/apt2-doc/apt-Cache-Format.html.
I stole one more idea from cupt and changed the configuration system to verify types of variables. APT2’s configuration system knows more types than cupt’s, though, including regular expressions, directory and filenames (i.e. it does not let you store a value /d/ in a file variable), strings (which store everything), unsigned and signed integers, and boolean options; all of which are checked when parsing files (producing warnings) or command-line options (producing errors).
I have also simplified the type world by removing all iterator types except for one, replacing them with
n_things() functions in the objects holding the arrays. Makes cool bindings slightly harder, but makes the C API much easier to use from C.
Most things expected from a package manager are still missing, but what is there looks good in most cases (especially
AptConfiguration has a nice API, and no complaints from valgrind anywhere). Currently I am working on Python bindings so I can interact with the functions easily and check things in an interactive fashion; and I am also writing a document explaining the concepts behind APT2, drafts at http://people.debian.org/~jak/midlevel.pdf. I also have some more code pending further thoughts (including complete index parsing), but it might still take some time before I have something usable in the wild.
On other package managers: From time to time I also use Cupt, look at Cupt code, hack Cupt code, and report bugs against Cupt. I still do not really understand the (extreme) nesting of directory structures in the source code, why there are so (extremely) many source files split all over them, or the general concepts of Cupt, but I can hack together what I need for my personal testing. I also play with yum whenever I end up on a Fedora system (which happens from time to time).
The last two weeks, two new python-apt releases were made. 0.8.0~exp3 did not add much, but 0.8.0~exp4 added some new bindings for our friends at the mancoosi project. I also committed several fixes to the APT repository, but did not upload them yet.
In #debian-devel, some people (including me and others on the Debian side; and sladen, sabdfl for the Ubuntu side) discussed the Ubuntu font license which is considered non-free by Debian, due to extreme naming restrictions in section 2 (unmodified versions must keep the name, slightly modified versions must keep the name and add something). Some consider those restrictions equivalent to invariant sections. After we discusses the font license, we quickly got to discuss Doctor Who and time travel, as those two are obviously connected.
Some other things happened as well, like closing more bugs, but all in all, the last two weeks where a bit less intensive than the two weeks before them.
python-apt 0.8.0~exp2 bug fix release
On Tuesday, I uploaded python-apt 0.8.0~exp2 to experimental, fixing about 10 bugs reported in Ubuntu and Debian bug trackers. It should know even convert integers correctly on all architectures, previously we could have passed long via varargs where int was expected.
Until Thursday, I went through the bug list in Launchpad and closed/fixed/reassigned/merged about 100 bugs in APT and python-apt.
APT & python-apt updates for squeeze
Today, I uploaded updates of apt and python-apt to stable. They include support for xz and parsing multi-arch dependencies, as wanted by ftpmasters.
APT 0.8.14 and wildcards/regular expression pinning
Today, I uploaded apt 0.8.14 to Debian unstable, introducing support for pinning using glob() like Syntax and POSIX extended regular expressions. Let’s say we want to pin all packages starting with gnome or kde to 990. The following example does this, using glob-like patterns for gnome, and a regular expression enclosed in / for kde:
Package: gnome* /^kde/
Pin: release a=experimental
This closes 10-year-old Bug#121132 in Debian. Have fun with this feature, but please note that it may not be the fastest thing on earth, as it checks every package in the cache on initialization of such queries, which may take a few 10 ms.
Since some time already, it’s also possible to use such expressions for the Pin field. Thus users of Ubuntu releases could use the following piece of preferences to pin all packages in archives starting with lucid (e.g. lucid, lucid-updates) to 990:
Pin: release a=lucid*
Those types of pins do not have the negative performance impact of complex expressions in the Package header, as they are only checked against a smaller set of packages, or if “Package: *”, simply checked against the package files in the cache.
Internship / APT stuff
This week was a rather busy week. I’m currently doing a (unpaid) 1 month internship as part of my education. Thanks to Michael Vogt and his boss at Canonical Ltd, this internship takes place in IRC and is dedicated to Debian and Ubuntu stuff, primarily APT-related things.
The first two days were spent on multi-arch support in python-apt: On Monday, I released python-apt 0.7.100.3, introducing initial minimal multi-arch support (just enough to not break anymore, but no really new multi-arch-specific API). This release is also the base for the version going to be shipped in Ubuntu natty, which is one of the reasons to keep the changes such minimal. I also fixed an RC bug related to Python 3.2 modules in python-apt, and implemented nocheck build option and disabled test errors on hurd.
On Tuesday, I released python-apt 0.8.0~exp1 to experimental. This release now has the old-style non-PEP8 API disabled and also introduces improved multi-arch support, by introducing bindings for APT’s GrpIterator class, and supporting indexing the cache by (name, architecture) tuples.
On Wednesday, I noticed a strange bug in APT (via python-apt’s test suite) where what the cache considered the native architecture was not the configured one. David Kalnischkies and I debugged the problem, and he found the source of the problem and implemented a fix in his branch of APT. I also introduced multi-arch support for the aptsources module, fixed all Python 3.2 ResourceWarnings in python-apt, and prepared an NMU for python-debian, to adjust it to python-apt’s new API. I also took over maintenance of software-properties in Debian, and did two uploads there (rebased on the Ubuntu package), both with python-apt 0.8 API support.
On Thursday, I shifted a bit more to the Ubuntu side and fixed several bugs in APT and aptdaemon, resulting in the aptdaemon 0.41+bzr614-0ubuntu2 upload and apt 0.8.13.3ubuntu2. I also fixed software-properties KDE version in Debian, as I broke it the previous day.
Today, on Friday, I fixed one more bug in APT. APT now treats Release files that cannot be verified identical to Release files without signature, that is, they are actually parsed now (no more missing Origin fields) – see LP: #704595.
I uploaded dh-autoreconf 3, fixing all bugs in the BTS except for one (if someone knows why autopoint depends on git, please tell me, and I may fix this bug as well). For those who don’t know dh-autoreconf, it is a tool to run autoreconf automatically during the package build, so no need for manual cleanup or autoreconf patches.
I now thought about adding the option to automatically patch ltmain.sh to dh-autoreconf. As many know, ltmain.sh does not work correctly with -Wl,–as-needed. Now, if the libtool maintainer cooperates and provides a patch file in the libtool binary package, dh-autoreconf could automatically apply it during build-time, thus fixing this problem as well.
I’m now running GNOME 3, or the parts of it we have in Debian.
We’ll probably see python-apt 0.8.0~exp2 next week with more improved multi-arch support and other fixes.
As some already know, is that I use sbuild to build all my packages in clean chroot environments. For this, I use the ‘aufs’ “mode” of schroot, that allows you to setup a chroot with one read-only base directory and one writeable overlay where changes are written to.
One thing I had problems with was the time required to install build dependencies due to disk I/O. Given that I have 4G RAM in my computer, I decided to use a tmpfs as the writable overlay. For those of you who want to do the same, putting the following in /etc/fstab makes it work:
overlay /var/lib/schroot/union/overlay tmpfs defaults 0 0
Build times are much faster this way because all write I/O is directed to the RAM
In today’s Linux distributions, there are usually two to three levels of package management. In this blog post, I will explain the three levels of package management.
1. Local (dpkg, rpm)
The first level of package management is the ‘local’ level. This level consists of package management tools that install and/or remove packages via package archives such as .deb or .rpm files and a database of the local’s system state.
2. Connected (APT, YUM, Smart)
The second level of package management is the ‘connected’ level. In this level, tools are fetching packages and information about them from (remote) locations known as ‘repositories’ or ‘sources’. A level 2 package manager is usually built on top of a level 1 package manager, for example, APT uses dpkg – but there are also cases where one tool is both, such as opkg.
3. Abstract (PackageKit, Smart)
A level 3 package manager is a tool that supports different level 1 package managers via one generic interface. It may be implemented on top of a level 2 package manager (PackageKit), or may be implemented directly in level 2 (Smart).
Conclusion: Merge Level 2 and 3
Most level 2 package managers share a great deal of tasks, such as solving dependency problems. Merging level 2 package management and level 3 package management would enable us to reduce the number of dependency problem solvers and combine our efforts into a few package managers. Such an implementation would need to be written in C and support all common level 1 package managers in order to be successful. As some might have guessed, this is what I plan to do with the project codenamed APT2, although work is not progressing very fast.
PS: Back from vacation
I spent the more than the complete last week in (mostly rainy) Corfu, Greece and am now getting back to my usual development stuff. I have already processed my news & blogs backlog (1000+ messages) and will now start with my email backlog, so don’t be angry if the answer to your email takes some time.