Archive for the ‘Debian’ Category
Internship / APT stuff
This week was a rather busy week. I’m currently doing a (unpaid) 1 month internship as part of my education. Thanks to Michael Vogt and his boss at Canonical Ltd, this internship takes place in IRC and is dedicated to Debian and Ubuntu stuff, primarily APT-related things.
The first two days were spent on multi-arch support in python-apt: On Monday, I released python-apt 0.7.100.3, introducing initial minimal multi-arch support (just enough to not break anymore, but no really new multi-arch-specific API). This release is also the base for the version going to be shipped in Ubuntu natty, which is one of the reasons to keep the changes such minimal. I also fixed an RC bug related to Python 3.2 modules in python-apt, and implemented nocheck build option and disabled test errors on hurd.
On Tuesday, I released python-apt 0.8.0~exp1 to experimental. This release now has the old-style non-PEP8 API disabled and also introduces improved multi-arch support, by introducing bindings for APT’s GrpIterator class, and supporting indexing the cache by (name, architecture) tuples.
On Wednesday, I noticed a strange bug in APT (via python-apt’s test suite) where what the cache considered the native architecture was not the configured one. David Kalnischkies and I debugged the problem, and he found the source of the problem and implemented a fix in his branch of APT. I also introduced multi-arch support for the aptsources module, fixed all Python 3.2 ResourceWarnings in python-apt, and prepared an NMU for python-debian, to adjust it to python-apt’s new API. I also took over maintenance of software-properties in Debian, and did two uploads there (rebased on the Ubuntu package), both with python-apt 0.8 API support.
On Thursday, I shifted a bit more to the Ubuntu side and fixed several bugs in APT and aptdaemon, resulting in the aptdaemon 0.41+bzr614-0ubuntu2 upload and apt 0.8.13.3ubuntu2. I also fixed software-properties KDE version in Debian, as I broke it the previous day.
Today, on Friday, I fixed one more bug in APT. APT now treats Release files that cannot be verified identical to Release files without signature, that is, they are actually parsed now (no more missing Origin fields) – see LP: #704595.
I uploaded dh-autoreconf 3, fixing all bugs in the BTS except for one (if someone knows why autopoint depends on git, please tell me, and I may fix this bug as well). For those who don’t know dh-autoreconf, it is a tool to run autoreconf automatically during the package build, so no need for manual cleanup or autoreconf patches.
I now thought about adding the option to automatically patch ltmain.sh to dh-autoreconf. As many know, ltmain.sh does not work correctly with -Wl,–as-needed. Now, if the libtool maintainer cooperates and provides a patch file in the libtool binary package, dh-autoreconf could automatically apply it during build-time, thus fixing this problem as well.
I’m now running GNOME 3, or the parts of it we have in Debian.
We’ll probably see python-apt 0.8.0~exp2 next week with more improved multi-arch support and other fixes.
As some already know, is that I use sbuild to build all my packages in clean chroot environments. For this, I use the ‘aufs’ “mode” of schroot, that allows you to setup a chroot with one read-only base directory and one writeable overlay where changes are written to.
One thing I had problems with was the time required to install build dependencies due to disk I/O. Given that I have 4G RAM in my computer, I decided to use a tmpfs as the writable overlay. For those of you who want to do the same, putting the following in /etc/fstab makes it work:
overlay /var/lib/schroot/union/overlay tmpfs defaults 0 0
Build times are much faster this way because all write I/O is directed to the RAM
In today’s Linux distributions, there are usually two to three levels of package management. In this blog post, I will explain the three levels of package management.
1. Local (dpkg, rpm)
The first level of package management is the ‘local’ level. This level consists of package management tools that install and/or remove packages via package archives such as .deb or .rpm files and a database of the local’s system state.
2. Connected (APT, YUM, Smart)
The second level of package management is the ‘connected’ level. In this level, tools are fetching packages and information about them from (remote) locations known as ‘repositories’ or ‘sources’. A level 2 package manager is usually built on top of a level 1 package manager, for example, APT uses dpkg – but there are also cases where one tool is both, such as opkg.
3. Abstract (PackageKit, Smart)
A level 3 package manager is a tool that supports different level 1 package managers via one generic interface. It may be implemented on top of a level 2 package manager (PackageKit), or may be implemented directly in level 2 (Smart).
Conclusion: Merge Level 2 and 3
Most level 2 package managers share a great deal of tasks, such as solving dependency problems. Merging level 2 package management and level 3 package management would enable us to reduce the number of dependency problem solvers and combine our efforts into a few package managers. Such an implementation would need to be written in C and support all common level 1 package managers in order to be successful. As some might have guessed, this is what I plan to do with the project codenamed APT2, although work is not progressing very fast.
PS: Back from vacation
I spent the more than the complete last week in (mostly rainy) Corfu, Greece and am now getting back to my usual development stuff. I have already processed my news & blogs backlog (1000+ messages) and will now start with my email backlog, so don’t be angry if the answer to your email takes some time.
APT2 is now called UPS (Universal Package System). The name is inspired by the company that delivers packages in those brown trucks and from the Image Packaging System (IPS) of OpenSolaris; and mvo writing ups after I proposed upt (über package tool) in IRC. It’s definitely better than my first thought which was ‘moo’ (and libmoo).
Update: OK, let’s cancel this rename insanity.
As I wrote a few hours ago on firstname.lastname@example.org (see http://lists.debian.org/deity/2010/08/msg00057.html), APT2 is back again. The first time, I tried Vala; but this time I wrote it in C (with the help of GLib, but no GObject) and the cache uses GVariant instead of an SQLite database. It’s really basic at the moment (no solver, package installation/removal), but it will improve. Read operation should be faster than with APT, although writes are slower (this will be fixed by reusing unchanged parts of the cache).
See the announcement for further information.
Today, Nokia and Intel announced the merge of Maemo and Moblin into the MeeGo project. This is sad, because it will end the era of the Debian-based mobile operating system Maemo and replace it with a system using RPM and probably some other evil stuff as well. In fact, dpkg & apt-get where two of my main reasons to buy the N900.
And another question is why yet another name. Moblin was already a well-known name and they shouldn’t have changed the name just because they switch the servers and add some Nokia developers.
Furthermore, does this all mean that there will be no Maemo 6? What will happen to the Maemo users on the N900, will it be possible for them to use MeeGo?
Maybe this is the time to start a new project, Debian Mobile. Debian Mobile would take the MeeGo experience and apply it to a Debian system. This would probably the best mobile operating system ever, being backed by Debian’s large repositories with thousands of software packages.
I have just uninstalled hal from my laptop running unstable, and everything still seems to work, including suspend & resume.
I just uploaded python-apt 0.7.93 to unstable with support for Python 2.6 and Python 3.1, meaning that there is now a single development branch again.
This uploads brings developers the new API with real classes in apt_pkg (you can now use pydoc to view documentation), C++ bindings for making apt-pkg applications scriptable (although they should be considered experimental), a test suite (although aptsources fails in one test for now) and many new context managers for enhanced Python 3 coding fun. And objects are now freed when their reference count reaches 0. A more complete list of news can be found in the What’s New In python-apt 0.7.100 part of the documentation.
For the next releases until 0.7.100 release, the focus is clearly on fixing bugs and improving the documentation. We need more tests of the Python 3 builds, especially in areas dealing with str and unicode stuff.
Have fun, read the documentation, and code.
Well, it seems that several news sites (golem.de, pro-linux.de, linux-magazin.de, linux-magazine.com, ubuntu-user.de [the last ones all from the same publishing house]), especially German ones have picked up the last blog post with same false impressions.
First, they stated that I am planning an APT2 release for Christmas. They took the statement
[...] the internal branch has seen a lot of new code[...]. Most of the code will need to be reworked before it will be published, but I hope to have this completed until Christmas.
as a proof for this. But in the context of this paragraph, ‘publish’ was not meant in the term of publishing a release, but in the term of publishing the code (of the internal branch) in the public repository. The code is public now and lives in a ‘temp’ branch and will be reworked there for inclusion in the master branch.
Secondly, those news sites called me an Ubuntu developer. While I do contribute to Ubuntu, and am an Ubuntu Member, I am NOT an Ubuntu Developer, as I am NOT a member of the relevant ubuntu-dev team at launchpad.
Thirdly, those who stated that speed is a goal (mostly pro-linux, at least from my understanding): It is not, at least not now. It is just a coincidence caused by using a relational database. Furthermore, the test was not really fair, since the other package managers both provide more information than capt; information which has yet to be made accessible in APT2. It was just an initial conclusion that SQLite is quite fast.
Please note that APT2 is a free time project in development, and the programming language used is still in development as well; as well as some other reverse dependencies. I should also state that neither Debian nor Ubuntu have any plans to drop apt at the moment; and APT is still actively developed.
This week was successful. I have pushed some changes from November to the repository which change the license to LGPL-2.1+ (which makes bi-directional sharing of code with other projects easier, since most Vala code is under the same license) and implement HTTP using libsoup2.4 directly, instead of using GIO and GVFS for this. I also added a parser for the sources.list format which uses regular expressions to parse the file and is relatively fast. The code needs a current checkout of Vala’s git master to work correctly; as released versions had a bug which I noticed today and Jürg Billeter fixed in Vala 25 minutes later; thank you Jürg.
While nothing else happened in the public repository, the internal branch has seen a lot of new code; including SQLite 3 caches; Acquire text progress handling; and capt; the command-line advanced package tool. Most of the code will need to be reworked before it will be published, but I hope to have this completed until Christmas. It will also depend on Vala 0.7.9 or newer, which is yet to be released.
The decision to use SQLite 3 as a backend means that we won’t see the size limitations APT has and that development can be simplified by using SQL queries for filtering requests. It also means that APT2 will be very fast in most actions, like searching; which currently happens in 0.140 seconds (unstable,experimental and more repositories enabled), whereas aptitude takes 1.101 seconds, cupt (which has no on-disk cache) 1.292 seconds, and apt-cache 0.475 seconds. Searching is performed by one SQL query. I also want to thank Jens Georg <email@example.com>, who wrote Rygel’s Database class which is also used with minor modifications (like defaulting to in-memory journals) in APT2 as well. Rygel.Database is a small wrapper around sqlite3 which makes it easier to program for Vala programmers.
The command-line application ‘capt’ provides a shell based on readline with history (and later on command completion) as well as direct usage like ‘capt config dump’ or ‘capt search python-apt’. Just as with Eugene’s cupt, ‘capt’ will be the only program in the core APT2 distribution and provide the same functionality currently provided by apt-get, apt-config and friends. The name is not perfect and can be easily confused with ‘cupt’, but it was the closest option for now; considering that the name ‘apt’ is already used by Java (for its “Annotation Processing Tool”).
That’s all for now, I’ll tell you once all those features have passed my QA, and there is really something usable in the repository. In the meanwhile, you can discuss external dependency solvers, database layouts and other stuff in their threads on firstname.lastname@example.org.
And a ‘screenshot’ from capt:
jak@hp:~/Desktop/APT2:temp$ capt apt$ help APT2 0.0.20091213 command-line frontend Commands: config dump Dump the configuration config get OPTION Get the given option config set OPTION VALUE Set the given option search EXPRESSION Search for the given expression show PACKAGE Show all versions of the given package sources list Print a list of all sources version Print the version of APT2 apt$ search python-apt build-depends-python-apt - Dummy package to fulfill package dependencies python-apt - Python interface to libapt-pkg python-apt-dbg - Python interface to libapt-pkg (debug extension) python-apt-dev - Python interface to libapt-pkg (development files) python-aptdaemon - Python module for the server and client of aptdaemon python-aptdaemon-gtk - Python GTK+ widgets to run an aptdaemon client apt$