Back to the ’90s – Bye PC, welcome back thin clients

In the ’90s, you had a large machine and several thin clients accessing it by using X11 via network. In 2010, you will have large datacenters providing applications to and storing the data of millions of users. As you might have guessed, I am talking about Google Chrome OS.

It seems that the PC era is slowly coming to an end, with devices being increasingly connected ‘to the cloud’ and people being always online; and storing their data on Google’s servers. We do emails online using Google Mail, we do navigation online using Google Maps, we edit and view our documents using Google Docs, our newspaper is Google News; and when we want entertainment we open the browser and type youtube.com into the URL bar. Even if we were formatting the hard disk and reinstalling the system, most people wouldn’t even notice; because all there data is stored online.

There is also the question of freedom. Free software is not very widespread in the SaaS world. You also lose the control over your data. But RMS can tell you more about it.

So it seems that 2010, Google is the new mainframe and netbooks and smartphones are the new terminals. Whether this is good or not cannot be said. The question you have to ask yourself is whether you can trust Google to keep your data secure or not. I trust them enough to host all of my emails, the RSS feeds I read, my searches.

My First upload with new source format

Yesterday, I uploaded command-not-found 0.2.38-1 (based on version 0.2.38ubuntu4) to Debian unstable, using the “3.0 (quilt)” source format. All steps worked perfectly, including stuff like cowbuilder, lintian, debdiff, dput and the processing on ftp-master. Next steps are reverting my machine from Ubuntu 9.10 to my Debian unstable system and uploading new versions of gnome-main-menu, python-apt (0.7.93, not finished yet) and some other packages.

In other news, the development of Ubuntu 10.04 Lucid Lynx just started. For the first time in Ubuntu’s history, the development will be based on the testing tree of Debian and not on the unstable tree. This is done in order to increase the stability of the distribution, as this release is going to be a long term supported release. Ubuntu will freeze in February, one month before the freeze of Debian Squeeze. This should give us enough room to collaborate, especially on bugfixes. This also means that I will freeze my packages in February, so they will have the same version in Squeeze and Lucid (applying the earliest freeze to both distributions; exceptions where needed).

I’m back

I returned to Germany from my vacation in Greece yesterday, and I just installed my new hard drive into my laptop. The old Hitachi hard drive had some bad sectors after a very long usage time (compared to my other disks) – we’ll see how the new Samsung SpinPoint M7 will work. Another side benefit is the upgrade from 120GB to 500GB which means I don’t have to delete files during the next months. It is also much faster (hdparm -t was 80MB/s, 2 times faster than the old one).

I’m currently running the release candidate of Ubuntu 9.10 “Karmic Koala” on this system, but I expect to return to my full Debian unstable developer environment during the next week(s). Karmic seems to be pretty stable already, but I have experienced problems with PulseAudio and my PCM control getting set to +13dB which is horrible.

I will probably also pre-order a Nokia N900 soon, which will be my largest investment this year – but I really need a new mobile phone, camera and music player. And not to forget the ability to develop software for it, and testing my software on those less powerful devices (compared to my laptop).

All in all, I’m back and soon ready to hack again.

upgrade failures

Today I wanted to upgrade my “sid” system again (like I do 1-3 times per day, especially when I have nothing else to do). First of all, I was hit by a bug in APT “Could not perform immediate configuration (2) on perl”. I worked around it by running the latest git commit of cupt, which then started to upgrade my system. At some stage however, after upgrading parts of perl it seems, the progress stopped and perl complained that it could not find English.pm anymore. I tried to manually install the perl packages using dpkg -i, but this did not work. Then I just thought I should just try to unpack it and run dpkg -x <package> / on the perl packages. Now dpkg -i was working again and I could run dpkg -i on the packages again and continue my upgrade using apt-get.

Results of the APT2 config parser testing

Thanks to those who have tested it (and/or will test it). The results where helpful and resulted in Bug#548443 filed against localepurge and GNOME Bug #596429 against glib. The first one is a case where quotes where used inside a value, although this has never been defined to work, and the second one is a problem with GLib’s GScanner not ignoring multi-line C-style comments although it was configured to do so.

I also fixed some bugs in APT2, like the missing build-depends on libgee-dev and the configuration parser now accepts ‘.’, ‘_’, ‘+’ in the option name. I also talked with Eugene about some differences in the way cupt and APT2 handle quotes and about some other parts of the configuration format. Seems this was a good day.

APT2: config parser testing

If you have an amd64 system, install the apt2 package from “deb http://people.debian.org/~jak/debian/ unstable/” and run the apt2-config command. Make sure that the parser reports no errors, otherwise send me an email or leave a comment here. One known exception is that all values must be quoted in the configuration file, I have no plans to fix this (probably just deprecate unquoted strings in APT instead). The parser is not as strict as cupt’s parser, but it gives you more help if something wents wrong. We also ignore most semicolons for now (they will be turned into warnings or errors later on). It is using GScanner from GLib for parsing the files.

apt2-config replicates the functionality of apt-config. It currently do not read the configuration files in the correct order, so don’t expect “apt2-config dump” to produce exactly the same output as “apt-config dump”. To build APT2 yourself, you need a patch for waf from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=548329.

Chromium on Linux can print now

Just updated to revision 26808 of Chromium today, and it supports printing now. But it still does not support password encryption it seems. It also has some problems with displaying pages sometimes (buttons missing, style not loaded, etc.). But it improved a lot since I first tried it.

cupt and how to write package managers

cupt is a new package manager written in Perl by Eugene V. Lyubimkin, who previously contributed to APT. And more than all, the project makes no sense at all.

First of all, there is a language issue. Implementing a package manager in Perl has some major drawbacks. One of the features of APT was it being written in a lower-level language (i.e. C++ which really is below Perl), making it possible to write applications like synaptic and python bindings which in turn lead to applications like gnome-app-install or Ubuntu’s new Software Store.

Furthermore, writing a package manager in Perl means that Distributions such as Emdebian might not be able to use it since they have excluded Perl due to its space requirements. This becomes even more important considering that cupt depends on even more perl libraries. This means that cupt will never be able to replace APT.

Secondly, a package manager should not be designed specifically for one distribution. This is another major drawback of cupt and other package managers such as yum or zypper. The smart package manager, written in Python and funded by Canonical Ltd. is an example for a distribution-neutral package manager.

Now let’s take a look at package management in modern distributions. Usually we have two levels of package managers, the first being tools like dpkg and rpm which take care of installing and removing the packages and level-2 package managers implementing dependency resolvers and package retrieval from remote locations. Recently, distributions started to add a third layer named PackageKit, which shall provide distribution-independent package management user interfaces. The project was well-received by RPM-based distributions, but failed in Debian-based distributions due to not supporting debconf. Furthermore, adding a third layer just increases the possibility of problems.

The right way to do package management is a distribution-independent level-2 package manager written in C. The smart project shows us that this is possible although itself fails to meet the lower-level language (C) requirement. That’s why I decided to write a package manager in Vala, a GObject-based language which gets converted to C and then compiled. If successful, this project will be able to replace most of the current level-2 package managers and will also provide the same distribution-independence as provided by a level-3 package manager such as PackageKit. It is also easy to create binding for other programming language such as Python or Perl thus enabling application developers to choose the language they like most.

The core of this project is a vendor-neutral library, temporarily called libapt (as the project is called APT2 for now). This library contains all the code which is not specific to a vendor i.e. file retrieval, dependency resolver, caches, etc and  is then enhanced by several vendor-specific plugins, each implementing a PackageManager (interface to the distribution’s level-1 package manager) and a Repository (well, repositories from which you can download packages) interface.

We could even enhance the vendor-independent interface to include more details of a repository. Most repositories nowadays consist of 4 components: A meta index (Release files for Debian, repomd.xml for Fedora/openSUSE/etc.), a package index (e.g. Packages files for Debian, *primary.xml.gz on Fedora, etc. ), a source index (e.g Sources files in Debian) and a files index (e.g. Contents-*.gz for Debian). I took a look at the repository formats of Slackware, openSUSE and Fedora and it seems that this concept can be applied to all of them. So maybe all we need are distribution dependent parsers for those files.

One of the most important issues with APT is its use of mmap() for the cache. Using mmap() makes it hard to grow the cache, which is sometimes needed. We see a lot of bug reports from people with too small cache sizes. We can circumvent this problem by utilizing an embedded database like SQLite for this, but we would probably loose some speed and it may be harder to maintain a flexible API. We should see what the best option is here, both ways are possible since Vala 0.7.6 includes my patches for adding mmap(), ftruncate(), mremap() and some other functions. An idea to circumvent the mmap() issue is gathering statistics about the relation between the number of repositories and the size of the cache and then using a value which is slightly above the average statistical value.

The project is not very mature yet, it only includes basic library functions for downloading files and parsing configuration files, etc. You can find the (MIT licensed) code at http://git.debian.org/?p=users/jak/apt2.git. I also have some local code for repository management and multi-threaded file fetching, but it’s just not ready to be merged yet.

Chromium

I have just switched to Chromium as my primary browser. I am running the daily-built version from the Ubuntu Jaunty PPA at https://edge.launchpad.net/~chromium-daily/+archive/ppa on my Debian unstable box. It seems quite stable, plugins are also working (with the –enable-plugins option) and it can use the system’s GTK+ theme for most parts (the buttons, etc. inside webpages are not rendered using GTK+ yet, but the UI is). It currently cannot print and it also has some formatting issues on some websites, and it’s not passing the ACID3 test yet (there is a ‘X’ in the top-right corner). I could have switched to Midori instead, but Midori is missing a cache it seems (the option can not be enabled).

I ran the V8 and SunSpider benchmarks to compare Iceweasel 3.5, Midori 0.1.9 (using WebKit GTK+ 1.1.12) and Chromium 4.0.206.0 (r25168). The result was that Iceweasel was 10x times slower than the others in the V8 benchmark and about 5 times slower in the SunSpider benchmark. The others were almost equally fast, but Chromium won the V8 benchmark with 2338 points compared to Midori’s 1666. More details are in the PDF Browser Performance, which should have been an ODP, but uploading ODPs is not allowed on wordpress.com. I also ran the V8 benchmark on Arora some time ago, but it was almost as slow as iceweasel. All tests were done on my laptop running Debian GNU/Linux unstable (amd64 architecture).

The APT2 project

I just started working on a replacement for APT written in Vala and called APT2 (I know, the name could be better). The main idea behind the project is to create a library for working with Debian repositories and packages, and on top of this library a few applications. This is different from APT because the project focuses on the library part, whereas APT is primarily focused on the application part.

I chose Vala for this task because it has an easy syntax and allows me to use the features provided by GLib and Co. really easily. GLib provides many functions needed for APT like file access, unicode handling, hashsums creation. It also provides us with signals which are useful for implementing progress reporting.

Another project, called ‘cupt’ tries to provide an APT replacement in Perl. Apart from the fact that I don’t like Perl, it also has the disadvantage of being too big to ship on embedded devices and is not useable from other languages such as Python or C/C++. Yet another project, ’smart’ provides a different package manager written in Python. While I like Python, it can’t be used from languages such as Perl or C/C++ and is too large. APT2 will only require GLib, Gee and libarchive, which require about 2MB of space; about 1/10 of Perl’s size.

The core of APT2 is a library called ‘libapt’. At the moment, this library provides classes for reading tag files (TagFile&TagSection) and an incomplete configuration class based on libgee’s HashMap (to ease coding, it will probably be changed to a special data structure like those in apt-pkg). The next step will be implementing a parser for configuration files (currently we only support in-memory objects) and writing the Acquire system.

The planned Acquire system uses GModule to modularize the support for different URI schemes. Each module provides a worker class which implements one or more URI schemes. The first of these modules will provide a worker using GIO, which deals with local file access, samba shares, FTP, SFTP, WebDAV and various other protocols, including HTTP until a replacement has been written (since we don’t want to force gvfs-backends). The workers communicate with the parent Acquire object using signals and can cause the whole acquisition to be aborted by emiting an “abort” signal (or similar).

The whole project has just been started and may take some months until it becomes useful (if it becomes useful at all). The code can be found at http://git.debian.org/?p=users/jak/apt2.git and is licensed under the MIT license (see http://www.opensource.org/licenses/mit-license.php). It is a secondary project and will not affect my work on apt (not much yet) and python-apt (0.8 series).