Currently if you have an automatically installed package A (= 1) where

• A (= 1) Depends B (= 1)
• A (= 2) Depends B (= 2)

and you upgrade B from 1 to 2; then you can:

1. Remove A (= 1)
2. Upgrade A to version 2

If A was installed by a chain initiated by Recommends (say X Rec Y, Y Depends A), the solver sometimes preferred removing A (and anything depending on it until it got).

I have a fix pending to introduce eager Recommends which fixes the practical case, but this is still not sound.

In fact we can show that the solver produces the wrong result for small minimal test cases, as well as the right result for some others without the fix (hooray?).

Ensuring sound removals is more complex, and first of all it begs the question: When is a removal sound? This, of course, is on us to define.

An easy case can be found in the Debian policy, 7.6.2 “Replacing whole packages, forcing their removal”:

If B (= 2) declares a Conflicts: A (= 1) and Replaces: A (= 1), then the removal is valid. However this is incomplete as well, consider it declares Conflicts: A (< 1) and Replaces: A (< 1); the solution to remove A rather than upgrade it would still be wrong.

This indicates that we should only allow removing A if the conflicts could not be solved by upgrading it.

The other case to explore is package removals. If B is removed, A should be removed as well; however it there is another package X that Provides: B (= 1) and it is marked for install, A should not be removed. That said, the solver is not allowed to install X to satisfy the depends B (= 1) - only to satisfy other dependencies [we do not want to get into endless loops where we switch between alternatives to keep reverse dependencies installed].

Proposed solution

To solve this, I propose the following definition:

Definition (sound removal): A removal of package P is sound if either:

1. A version v is installed that package-conflicts with B.
2. A package Q is removed and the installable versions of P package-depends on Q.

where the other definitions are:

Definition (installable version): A version v is installable if either it is installed, or it is newer than an installed version of the same package (you may wish to change this to accomodate downgrades, or require strict pinning, but here be dragons).

Definition (package-depends): A version v package-depends on a package B if either:

1. there exists a dependency in v that can be solved by any version of B, or
2. there exists a package C where v package-depends C and any (c in C) package-depends B (transitivity)

Definition (package-conflicts): A version v package-conflicts with an installed package B if either:

1. it declares a conflicts against an installable version of B; or
2. there exists a package C where v package-conflicts C, and b package-depends C for installable versions b.

Translating this into a (modified) SAT solver

One approach may be to implement the logic in the conflict analysis that drives backtracking, i.e. we assume a package A and when we reach not A, we analyse if the implication graph for not A constitutes a sound removal, and then replace the assumption A with the assumption A or "learned reason.

However, while this seems a plausible mechanism for a DPLL solver, for a modern CDCL solver, it’s not immediately evident how to analyse whether not A is sound if the reason for it is a learned clause, rather than a problem clause.

Instead we propose a static encoding of the rules into a slightly modified SAT solver:

Given c1, …, cn that transitive-conflicts A and D1, …, Dn that A package-depends on, introduce the rule:

A unless c1 or c2 or ... cn ... or not D1 or not D2 ... or not Dn

Rules of the form A... unless B... - where A... and B... are CNF - are intuitively the same as A... or B..., however the semantic here is different: We are not allowed to select B... to satisfy this clause.

This requires a SAT solver that tracks a reason for each literal being assigned, such as solver3, rather than a SAT solver like MiniSAT that only tracks reasons across propagation (solver3 may track A depends B or C as the reason for B without evaluating C, whereas MiniSAT would only track it as the reason given not C).

Is it actually sound?

The proposed definition of a sound removal may still proof unsound as I either missed something in the conclusion of the proposed definition that violates my goal I set out to achieve, or I missed some of the goals.

I challenge you to find cases that cause removals that look wrong :D

Consider the dependencies A|B|C, A|B, B|X. In most package managers these just express alternatives, that is, the “or” relationship, but in Debian packages, it also expresses a preference relationship between its operands, so in A|B|C, A is preferred over B and B over C (and A transitively over C).

This means that we can convert the three dependencies into a trie as follows:

Solving the dependency here becomes a matter of trying to install the package referenced by the first edge of the root, and seeing if that sticks. In this case, that would be ‘a’. Let’s assume that ‘a’ failed to install, the next step is to remove the empty node of a, and merging its children into the root.

For ease of visualisation, we remove “a” from the dependency nodes as well, leading us to a trie of the dependencies “b”, “b|c”, and “b|x”.

Presenting the Debian dependency problem, or the positive part of it as a trie allows us for a great visualization of the problem but it may not proof to be an effective implementation choice.

In the real world we may actually store this as a priority queue that we can delete from. Since we don’t actually want to delete from the queue for real, our queue items are pairs of a pointer to dependency and an activitity level, say A|B@1. Whenever a variable is assigned false, we look at its reverse dependencies and bump their activity, and reinsert them (the priority of the item being determined by the leftmost solution still possible, it has now changed). When we iterate the queue, we remove items with a lower activity level:

1. Our queue is A|B@1, A|B|C@1, B|X@1
2. Rejecting A bump the activity for its reverse dependencies and reinset them: Our queue is A|B@1, A|B|C@1, (A|)B@2, (A|)B|C@2, B|X@1
3. We visit A|B@1 but see the activity of the underlying dependency is now 2 and remove it Our queue is A|B|C@1, (A|)B@2, (A|)B|C@2, B|X@1
4. We visit A|B|C@1 but see the activity of the underlying dependency is now 2 and remove it Our queue is (A|)B@2, (A|)B|C@2, B|X@1
5. We visit A|B@2, see the activity matches and find B is the solution.

The idea for the solver was that manually installed packages are always protected from removals – in terms of SAT solving, they are facts. Automatically installed packages become optional unit clauses. Optional clauses are solved after manual ones, they don’t partake in normal unit propagation.

This worked fine, say you had

A                                   # install request for A
B                                   # manually installed, keep it
A depends on: conflicts-B | C

Installing A on a system with B installed installed C, as it was not allowed to install the conflicts-B package since B is installed.

However, I also introduced a mode to allow removing manually installed packages, and that’s where it broke down, now instead of B being a fact, our clauses looked like:

A                               # install request for A
A depends on: conflicts-B | C
Optional: B                     # try to keep B installed

As a result, we installed conflicts-B and removed B; the steps the solver takes are:

1. A is a fact, mark it
2. A depends on: conflicts-B | C is the strongest clause, try to install conflicts-B
3. We unit propagate that conflicts-B conflicts with B, so we mark not B
4. Optional: B is reached, but not satisfiable, ignore it because it’s optional.

This isn’t correct: Just because we allow removing manually installed packages doesn’t mean that we should remove manually installed packages if we don’t need to.

Fixing this turns out to be surprisingly easy. In addition to adding our optional (soft) clauses, let’s first assume all of them!

But to explain how this works, we first need to explain some terminology:

1. The solver operates on a stack of decisions
2. “enqueue” means a fact is being added at the current decision level, and enqueued for propagation
3. “assume” bumps the decision level, and then enqueues the assumed variable
4. “propagate” looks at all the facts and sees if any clause becomes unit, and then enqueues it
5. “unit” is when a clause has a single literal left to assign

To illustrate this in pseudo Python code:

1. We introduce all our facts, and if they conflict, we are unsat:

   for fact in facts:
       enqueue(fact)
   if not propagate():
       return False
   

2. For each optional literal, we register a soft clause and assume it. If the assumption fails, we ignore it. If it succeeds, but propagation fails, we undo the assumption.

   for optionalLiteral in optionalLiterals:
       registerClause(SoftClause([optionalLiteral]))
       if assume(optionalLiteral) and not propagate():
           undo()
   

3. Finally we enter the main solver loop:

   while True:
       if not propagate():
           if not backtrack():
               return False
       elif <all clauses are satisfied>:
           return True
       elif it := find("best unassigned literal satisfying a hard clause"):
           assume(it)
       elif it := find("best unassigned literal satisfying a soft clause"):
           assume(it)
   

The key point to note is that the main loop will undo the assumptions in order; so if you assume A,B,C and B is not possible, we will have also undone C. But since C is also enqueued as a soft clause, we will then later find it again:

1. Assume A: State=[Assume(A)], Clauses=[SoftClause([A])]
2. Assume B: State=[Assume(A),Assume(B)], Clauses=[SoftClause([A]),SoftClause([B])]
3. Assume C: State=[Assume(A),Assume(B),Assume(C)], Clauses=[SoftClause([A]),SoftClause([B]),SoftClause([C])]
4. Solve finds a conflict, backtracks, and sets not C: State=[Assume(A),Assume(B),not(C)]
5. Solve finds a conflict, backtracks, and sets not B: State=[Assume(A),not(B)] – C is no longer assumed either
6. Solve, assume C as it satisfies SoftClause([C]) as next best literal: State=[Assume(A),not(B),Assume(C)]
7. All clauses are satisfied, solution is A, not B, and C.

This is not (correct) MaxSAT, because we actually do not guarantee that we satisfy as many soft clauses as possible. Consider you have the following clauses:

Optional: A
Optional: B
Optional: C
B Conflicts with A
C Conflicts with A

There are two possible results here:

1. {A} – If we assume A first, we are unable to satisfy B or C.
2. {B,C} – If we assume either B or C first, A is unsat.

The question to ponder though is whether we actually need a global maximum or whether a local maximum is satisfactory in practice for a dependency solver If you look at it, a naive MaxSAT solver needs to run the SAT solver 2**n times for n soft clauses, whereas our heuristic only needs n runs.

For dependency solving, it seems we do not seem have a strong need for a global maximum: There are various other preferences between our literals, say priorities; and empirically, from evaluating hundreds of regressions without the initial assumptions, I can say that the assumptions do fix those cases and the result is correct.

Further improvements exist, though, and we can look into them if they are needed, such as:

• Use a better heuristic:

  If we assume 1 clause and solve, and we cause 2 or more clauses to become unsatisfiable, then that clause is a local minimum and can be skipped. This is a more common heuristical MaxSAT solver. This gives us a better local maximum, but not a global one.

  This is more or less what the Smart package manager did, except that in Smart, all packages were optional, and the entire solution was scored. It calculated a basic solution without optimization and then toggled each variable and saw if the score improved.

• Implement an actual search for a global maximum:

  This involves reading the literature. There are various versions of this, for example:

  1. Find unsatisfiable cores and use those to guide relaxation of clauses.

  2. A bounds-based search, where we translate sum(satisifed clauses) > k into SAT, and then search in one of the following ways:

     1. from 0 upward
     2. from n downward
     3. perform a binary search on [0, k] satisfied clauses.

     Actually we do not even need to calculate sum constraints into CNF, because we can just add a specialized new type of constraint to our code.

You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just so happens we don’t actually have any pure literals in there). We can control it in a bunch of ways:

1. We can mark packages as “install” or “reject”
2. We can order actions/clauses. When backtracking the action that came later will be the first we try to backtrack on
3. We can order the choices of a dependency - we try them left to right.

This is about all that we really want to do, we can’t go if we reach a conflict, say “oh but this conflict was introduced by that upgrade, and it seems more important, so let’s not backtrack on the upgrade request but on this dependency instead.”.

This forces us to think about lowering the dependency problem into this form, such that not only do we get formally correct solutions, but also semantically correct ones. This is nice because we can apply a systematic way to approach the issue rather than introducing ad-hoc rules in the old solver which had a “which of these packages should I flip the opposite way to break the conflict” kind of thinking.

Now our test suite has a whole bunch of these semantics encoded in it, and I’m going to share some problems and ideas for how to solve them. I can’t wait to fix these and the error reporting and then turn it on in Ubuntu and later Debian (the defaults change is a post-trixie change, let’s be honest).

apt upgrade is hard

The apt upgrade commands implements a safe version of dist-upgrade that essentially calculates the dist-upgrade, and then undoes anything that would cause a package to be removed, but it (unlike its apt-get counterpart) allows the solver to install new packages.

Now, consider the following package is installed:

X Depends: A (= 1) | B

An upgrade from A=1 to A=2 is available. What should happen?

The classic solver would choose to remove X in a dist-upgrade, and then upgrade A, so it’s answer is quite clear: Keep back the upgrade of A.

The new solver however sees two possible solutions:

1. Install B to satisfy X Depends A (= 1) | B.
2. Keep back the upgrade of A

Which one does it pick? This depends on the order in which it sees the upgrade action for A and the dependency, as it will backjump chronologically. So

1. If it gets to the dependency first, it marks A=1 for install to satisfy A (= 1). Then it gets to the upgrade request, which is just A Depends A (= 2) | A (= 1) and sees it is satisfied already and is content.

2. If it gets to the upgrade request first, it marks A=2 for install to satisfy A (= 2). Then later it gets to X Depends: A (= 1) | B, sees that A (= 1) is not satisfiable, and picks B.

We have two ways to approach this issue:

1. We always order upgrade requests last, so they will be kept back in case of conflicting dependencies
2. We require that, for apt upgrade a currently satisfied dependency must be satisfied by currently installed packages, hence eliminating B as a choice.

Recommends are hard too

See if you have a X Recommends: A (= 1) and a new version of A, A (= 2), the solver currently will silently break the Recommends in some cases.

But let’s explore what the behavior of a X Recommends: A (= 1) in combination with an available upgrade of A (= 2) should be. We could say the rule should be:

• An upgrade should keep back A instead of breaking the Recommends
• A dist-upgrade should either keep back A or remove X (if it is obsolete)

This essentially leaves us the same choices as for the previous problem, but with an interesting twist. We can change the ordering (and we already did), but we could also introduce a new rule, “promotions”:

A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently satisfied, must continue to be satisfied, that is, it effectively is promoted to a Depends.

This neatly solves the problem for us. We will never break Recommends that are satisfied.

Likewise, we already have a Recommends demotion rule:

A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently unsatisfied, will not be further evaluated (it is treated like a Suggests is in the default configuration).

Whether we should be allowed to break Suggests with our decisions or not (the old autoremover did not, for instance) is a different decision. Should we promote currently satisified Suggests to Depends as well? Should we follow currently satisified Suggests so the solver sees them and doesn’t autoremove them, but treat them as optional?

tightening of versioned dependencies

Another case of versioned dependencies with alternatives that has complex behavior is something like

X Depends: A (>= 2) | B
X Recommends: A (>= 2) | B

In both cases, installing X should upgrade an A < 2 in favour of installing B. But a naive SAT solver might not. If your request to keep A installed is encoded as A (= 1) | A (= 2), then it first picks A (= 1). When it sees the Depends/Recommends it will switch to B.

We can solve this again as in the previous example by ordering the “keep A installed” requests after any dependencies. Notably, we will enqueue the common dependencies of all A versions first before selecting a version of A, so something may select a version for us.

version narrowing instead of version choosing

A different approach to dealing with the issue of version selection is to not select a version until the very last moment. So instead of selecting a version to satisfy A (>= 2) we instead translate

Depends: A (>= 2)

into two rules:

1. The package selection rule:

    Depends: A
   

   This ensures that any version of A is installed (i.e. it adds a version choice clause, A (= 1) | A (= 2) in an example with two versions for A.

2. The version narrowing rule:

    Conflicts: A (<< 2)
   

   This outright would reject a choice of A (= 1).

So now we have 3 kinds of clauses:

1. package selection
2. version narrowing
3. version selection

If we process them in that order, we should surely be able to find the solution that best matches the semantics of our Debian dependency model, i.e. selecting earlier choices in a dependency before later choices in the face of version restrictions.

This still leaves one issue: What if our maintainer did not use Depends: A (>= 2) | B but e.g. Depends: A (= 3) | B | A (= 2). He’d expect us to fall back to B if A (= 3) is not installable, and not to B. But we’d like to enqueue A and reject all choices other than 3 and 2. I think it’s fair to say: “Don’t do that, then” here.

Implementing strict pinning correctly

APT knows a single candidate version per package, this makes the solver relatively deterministic: It will only ever pick the candidate, or an installed version. This also happens to significantly reduce the search space which is good - less backtracking. An uptodate system will only ever have one version per package that can be installed, so we never actually have to choose versions.

But of course, APT allows you to specify a non-candidate version of a package to install, for example:

apt install foo/oracular-proposed

The way this works is that the core component of the previous solver, which is the pkgDepCache maintains what essentially amounts to an overlay of the policy that you could see with apt-cache policy.

The solver currently however validates allowed version choices against the policy directly, and hence finds these versions are not allowed and craps out. This is an interesting problem because the solver should not be dependent on the pkgDepCache as the pkgDepCache initialization (Building dependency tree...) accounts for about half of the runtime of APT (until the Y/n prompt) and I’d really like to get rid of it.

But currently the frontend does go via the pkgDepCache. It marks the packages in there, building up what you could call a transaction, and then we translate it to the new solver, and once it is done, it translates the result back into the pkgDepCache.

The current implementation of “allowed version” is implemented by reducing the search space, i.e. every dependency, we outright ignore any non-allowed versions. So if you have a version 3 of A that is ignored a Depends: A would be translated into A (= 2) | A (= 1).

However this has two disadvantages. (1) It means if we show you why A could not be installed, you don’t even see A (= 3) in the list of choices and (2) you would need to keep the pkgDepCache around for the temporary overrides.

So instead of actually enforcing the allowed version rule by filtering, a more reasonable model is that we apply the allowed version rule by just marking every other version as not allowed when discovering the package in the from depcache translation layer. This doesn’t really increase the search space either but it solves both our problem of making overrides work and giving you a reasonable error message that lists all versions of A.

pulling up common dependencies to minimize backtracking cost

One of the common issues we have is that when we have a dependency group

`A | B | C | D`

we try them in order, and if one fails, we undo everything it did, and move on to the next one. However, this isn’t perhaps the best choice of operation.

I explained before that one thing we do is queue the common dependencies of a package (i.e. dependencies shared in all versions) when marking a package for install, but we don’t do this here: We have already lowered the representation of the dependency group into a list of versions, so we’d need to extract the package back out of it.

This can of course be done, but there may be a more interesting solution to the problem, in that we simply enqueue all the common dependencies. That is, we add n backtracking levels for n possible solutions:

1. We enqueue the common dependencies of all possible solutions deps(A)&deps(B)&deps(C)&deps(D)
2. We decide (adding a decision level) not to install D right now and enqueue deps(A)&deps(B)&deps(C)
3. We decide (adding a decision level) not to install C right now and enqueue deps(A)&deps(B)
4. We decide (adding a decision level) not to install B right now and enqueue A

Now if we need to backtrack from our choice of A we hopefully still have a lot of common dependencies queued that we do not need to redo. While we have more backtracking levels, each backtracking level would be significantly cheaper, especially if you have cheap backtracking (which admittedly we do not have, yet anyway).

The caveat though is: It may be pretty expensive to find the common dependencies. We need to iterate over all dependency groups of A and see if they are in B, C, and D, so we have a complexity of roughly

#A * (#B+#C+#D)

Each dependency group we need to check i.e. is X|Y in B meanwhile has linear cost: We need to compare the memory content of two pointer arrays containing the list of possible versions that solve the dependency group. This means that X|Y and Y|X are different dependencies of course, but that is to be expected – they are. But any dependency of the same order will have the same memory layout.

So really the cost is roughly N^4. This isn’t nice.

You can apply various heuristics here on how to improve that, or you can even apply binary logic:

1. Enqueue common dependencies of A|B|C|D
2. Move into the left half, enqueue of A|B
3. Again divide and conquer and select A.

This has a significant advantage in long lists of choices, and also in the common case, where the first solution should be the right one.

Or again, if you enqueue the package and a version restriction instead, you already get the common dependencies enqueued for the chosen package at least.

How does it work?

Solver3 is a fully backtracking dependency solving algorithm that defers choices to as late as possible. It starts with an empty set of packages, then adds the manually installed packages, and then installs packages automatically as necessary to satisfy the dependencies.

Deferring the choices is implemented multiple ways:

First, all install requests recursively mark dependencies with a single solution for install, and any packages that are being rejected due to conflicts or user requests will cause their reverse dependencies to be transitively marked as rejected, provided their or group cannot be solved by a different package.

Second, any dependency with more than one choice is pushed to a priority queue that is ordered by the number of possible solutions, such that we resolve a|b before a|b|c.

Not just by the number of solutions, though. One important point to note is that optional dependencies, that is, Recommends, are always sorting after mandatory dependencies. Do note on that: Recommended packages do not “nest” in backtracking - dependencies of a Recommended package themselves are not optional, so they will have to be resolved before the next Recommended package is seen in the queue.

Another important step in deferring choices is extracting the common dependencies of a package across its version and then installing them before we even decide which of its versions we want to install - one of the dependencies might cycle back to a specific version after all.

Decisions about package levels are recorded at a certain decision level, if we reach a conflict we backtrack to the previous decision level, mark the decision we made (install X) in the inverse (DO NOT INSTALL X), reset all the state all decisions made at the higher level, and restore any dependencies that are no longer resolved to the work queue.

Comparison to SAT solver design.

If you have studied SAT solver design, you’ll find that essentially this is a DPLL solver without pure literal elimination. A pure literal eliminitation phase would not work for a package manager: First negative pure literals (packages that everything conflicts with) do not exist, and positive pure literals (packages nothing conflicts with) we do not want to mark for install - we want to install as little as possible (well subject, to policy).

As part of the solving phase, we also construct an implication graph, albeit a partial one: The first package installing another package is marked as the reason (A -> B), the same thing for conflicts (not A -> not B).

Once we have added the ability to have multiple parents in the implication graph, it stands to reason that we can also implement the much more advanced method of conflict-driven clause learning; where we do not jump back to the previous decision level but exactly to the decision level that caused the conflict. This would massively speed up backtracking.

What changes can you expect in behavior?

The most striking difference to the classic APT solver is that solver3 always keeps manually installed packages around, it never offers to remove them. We will relax that in a future iteration so that it can replace packages with new ones, that is, if your package is no longer available in the repository (obsolete), but there is one that Conflicts+Replaces+Provides it, solver3 will be allowed to install that and remove the other.

Implementing that policy is rather trivial: We just need to queue obsolete | replacement as a dependency to solve, rather than mark the obsolete package for install.

Another critical difference is the change in the autoremove behavior: The new solver currently only knows the strongest dependency chain to each package, and hence it will not keep around any packages that are only reachable via weaker chains. A common example is when gcc-<version> packages accumulate on your system over the years. They all have Provides: c-compiler and the libtool Depends: gcc | c-compiler is enough to keep them around.

New features

The new option --no-strict-pinning instructs the solver to consider all versions of a package and not just the candidate version. For example, you could use apt install foo=2.0 --no-strict-pinning to install version 2.0 of foo and upgrade - or downgrade - packages as needed to satisfy foo=2.0 dependencies. This mostly comes in handy in use cases involving Debian experimental or the Ubuntu proposed pockets, where you want to install a package from there, but try to satisfy from the normal release as much as possible.

The implication graph building allows us to implement an apt why command, that while not as nicely detailed as aptitude, at least tells you the exact reason why a package is installed. It will only show the strongest dependency chain at first of course, since that is what we record.

What is left to do?

At the moment, error information is not stored across backtracking in any way, but we generally will want to show you the first conflict we reach as it is the most natural one; or all conflicts. Currently you get the last conflict which may not be particularly useful.

Likewise, errors currently are just rendered as implication graphs of the form [not] A -> [not] B -> ..., and we need to put in some work to present those nicely.

The test suite is not passing yet, I haven’t really started working on it. A challenge is that most packages in the test suite are manually installed as they are mocked, and the solver now doesn’t remove those.

We plan to implement the replacement logic such that foo can be replaced by foo2 Conflicts/Replaces/Provides foo without needing to be automatically installed.

Improving the backtracking to be non-chronological conflict-driven clause learning would vastly enhance our backtracking performance. Not that it seems to be an issue right now in my limited testing (mostly noble 64-bit-time_t upgrades). A lot of that complexity you have normally is not there because the manually installed packages and resulting unit propagation (single-solution Depends/Reverse-Depends for Conflicts) already ground us fairly far in what changes we can actually make.

Once all the stuff has landed, we need to start rolling it out and gather feedback. On Ubuntu I’d like automated feedback on regressions (running solver3 in parallel, checking if result is worse and then submitting an error to the error tracker), on Debian this could just be a role email address to send solver dumps to.

At the same time, we can also incrementally start rolling this out. Like phased updates in Ubuntu, we can also roll out the new solver as the default to 10%, 20%, 50% of users before going to the full 100%. This will allow us to capture regressions early and fix them.

• upgrade without new packages (apt-get upgrade)
• upgrade with new packages (apt upgrade)
• upgrade with new packages and deletions (apt{,-get} {dist,full}-upgrade)

All of these upgrade types are necessary to deal with upgrades within a distribution release. Yes, sometimes even removals may be needed because bug fixes require adding a Conflicts somewhere.

In Ubuntu we have a third type of upgrades, handled by a separate tool: release upgrades. ubuntu-release-upgrader changes your sources.list, and applies various quirks to the upgrade.

In this post, I want to look not at the quirk aspects but discuss how dependency solving should differ between intra-release and inter-release upgrades.

Previous solver projects (such as Mancoosi) operated under the assumption that minimizing the number of changes performed should ultimately be the main goal of a solver. This makes sense as every change causes risks. However it ignores a different risk, which especially applies when upgrading from one distribution release to a newer one: Increasing divergence from the norm.

Consider a person installs foo in Debian 12. foo depends on a | b, so a will be automatically installed to satisfy the dependency. A release later, a has some known issues and b is prefered, the dependency now reads: b|a.

A classic solver would continue to keep a installed because it was installed before, leading upgraded installs to have foo, a installed whereas new systems have foo, b installed. As systems get upgraded over and over, they continue to diverge further and further from new installs to the point that it adds substantial support effort.

My proposal for the new APT solver is that when we perform release upgrades, we forget which packages where previously automatically installed. We effectively perform a normalization: All systems with the same set of manually installed packages will end up with the same set of automatically installed packages. Consider the solving starting with an empty set and then installing the latest version of each previously manually installed package: It will see now that foo depends b|a and install b (and a will be removed later on as its not part of the solution).

Another case of divergence is Suggests handling. Consider that foo also Suggests s. You now install another package bar that depends s, hence s gets installed. Upon removing bar, s is not being removed automatically because foo still suggests it (and you may have grown used to foo’s integration of s). This is because apt considers Suggests to be important - they won’t be automatically installed, but will not be automatically removed.

In Ubuntu, we unset that policy on release upgrades to normalize the systems. The reasoning for that is simple: While you may have grown to use s as part of foo during the release, an upgrade to the next release already is big enough that removing s is going to have less of an impact - breakage of workflows is expected between release upgrades.

I believe that apt release-upgrade will benefit from both of these design choices, and in the end it boils down to a simple mantra:

• On upgrades within a release, minimize changes.
• On upgrades between releases, minimize divergence from fresh installs.

taking a step back: how does secure boot on Ubuntu work?

Booting on Ubuntu involves three components after the firmware:

1. shim
2. grub
3. linux

Each of these is a PE binary signed with a key. The shim is signed by Microsoft’s 3rd party key and embeds a self-signed Canonical CA certificate, and optionally a vendor dbx (a list of revoked certificates or binaries). grub and linux (and fwupd) are then signed by a certificate issued by that CA

In Ubuntu’s case, the CA certificate is sharded: Multiple people each have a part of the key and they need to meet to be able to combine it and sign things, such as new code signing certificates.

BootHole

When BootHole happened in 2020, travel was suspended and we hence could not rotate to a new signing certificate. So when it came to updating our shim for the CVEs, we had to revoke all previously signed kernels, grubs, shims, fwupds by their hashes.

This generated a very large vendor dbx which caused lots of issues as shim exported them to a UEFI variable, and not everyone had enough space for such large variables. Sigh.

We decided we want to rotate our signing key next time.

This was also when upstream added SBAT metadata to shim and grub. This gives a simple versioning scheme for security updates and easy revocation using a simple EFI variable that shim writes to and reads from.

Spring 2022 CVEs

We still were not ready for travel in 2021, but during BootHole we developed the SBAT mechanism, so one could revoke a grub or shim by setting a single EFI variable.

We actually missed rotating the shim this cycle as a new vulnerability was reported immediately after it, and we decided to hold on to it.

2022 key rotation and the fall CVEs

This caused some problems when the 2nd CVE round came, as we did not have a shim with the latest SBAT level, and neither did a lot of others, so we ended up deciding upstream to not bump the shim SBAT requirements just yet. Sigh.

Anyway, in October we were meeting again for the first time at a Canonical sprint, and the shardholders got together and created three new signing keys: 2022v1, 2022v2, and 2022v3. It took us until January before they were installed into the signing service and PPAs setup to sign with them.

We also submitted a shim 15.7 with the old keys revoked which came back at around the same time.

Now we were in a hurry. The 22.04.2 point release was scheduled for around middle of February, and we had nothing signed with the new keys yet, but our new shim which we need for the point release (so the point release media remains bootable after the next round of CVEs), required new keys.

So how do we ensure that users have kernels, grubs, and fwupd signed with the new key before we install the new shim?

upgrade ordering

grub and fwupd are simple cases: For grub, we depend on the new version. We decided to backport grub 2.06 to all releases (which moved focal and bionic up from 2.04), and kept the versioning of the -signed packages the same across all releases, so we were able to simply bump the Depends for grub to specify the new minimum version. For fwupd-efi, we added Breaks.

(Actually, we also had a backport of the CVEs for 2.04 based grub, and we did publish that for 20.04 signed with the old keys before backporting 2.06 to it.)

Kernels are a different story: There are about 60 kernels out there. My initial idea was that we could just add Breaks for all of them. So our meta package linux-image-generic which depends on linux-image-$(uname -r)-generic, we’d simply add Breaks: linux-image-generic (« 5.19.0-31) and then adjust those breaks for each series. This would have been super annoying, but ultimately I figured this would be the safest option. This however caused concern, because it could be that apt decides to remove the kernel metapackage.

I explored checking the kernels at runtime and aborting if we don’t have a trusted kernel in preinst. This ensures that if you try to upgrade shim without having a kernel, it would fail to install. But this ultimately has a couple of issues:

1. It aborts the entire transaction at that point, so users will be unable to run apt upgrade until they have a recent kernel.
2. We cannot even guarantee that a kernel would be unpacked first. So even if you got a new kernel, apt/dpkg might attempt to unpack it first and then the preinst would fail because no kernel is present yet.

Ultimately we believed the danger to be too large given that no kernels had yet been released to users. If we had kernels pushed out for 1-2 months already, this would have been a viable choice.

So in the end, I ended up modifying the shim packaging to install both the latest shim and the previous one, and an update-alternatives alternative to select between the two:

In it’s post-installation maintainer script, shim-signed checks whether all kernels with a version greater or equal to the running one are not revoked, and if so, it will setup the latest alternative with priority 100 and the previous with a priority of 50. If one or more of those kernels was signed with a revoked key, it will swap the priorities around, so that the previous version is preferred.

Now this is fairly static, and we do want you to switch to the latest shim eventually, so I also added hooks to the kernel install to trigger the shim-signed postinst script when a new kernel is being installed. It will then update the alternatives based on the current set of kernels, and if it now points to the latest shim, reinstall shim and grub to the ESP.

Ultimately this means that once you install your 2nd non-revoked kernel, or you install a non-revoked kernel and then reconfigure shim or the kernel, you will get the latest shim. When you install your first non-revoked kernel, your currently booted kernel is still revoked, so it’s not upgraded immediately. This has a benefit in that you will most likely have two kernels you can boot without disabling secure boot.

regressions

Of course, the first version I uploaded had still some remaining hardcoded “shimx64” in the scripts and so failed to install on arm64 where “shimaa64” is used. And if that were not enough, I also forgot to include support for gzip compressed kernels there. Sigh, I need better testing infrastructure to be able to easily run arm64 tests as well (I only tested the actual booting there, not the scripts).

shim-signed migrated to the release pocket in lunar fairly quickly, but this caused images to stop working, because the new shim was installed into images, but no kernel was available yet, so we had to demote it to proposed and block migration. Despite all the work done for end users, we need to be careful to roll this out for image building.

another grub update for OOM issues.

We had two grubs to release: First there was the security update for the recent set of CVEs, then there also was an OOM issue for large initrds which was blocking critical OEM work.

We fixed the OOM issue by cherry-picking all 2.12 memory management patches, as well as the red hat patches to the loader we take from there. This ended up a fairly large patch set and I was hesitant to tie the security update to that, so I ended up pushing the security update everywhere first, and then pushed the OOM fixes this week.

With the OOM patches, you should be able to boot initrds of between 400M and 1GB, it also depends on the memory layout of your machine and your screen resolution and background images. So OEM team had success testing 400MB irl, and I tested up to I think it was 1.2GB in qemu, I ran out of FAT space then and stopped going higher :D

other features in this round

• Intel TDX support in grub and shim
• Kernels are allocated as CODE now not DATA as per the upstream mm changes, might fix boot on X13s

am I using this yet?

The new signing keys are used in:

• shim-signed 1.54 on 22.10+, 1.51.3 on 22.04, 1.40.9 on 20.04, 1.37~18.04.13 on 18.04
• grub2-signed 1.187.2~ or newer (binary packages grub-efi-amd64-signed or grub-efi-arm64-signed), 1.192 on 23.04.
• fwupd-signed 1.51~ or newer
• various linux updates. Check apt changelog linux-image-unsigned-$(uname -r) to see if Revoke & rotate to new signing key (LP: #2002812) is mentioned in there to see if it signed with the new key.

If you were able to install shim-signed, your grub and fwupd-efi will have the correct version as that is ensured by packaging. However your shim may still point to the old one. To check which shim will be used by grub-install, you can check the status of the shimx64.efi.signed or (on arm64) shimaa64.efi.signed alternative. The best link needs to point to the file ending in latest:

$ update-alternatives --display shimx64.efi.signed
shimx64.efi.signed - auto mode
  link best version is /usr/lib/shim/shimx64.efi.signed.latest
  link currently points to /usr/lib/shim/shimx64.efi.signed.latest
  link shimx64.efi.signed is /usr/lib/shim/shimx64.efi.signed
/usr/lib/shim/shimx64.efi.signed.latest - priority 100
/usr/lib/shim/shimx64.efi.signed.previous - priority 50

If it does not, but you have installed a new kernel compatible with the new shim, you can switch immediately to the new shim after rebooting into the kernel by running dpkg-reconfigure shim-signed. You’ll see in the output if the shim was updated, or you can check the output of update-alternatives as you did above after the reconfiguration has finished.

For the out of memory issues in grub, you need grub2-signed 1.187.3~ (same binaries as above).

how do I test this (while it’s in proposed)?

1. upgrade your kernel to proposed and reboot into that
2. upgrade your grub-efi-amd64-signed, shim-signed, fwupd-signed to proposed.

If you already upgraded your shim before your kernel, don’t worry:

1. upgrade your kernel and reboot
2. run dpkg-reconfigure shim-signed

And you’ll be all good to go.

deep dive: uploading signed boot assets to Ubuntu

For each signed boot asset, we build one version in the latest stable release and the development release. We then binary copy the built binaries from the latest stable release to older stable releases. This process ensures two things: We know the next stable release is able to build the assets and we also minimize the number of signed assets.

OK, I lied. For shim, we actually do not build in the development release but copy the binaries upward from the latest stable, as each shim needs to go through external signing.

The entire workflow looks something like this:

1. Upload the unsigned package to one of the following “build” PPAs:

   • https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa for non-embargoed updates
   • https://launchpad.net/~ubuntu-security-embargoed-shared/+archive/ubuntu/grub2 for embargoed updates

2. Upload the signed package to the same PPA

3. For stable release uploads:

   • Copy the unsigned package back across all stable releases in the PPA
   • Upload the signed package for stable releases to the same PPA with ~<release>.1 appended to the version

4. Submit a request to canonical-signing-jobs to sign the uploads.

   The signing job helper copies the binary -unsigned packages to the primary-2022v1 PPA where they are signed, creating a signing tarball, then it copies the source package for the -signed package to the same PPA which then downloads the signing tarball during build and places the signed assets into the -signed deb.

   Resulting binaries will be placed into the proposed PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed

5. Review the binaries themselves

6. Unembargo and binary copy the binaries from the proposed PPA to the proposed-public PPA: https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed-public.

   This step is not strictly necessary, but it enables tools like sru-review to work, as they cannot access the packages from the normal private “proposed” PPA.

7. Binary copy from proposed-public to the proposed queue(s) in the primary archive

Lots of steps!

WIP

As of writing, only the grub updates have been released, other updates are still being verified in proposed. An update for fwupd in bionic will be issued at a later point, removing the EFI bits from the fwupd 1.2 packaging and using the separate fwupd-efi project instead like later release series.

Z3 does not need normalized formulas, but offers higher level abstractions like atmost and atleast and implies, that we will make use of together with boolean variables to translate the dependency problem to a form Z3 understands.

In this post, we’ll see how we can apply Z3 to the dependency resolution in APT. We’ll only discuss the basics here, a future post will explore optimization criteria and recommends.

Translating the universe

APT’s package universe consists of 3 relevant things: packages (the tuple of name and architecture), versions (basically a .deb), and dependencies between versions.

While we could translate our entire universe to Z3 problems, we instead will construct a root set from packages that were manually installed and versions marked for installation, and then build the transitive root set from it by translating all versions reachable from the root set.

For each package P in the transitive root set, we create a boolean literal P. We then translate each version P1, P2, and so on. Translating a version means building a boolean literal for it, e.g. P1, and then translating the dependencies as shown below.

We now need to create two more clauses to satisfy the basic requirements for debs:

1. If a version is installed, the package is installed; and vice versa. We can encode this requirement for P above as P == atleast({P1,P2}, 1).
2. There can only be one version installed. We add an additional constraint of the form atmost({P1,P2}, 1).

We also encode the requirements of the operation.

1. For each package P that is manually installed, add a constraint P.
2. For each version V that is marked for install, add a constraint V.
3. For each package P that is marked for removal, add a constraint !P.

Dependencies

Packages in APT have dependencies of two basic forms: Depends and Conflicts, as well as variations like Breaks (identical to Conflicts in solving terms), and Recommends (soft Depends) - we’ll ignore those for now. We’ll discuss Conflicts in the next section.

Let’s take a basic dependency list: A Depends: X|Y, Z. To represent that dependency, we expand each name to a list of versions that can satisfy the dependency, for example X1|X2|Y1, Z1.

Translating this dependency list to our Z3 solver, we create boolean variables X1,X2,Y1,Z1 and define two rules:

1. A implies atleast({X1,X2,Y1}, 1)
2. A implies atleast({Z1}, 1)

If there actually was nothing that satisfied the Z requirement, we’d have added a rule not A. It would be possible to simply not tell Z3 about the version at all as an optimization, but that adds more complexity, and the not A constraint should not cause too many problems.

Conflicts

Conflicts cannot have or in them. A dependency B Conflicts: X, Y means that only one of B, X, and Y can be installed. We can directly encode this in Z3 by using the constraint atmost({B,X,Y}, 1). This is an optimized encoding of the constraint: We could have encoded each conflict in the form !B or !X, !B or !X, and so on. Usually this leads to worse performance as it introduces additional clauses.

Complete example

Let’s assume we start with an empty install and want to install the package a below.

Package: a
Version: 1
Depends: c | b

Package: b
Version: 1

Package: b
Version: 2
Conflicts: x

Package: d
Version: 1

Package: x
Version: 1

The translation in Z3 rules looks like this:

1. Package rules for a:
   1. a == atleast({a1}, 1) - package is installed iff one version is
   2. atmost({a1}, 1) - only one version may be installed
   3. a – a must be installed
2. Dependency rules for a
   1. implies(a1, atleast({b2, b1}, 1)) – the translated dependency above. note that c is gone, it’s not reachable.
3. Package rules for b:
   1. b == atleast({b1,b2}, 1) - package is installed iff one version is
   2. atmost({b1, b2}, 1) - only one version may be installed
4. Dependencies for b (= 2):
   1. atmost({b2, x1}, 1) - the conflicts between x and b = 2 above
5. Package rules for x:
   1. x == atleast({x1}, 1) - package is installed iff one version is
   2. atmost({x1}, 1) - only one version may be installed

The package d is not translated, as it is not reachable from the root set {a1}, the transitive root set is {a1,b1,b2,x1}.

Next iteration: Optimization

We have now constructed the basic set of rules that allows us to solve solve our dependency problems (equivalent to SAT), however it might lead to suboptimal solutions where it removes automatically installed packages, or installs more packages than necessary, to name a few examples.

In our next iteration, we have to look at introducing optimization; for example, have the minimum number of removals, the minimal number of changed packages, or satisfy as many recommends as possible. We will also look at the upgrade problem (upgrade as many packages as possible), the autoremove problem (remove as many automatically installed packages as possible).

The manual page already provides all you need to know for replacing apt-key add usage:

Note: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either “gpg” or “asc” as file extension

So it’s kind of surprising people need step by step instructions for how to copy/download a file into a directory.

I’ll also discuss the alternative security snakeoil approach with signed-by that’s become popular. Maybe we should not have added signed-by, people seem to forget that debs still run maintainer scripts as root.

Aside from this email, Debian users should look into extrepo, which manages curated external repositories for you.

Direct translation

Assume you currently have:

wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add –

To translate this directly for bionic and newer, you can use:

sudo wget -qO /etc/apt/trusted.gpg.d/myrepo.asc https://myrepo.example/myrepo.asc

or to avoid downloading as root:

wget -qO-  https://myrepo.example/myrepo.asc | sudo tee -a /etc/apt/trusted.gpg.d/myrepo.asc

Older (and all) releases only support unarmored files with an extension .gpg. If you care about them, provide one, and use

sudo wget -qO /etc/apt/trusted.gpg.d/myrepo.gpg https://myrepo.example/myrepo.gpg

Some people will tell you to download the .asc and pipe it to gpg --dearmor, but gpg might not be installed, so really, just offer a .gpg one instead that is supported on all systems.

wget might not be available everywhere so you can use apt-helper:

sudo /usr/lib/apt/apt-helper download-file https://myrepo.example/myrepo.asc /etc/apt/trusted.gpg.d/myrepo.asc

or, to avoid downloading as root:

/usr/lib/apt/apt-helper download-file https://myrepo.example/myrepo.asc /tmp/myrepo.asc && sudo mv /tmp/myrepo.asc /etc/apt/trusted.gpg.d

Pretending to be safer by using signed-by

People say it’s good practice to not use trusted.gpg.d and install the file elsewhere and then refer to it from the sources.list entry by using signed-by=<path to the file>. So this looks a lot safer, because now your key can’t sign other unrelated repositories. In practice, security increase is minimal, since package maintainer scripts run as root anyway. But I guess it’s better for publicity :)

As an example, here are the instructions to install signal-desktop from signal.org. As mentioned, gpg --dearmor use in there is not a good idea, and I’d personally not tell people to modify /usr as it’s supposed to be managed by the package manager, but we don’t have an /etc/apt/keyrings or similar at the moment; it’s fine though if the keyring is installed by the package. You can also just add the file there as a starting point, and then install a keyring package overriding it (pretend there is a signal-desktop-keyring package below that would override the .gpg we added).

# NOTE: These instructions only work for 64 bit Debian-based
# Linux distributions such as Ubuntu, Mint etc.

# 1. Install our official public software signing key
wget -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
cat signal-desktop-keyring.gpg | sudo tee -a /usr/share/keyrings/signal-desktop-keyring.gpg > /dev/null

# 2. Add our repository to your list of repositories
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main' |\
  sudo tee -a /etc/apt/sources.list.d/signal-xenial.list

# 3. Update your package database and install signal
sudo apt update && sudo apt install signal-desktop

I do wonder why they do wget | gpg --dearmor, pipe that into the file and then cat | sudo tee it, instead of having that all in one pipeline. Maybe they want nicer progress reporting.

Scenario-specific guidance

We have three scenarios:

For system image building, shipping the key in /etc/apt/trusted.gpg.d seems reasonable to me; you are the vendor sort of, so it can be globally trusted.

Chrome-style debs and repository config debs: If you ship a deb, embedding the sources.list.d snippet (calling it $myrepo.list) and shipping a $myrepo.gpg in /usr/share/keyrings is the best approach. Whether you ship that in product debs aka vscode/chromium or provide a repository configuration deb (let’s call it myrepo-repo.deb) and then tell people to run apt update followed by apt install <package inside the repo> depends on how many packages are in the repo, I guess.

Manual instructions (signal style): The third case, where you tell people to run wget themselves, I find tricky. As we see in signal, just stuffing keyring files into /usr/share/keyrings is popular, despite /usr supposed to be managed by the package manager. We don’t have another dir inside /etc (or /usr/local), so it’s hard to suggest something else. There’s no significant benefit from actually using signed-by, so it’s kind of extra work for little gain, though.

Addendum: Future work

This part is new, just for this blog post. Let’s look at upcoming changes and how they make things easier.

Bundled .sources files

Assuming I get my merge request merged, the next version of APT (2.4/2.3.something) will do away with all the complexity and allow you to embed the key directly into a deb822 .sources file (which have been available for some time now):

Types: deb
URIs: https://myrepo.example/ https://myotherrepo.example/
Suites: stable not-so-stable
Components: main
Signed-By:
 -----BEGIN PGP PUBLIC KEY BLOCK-----
 .
 mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY
 CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk
 IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS
 dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG
 3bHcln8DMpIJVXht78sL
 =IE0r
 -----END PGP PUBLIC KEY BLOCK-----

Then you can just provide a .sources files to users, they place it into `sources.list.d, and everything magically works

Probably adding a nice apt add-source command for it I guess.

Well, python-apt’s aptsources package still does not support deb822 sources, and never will, we’ll need an aptsources2 for that for backwards-compatibility reasons, and then port software-properties and other users to it.

OpenPGP vs aptsign

We do have a better, tighter replacement for gpg in the works which uses Ed25519 keys to sign Release files. It’s temporarily named aptsign, but it’s a generic signer for single-section deb822 files, similar to signify/minisign.

• Reference implementation: https://salsa.debian.org/apt-team/python-aptsign
• Specification: https://wiki.debian.org/Teams/Apt/Spec/AptSign

We believe that this solves the security nightmare that our OpenPGP integration is while reducing complexity at the same time. Keys are much shorter, so the bundled sources file above will look much nicer.

Let’s have a look at what changed compared to 2.2. Many of you who run Debian testing or unstable, or Ubuntu groovy or hirsute will already have seen most of those changes.

New features

• Various patterns related to dependencies, such as ?depends are now available (2.1.16)
• The Protected field is now supported. It replaces the previous Important field and is like Essential, but only for installed packages (some minor more differences maybe in terms of ordering the installs).
• The update command has gained an --error-on=any option that makes it error out on any failure, not just what it considers persistent ons.
• The rred method can now be used as a standalone program to merge pdiff files
• APT now implements phased updates. Phasing is used in Ubuntu to slow down and control the roll out of updates in the -updates pocket, but has previously only been available to desktop users using update-manager.

Other behavioral changes

• The kernel autoremoval helper code has been rewritten from shell in C++ and now runs at run-time, rather than at kernel install time, in order to correctly protect the kernel that is running now, rather than the kernel that was running when we were installing the newest one.

  It also now protects only up to 3 kernels, instead of up to 4, as was originally intended, and was the case before 1.1 series. This avoids /boot partitions from running out of space, especially on Ubuntu which has boot partitions sized for the original spec.

Performance improvements

• The cache is now hashed using XXH3 instead of Adler32 (or CRC32c on SSE4.2 platforms)
• The hash table size has been increased

Bug fixes

• * wildcards work normally again (since 2.1.0)
• The cache file now includes all translation files in /var/lib/apt/lists, so multi-user systems with different locales correctly show translated descriptions now.
• URLs are no longer dequoted on redirects only to be requoted again, fixing some redirects where servers did not expect different quoting.
• Immediate configuration is now best-effort, and failure is no longer fatal.
• various changes to solver marking leading to different/better results in some cases (since 2.1.0)
• The lower level I/O bits of the HTTP method have been rewritten to hopefully improve stability
• The HTTP method no longer infinitely retries downloads on some connection errors
• The pkgnames command no longer accidentally includes source packages
• Various fixes from fuzzing efforts by David

Security fixes

• Out-of-bound reads in ar and tar implementations (CVE-2020-3810, 2.1.2)
• Integer overflows in ar and tar (CVE-2020-27350, 2.1.13)

(all of which have been backported to all stable series, back all the way to 1.0.9.8.* series in jessie eLTS)

Incompatibilities

• N/A - there were no breaking changes in apt 2.2 that we are aware of.

Deprecations

• apt-key(1) is scheduled to be removed for Q2/2022, and several new warnings have been added.

  apt-key was made obsolete in version 0.7.25.1, released in January 2010, by /etc/apt/trusted.gpg.d becoming a supported place to drop additional keyring files, and was since then only intended for deleting keys in the legacy trusted.gpg keyring.

  Please manage files in trusted.gpg.d yourself; or place them in a different location such as /etc/apt/keyrings (or make up your own, there’s no standard location) or /usr/share/keyrings, and use signed-by in the sources.list.d files.

  The legacy trusted.gpg keyring still works, but will also stop working eventually. Please make sure you have all your keys in trusted.gpg.d. Warnings might be added in the upcoming months when a signature could not be verified using just trusted.gpg.d.

  Future versions of APT might switch away from GPG.

• As a reminder, regular expressions and wildcards other than * inside package names are deprecated (since 2.0). They are not available anymore in apt(8), and will be removed for safety reasons in apt-get in a later release.

Why get a Pixel?

Camera: OnePlus focuses on stuffing as many sensors as it can into a phone, rather than a good main sensor, resulting in pictures that are mediocre blurry messes - the dreaded oil painting effect. Pixel have some of the best camera in the smartphone world. Sure, other hardware is far more capable, but the Pixels manage consistent results, so you need to take less pictures because they don’t come out blurry half the time, and the post processing is so good that the pictures you get are just great. Other phones can shoot better pictures, sure - on a tripod.

Security updates: Pixels provide 3 years of monthly updates, with security updates being published on the 5th of each month. OnePlus only provides updates every 2 months, and then the updates they do release are almost a month out of date, not counting that they are only 1st-of-month patches, meaning vendor blob updates included in the 5th-of-month updates are even a month older. Given that all my banking runs on the phone, I don’t want it to be constantly behind.

Feature updates: Of course, Pixels also get Beta Android releases and the newest Android release faster than any other phone, which is advantageous for Android development and being nerdy.

Size and weight: OnePlus phones keep getting bigger and bigger. By today’s standards, the OnePlus 6 at 6.18" and 177g is a small an lightweight device. Their latest phone, the Nord, has 6.44" and weighs 184g, the OnePlus 8 comes in at 180g with a 6.55" display. This is becoming unwieldy. Eschewing glass and aluminium for plastic, the Pixel 4a comes in at 144g.

First impressions

Accessories

The Pixel 4a comes in a small box with a charger, USB-C to USB-C cable, a USB-OTG adapter, sim tray ejector. No pre-installed screen protector or bumper are provided, as we’ve grown accustomed to from Chinese manufacturers like OnePlus or Xiaomi. The sim tray ejector has a circular end instead of the standard oval one - I assume so it looks like the ‘o’ in Google?

Google sells you fabric cases for 45€. That seems a bit excessive, although I like that a lot of it is recycled.

Haptics

Coming from a 6.18" phablet, the Pixel 4a with its 5.81" feels tiny. In fact, it’s so tiny my thumb and my index finger can touch while holding it. Cute! Bezels are a bit bigger, resulting in slightly less screen to body. The bottom chin is probably impracticably small, this was already a problem on the OnePlus 6, but this one is even smaller. Oh well, form over function.

The buttons on the side are very loud and clicky. As is the vibration motor. I wonder if this Pixel thinks it’s a Model M. It just feels great.

The plastic back feels really good, it’s that sort of high quality smooth plastic you used to see on those high-end Nokia devices.

The finger print reader, is super fast. Setup just takes a few seconds per finger, and it works reliably. Other phones (OnePlus 6, Mi A1/A2) take like half a minute or a minute to set up.

Software

The software - stock Android 11 - is fairly similar to OnePlus’ OxygenOS. It’s a clean experience, without a ton of added bloatware (even OnePlus now ships Facebook out of box, eww). It’s cleaner than OxygenOS in some way - there are no duplicate photos apps, for example. On the other hand, it also has quite a bunch of Google stuff I could not care less about like YT Music. To be fair, those are minor noise once all 130 apps were transferred from the old phone.

There are various things I miss coming from OnePlus such as off-screen gestures, network transfer rate indicator in quick settings, or a circular battery icon. But the Pixel has an always on display, which is kind of nice. Most of the cool Pixel features, like call screening or live transcriptions are unfortunately not available in Germany.

The display is set to display the same amount of content as my 6.18" OnePlus 6 did, so everything is a bit tinier. This usually takes me a week or two to adjust too, and then when I look at the OnePlus again I’ll be like “Oh the font is huge”, but right now, it feels a bit small on the Pixel.

You can configure three colour profiles for the Pixel 4a: Natural, Boosted, and Adaptive. I have mine set to adaptive. I’d love to see stock Android learn what OnePlus has here: the ability to adjust the colour temperature manually, as I prefer to keep my devices closer to 5500K than 6500K, as I feel it’s a bit easier on the eyes. Or well, just give me the ability to load a ICM profile (though, I’d need to calibrate the screen then - work!).

Migration experience

Restoring the apps from my old phone only restore settings for a few handful out of 130, which is disappointing. I had to spent an hour or two logging in to all the other apps, and I had to fiddle far too long with openScale to get it to take its data over. It’s a mystery to me why people do not allow their apps to be backed up, especially something innocent like a weight tracking app. One of my banking apps restored its logins, which I did not really like. KeePass2Android settings were restored as well, but at least the key file was not restored.

I did not opt in to restoring my device settings, as I feel that restoring device settings when changing manufactures is bound to mess up some things. For example, I remember people migrating to OnePlus phones and getting their old DND schedule without any way to change it, because OnePlus had hidden the DND stuff. I assume that’s the reason some accounts, like my work GSuite account were not migrated (it said it would migrate accounts during setup).

I’ve setup Bitwarden as my auto-fill service, so I could login into most of my apps and websites using the stored credentials. I found that often that did not work. Like Chrome does autofill fine once, but if I then want to autofill again, I have to kill and restart it, otherwise I don’t get the auto-fill menu. Other apps did not allow any auto-fill at all, and only gave me the option to copy and paste. Yikes - auto-fill on Android still needs a lot of work.

Performance

It hangs a bit sometimes, but this was likely due to me having set 2 million iterations on my Bitwarden KDF and using Bitwarden a lot, and then opening up all 130 apps to log into them which overwhelmed the phone a bit. Apart from that, it does not feel worse than the OnePlus 6 which was to be expected, given that the benchmarks only show a slight loss in performance.

Photos do take a few seconds to process after taking them, which is annoying, but understandable given how much Google relies on computation to provide decent pictures.

Audio

The Pixel has dual speakers, with the earpiece delivering a tiny sound and the bottom firing speaker doing most of the work. Still, it’s better than just having the bottom firing speaker, as it does provide a more immersive experience. Bass makes this thing vibrate a lot. It does not feel like a resonance sort of thing, but you can feel the bass in your hands. I’ve never had this before, and it will take some time getting used to.

Final thoughts

This is a boring phone. There’s no wow factor at all. It’s neither huge, nor does it have high-res 48 or 64 MP cameras, nor does it have a ton of sensors. But everything it does, it does well. It does not pretend to be a flagship like its competition, it doesn’t want to wow you, it just wants to be the perfect phone for you. The build is solid, the buttons make you think of a Model M, the camera is one of the best in any smartphone, and you of course get the latest updates before anyone else. It does not feel like a “only 350€” phone, but yet it is. 128GB storage is plenty, 1080p resolution is plenty, 12.2MP is … you guessed it, plenty.

The same applies to the other two Pixel phones - the 4a 5G and 5. Neither are particularly exciting phones, and I personally find it hard to justify spending 620€ on the Pixel 5 when the Pixel 4a does job for me, but the 4a 5G might appeal to users looking for larger phones. As to 5G, I wouldn’t get much use out of it, seeing as its not available anywhere I am. Because I’m on Vodafone. If you have a Telekom contract or live outside of Germany, you might just have good 5G coverage already and it might make sense to get a 5G phone rather than sticking to the budget choice.

Outlook

The big question for me is whether I’ll be able to adjust to the smaller display. I now have a tablet, so I’m less often using the phone (which my hands thank me for), which means that a smaller phone is probably a good call.

Oh while we’re talking about calls - I only have a data-only SIM in it, so I could not test calling. I’m transferring to a new phone contract this month, and I’ll give it a go then. This will be the first time I get VoLTE and WiFi calling, although it is Vodafone, so quality might just be worse than Telekom on 2G, who knows. A big shoutout to congstar for letting me cancel with a simple button click, and to @vodafoneservice on twitter for quickly setting up my benefits of additional 5GB per month and 10€ discount for being an existing cable customer.

I’m also looking forward to playing around with the camera (especially night sight), and eSIM. And I’m getting a case from China, which was handed over to the Airline on Sep 17 according to Aliexpress, so I guess it should arrive in the next weeks. Oh, and screen protector is not here yet, so I can’t really judge the screen quality much, as I still have the factory protection film on it, and that’s just a blurry mess - but good enough for setting it up. Please Google, pre-apply a screen protector on future phones and include a simple bumper case.

I might report back in two weeks when I have spent some more time with the device.

Build & Accessories

The tablet is a fairly Pixel-style affair, in that the back has two components, one softer blue one housing the camera and a metal feeling gray one. Build quality is fairly good.

The volume and power buttons are located on the right side of the tablet, and this is one of the main issues: You end up accidentally pressing the power button when you want to turn your volume lower, despite the power button having a different texture.

Alongside the tablet, you also find a kickstand with a textile back, and a keyboard, both of which attach via magnets (and pogo pins for the keyboard). The keyboard is crammed, with punctuation keys being halfed in size, and it feels mushed compared to my usual experiences of ThinkPads and Model Ms, but it’s on par with other Chromebooks, which is surprising, given it’s a tablet attachment.

I mostly use the Duet as a tablet, and only attach the keyboard occasionally. Typing with the keyboard on your lap is suboptimal.

My first Duet had a few bunches of dead pixels, so I returned it, as I had a second one I could not cancel ordered as well. Oh dear. That one was fine!

Hardware & Connectivity

The Chromebook Duet is powered by a Mediatek Helio P60T SoC, 4GB of RAM, and a choice of 64 or 128 GB of main storage.

The tablet provides one USB-C port for charging, audio output (a 3.5mm adapter is provided in the box), USB hub, and video output; though, sadly, the latter is restricted to a maximum of 1080p30, or 1440x900 at 60 Hz. It can be charged using the included 10W charger, or use up to I believe 18W from a higher powered USB-C PD charger. I’ve successfully used the Chromebook with a USB-C monitor with attached keyboard, mouse, and DAC without any issues.

On the wireless side, the tablet provides 2x2 Wifi AC and Bluetooth 4.2. WiFi reception seemed just fine, though I have not done any speed testing, missing a sensible connection at the moment. I used Bluetooth to connect to my smartphone for instant tethering, and my Sony WH1000XM2 headphones, both of which worked without any issues.

The screen is a bright 400 nit display with excellent viewing angles, and the speakers do a decent job, meaning you can use easily use this for watching a movie when you’re alone in a room and idling around. It has a resolution of 1920x1200.

The device supports styluses following the USI standard. As of right now, the only such stylus I know about is an HP one, and it costs about 70€ or so.

Cameras are provided on the front and the rear, but produce terrible images.

Software: The tablet experience

The Chromebook Duet runs Chrome OS, and comes with access to Android apps using the play store (and sideloading in dev mode) and access to full Linux environments powered by LXD inside VMs.

The screen which has 1920x1200 is scaled to a ridiculous 1080x675 by default which is good for being able to tap buttons and stuff, but provides next to no content. Scaling it to 1350x844 makes things more balanced.

The Linux integration is buggy. Touches register in different places than where they happened, and the screen is cut off in full screen extremetuxracer, making it hard to recommend for such uses.

Android apps generally work fine. There are some issues with the back gesture not registering, but otherwise I have not found issues I can remember.

One major drawback as a portable media consumption device is that Android apps only work in Widevine level 3, and hence do not have access to HD content, and the web apps of Netflix and co do not support downloading. Though one of the Duets actually said L1 in check apps at some point (reported in issue 1090330). It’s also worth noting that Amazon Prime Video only renders in HD, unless you change your user agent to say you are Chrome on Windows - bad Amazon!

The tablet experience also lags in some other ways, as the palm rejection is overly extreme, causing it to reject valid clicks close to the edge of the display (reported in issue 1090326).

The on screen keyboard is terrible. It only does one language at a time, forcing me to switch between German and English all the time, and does not behave as you’d expect it when editing existing words - it does not know about them and thinks you are starting a new one. It does provide a small keyboard that you can move around, as well as a draw your letters keyboard, which could come in handy for stylus users, I guess. In any case, it’s miles away from gboard on Android.

Stability is a mixed bag right now. As of Chrome OS 83, sites (well only Disney+ so far…) sometimes get killed with SIGILL or SIGTRAP, and the device rebooted on its own once or twice. Android apps that use the DRM sometimes do not start, and the Netflix Android app sometimes reports it cannot connect to the servers.

Performance

Performance is decent to sluggish, with micro stuttering in a lot of places. The Mediatek CPU is comparable to Intel Atoms, and with only 4GB of RAM, and an entire Android container running, it’s starting to show how weak it is.

I found that Google Docs worked perfectly fine, as did websites such as Mastodon, Twitter, Facebook. Where the device really struggled was Reddit, where closing or opening a post, or getting a reply box could take 5 seconds or more. If you are looking for a Reddit browsing device, this is not for you. Performance in Netflix was fine, and Disney+ was fairly slow but still usable.

All in all, it’s acceptable, and given the price point and the build quality, probably the compromise you’d expect.

Summary

tl;dr:

• good: Build quality, bright screen, low price, included accessories
• bad: DRM issues, performance, limited USB-C video output, charging speed, on-screen keyboard, software bugs

The Chromebook Duet or IdeaPad Duet Chromebook is a decent tablet that is built well above its price point. It’s lackluster performance and DRM woes make it hard to give a general recommendation, though. It’s not a good laptop.

I can see this as the perfect note taking device for students, and as a cheap tablet for couch surfing, or as your on-the-go laptop replacement, if you need it only occasionally.

I cannot see anyone using this as their main laptop, although I guess some people only have phones these days, so: what do I know?

I can see you getting this device if you want to tinker with Linux on ARM, as Chromebooks are quite nice to tinker with, and a tablet is super nice.

• SMTP via Postfix
• Spam filtering via rspamd
• HTTP(S) via nginx and letsencrypt (certbot)
• Weechat relay
• OpenVPN server
• Shadowsocks proxy
• Unbound recursive DNS resolver, for the spam filtering

I rebooted one more time than necessary, though, as my cloud provider Hetzner recently started offering 2nd generation EPYC instances which I upgraded to from my Skylake Xeon based instance. I switched from the CX21 for 5.83€/mo to the CPX11 for 4.15€/mo. This involved a RAM downgrade - from 4GB to 2GB, but that’s fine, the maximum usage I saw was about 1.3 GB when running dose-distcheck (running hourly); and it’s good for everyone that AMD is giving Intel some good competition, I think.

Anyway, to get back to the distribution upgrade - it was fairly boring. I started yesterday by taking a copy of the server and launching it locally in a lxd container, and then tested the upgrade in there; to make sure I’m prepared for the real thing :)

I got a confusing prompt from postfix as to which site I’m operating (which is a normal prompt, but I don’t know why I see it on an upgrade); and a few config files I had changed locally.

As the server is managed by ansible, I just installed the distribution config files and dropped my changes (setting DPkg::Options { "--force-confnew"; };" in apt.conf), and then after the upgrade, ran ansible to redeploy the changes (after checking what changes it would do and adjusting a few things).

There are two remaining flaws:

1. I run rspamd from the upstream repository, and that’s not built for focal yet. So I’m still using the bionic binary, and have to keep bionic’s icu 60 and libhyperscan4 around for it.

   This is still preventing CI of the ansible config from passing for focal, because it won’t have the needed bionic packages around.

2. I run weechat from the upstream repository, and apt can’t tell the versions apart. Well, it can for the repositories, because they have Size fields - but status does not. Hence, it merges the installed version with the first repository it sees.

   What happens is that it installs from weechat.org, but then it believes the installed version is from archive.ubuntu.com and replaces it each dist-upgrade.

   I worked around it by moving the weechat.org repo to the front of sources.list, so that the it gets merged with that instead of the archive.ubuntu.com one, as it should be, but that’s a bit ugly.

I also should start the migration to EC certificates for TLS, and 0-RTT handshakes, so that the initial visit experience is faster. I guess I’ll have to move away from certbot for that, but I have not investigated this recently.

Compared to the 1.8 series, the APT 2.0 series features several new features, as well as improvements in performance, hardening. A lot of code has been removed as well, reducing the size of the library.

Highlighted Changes Since 1.8

New Features

• Commands accepting package names now accept aptitude-style patterns. The syntax of patterns is mostly a subset of aptitude, see apt-patterns(7) for more details.

• apt(8) now waits for the dpkg locks - indefinitely, when connected to a tty, or for 120s otherwise.

• When apt cannot acquire the lock, it prints the name and pid of the process that currently holds the lock.

• A new satisfy command has been added to apt(8) and apt-get(8)

• Pins can now be specified by source package, by prepending src: to the name of the package, e.g.:

  Package: src:apt
  Pin: version 2.0.0
  Pin-Priority: 990
  

  Will pin all binaries of the native architecture produced by the source package apt to version 2.0.0. To pin packages across all architectures, append :any.

Performance

• APT now uses libgcrypt for hashing instead of embedded reference implementations of MD5, SHA1, and SHA2 hash families.

• Distribution of rred and decompression work during update has been improved to take into account the backlog instead of randomly assigning a worker, which should yield higher parallelization.

Incompatibilities

• The apt(8) command no longer accepts regular expressions or wildcards as package arguments, use patterns (see New Features).

Hardening

• Credentials specified in auth.conf now only apply to HTTPS sources, preventing malicious actors from reading credentials after they redirected users from a HTTP source to an http url matching the credentials in auth.conf. Another protocol can be specified, see apt_auth.conf(5) for the syntax.

Developer changes

• A more extensible cache format, allowing us to add new fields without breaking the ABI

• All code marked as deprecated in 1.8 has been removed

• Implementations of CRC16, MD5, SHA1, SHA2 have been removed

• The apt-inst library has been merged into the apt-pkg library.

• apt-pkg can now be found by pkg-config

• The apt-pkg library now compiles with hidden visibility by default.

• Pointers inside the cache are now statically typed. They cannot be compared against integers (except 0 via nullptr) anymore.

python-apt 2.0

python-apt 2.0 is not yet ready, I’m hoping to add a new cleaner API for cache access before making the jump from 1.9 to 2.0 versioning.

libept 1.2

I’ve moved the maintenance of libept to the APT team. We need to investigate how to EOL this properly and provide facilities inside APT itself to replace it. There are no plans to provide new features, only bugfixes / rebuilds for new apt versions.

so, what are patterns?

Patterns allow you to specify complex search queries to select the packages you want to install/show. For example, the pattern ?garbage can be used to find all packages that have been automatically installed but are no longer depended upon by manually installed packages. Or the pattern ?automatic allows you find all automatically installed packages.

You can combine patterns into more complex ones; for example, ?and(?automatic,?obsolete) matches all automatically installed packages that do not exist any longer in a repository.

There are also explicit targets, so you can perform queries like ?for x: ?depends(?recommends(x)): Find all packages x that depend on another package that recommends x. I do not fully comprehend those yet - I did not manage to create a pattern that matches all manually installed packages that a meta-package depends upon. I am not sure it is possible.

reducing pattern syntax

aptitude’s syntax for patterns is quite context-sensitive. If you have a pattern ?foo(?bar) it can have two possible meanings:

1. If ?foo takes arguments (like ?depends did), then ?bar is the argument.
2. Otherwise, ?foo(?bar) is equivalent to ?foo?bar which is short for ?and(?foo,?bar)

I find that very confusing. So, when looking at implementing patterns in APT, I went for a different approach. I first parse the pattern into a generic parse tree, without knowing anything about the semantics, and then I convert the parse tree into a APT::CacheFilter::Matcher, an object that can match against packages.

This is useful, because the syntactic structure of the pattern can be seen, without having to know which patterns have arguments and which do not - basically, for the parser ?foo and ?foo() are the same thing. That said, the second pass knows whether a pattern accepts arguments or not and insists on you adding them if required and not having them if it does not accept any, to prevent you from confusing yourself.

aptitude also supports shortcuts. For example, you could write ~c instead of config-files, or ~m for automatic; then combine them like ~m~c instead of using ?and. I have not implemented these short patterns for now, focusing instead on getting the basic functionality working.

So in our example ?foo(?bar) above, we can immediately dismiss parsing that as ?foo?bar:

1. we do not support concatenation instead of ?and.
2. we automatically parse ( as the argument list, no matter whether ?foo supports arguments or not

Supported syntax

At the moment, APT supports two kinds of patterns: Basic logic ones like ?and, and patterns that apply to an entire package as opposed to a specific version. This was done as a starting point for the merge, patterns for versions will come in the next round.

We also do not have any support for explicit search targets such as ?for x: ... yet - as explained, I do not yet fully understand them, and hence do not want to commit on them.

The full list of the first round of patterns is below, helpfully converted from the apt-patterns(7) docbook to markdown by pandoc.

logic patterns

These patterns provide the basic means to combine other patterns into more complex expressions, as well as ?true and ?false patterns.

?and(PATTERN, PATTERN, ...)

Selects objects where all specified patterns match.

?false

Selects nothing.

?not(PATTERN)

Selects objects where PATTERN does not match.

?or(PATTERN, PATTERN, ...)

Selects objects where at least one of the specified patterns match.

?true

Selects all objects.

package patterns

These patterns select specific packages.

?architecture(WILDCARD)

Selects packages matching the specified architecture, which may contain wildcards using any.

?automatic

Selects packages that were installed automatically.

?broken

Selects packages that have broken dependencies.

?config-files

Selects packages that are not fully installed, but have solely residual configuration files left.

?essential

Selects packages that have Essential: yes set in their control file.

?exact-name(NAME)

Selects packages with the exact specified name.

?garbage

Selects packages that can be removed automatically.

?installed

Selects packages that are currently installed.

?name(REGEX)

Selects packages where the name matches the given regular expression.

?obsolete

Selects packages that no longer exist in repositories.

?upgradable

Selects packages that can be upgraded (have a newer candidate).

?virtual

Selects all virtual packages; that is packages without a version. These exist when they are referenced somewhere in the archive, for example because something depends on that name.

examples

apt remove ?garbage

Remove all packages that are automatically installed and no longer needed - same as apt autoremove

apt purge ?config-files

Purge all packages that only have configuration files left

oddities

Some things are not yet where I want them:

• ?architecture does not support all, native, or same
• ?installed should match only the installed version of the package, not the entire package (that is what aptitude does, and it’s a bit surprising that ?installed implies a version and ?upgradable does not)

the future

Of course, I do want to add support for the missing version patterns and explicit search patterns. I might even add support for some of the short patterns, but no promises. Some of those explicit search patterns might have slightly different syntax, e.g. ?for(x, y) instead of ?for x: y in order to make the language more uniform and easier to parse.

Another thing I want to do ASAP is to disable fallback to regular expressions when specifying package names on the command-line: apt install g++ should always look for a package called g++, and not for any package containing g (g++ being a valid regex) when there is no g++ package. I think continuing to allow regular expressions if they start with ^ or end with $ is fine - that prevents any overlap with package names, and would avoid breaking most stuff.

There also is the fallback to fnmatch(): Currently, if apt cannot find a package with the specified name using the exact name or the regex, it would fall back to interpreting the argument as a glob(7) pattern. For example, apt install apt* would fallback to installing every package starting with apt if there is no package matching that as a regular expression. We can actually keep those in place, as the glob(7) syntax does not overlap with valid package names.

Maybe I should allow using [] instead of () so larger patterns become more readable, and/or some support for comments.

There are also plans for AppStream based patterns. This would allow you to use apt install ?provides-mimetype(text/xml) or apt install ?provides-lib(libfoo.so.2). It’s not entirely clear how to package this though, we probably don’t want to have libapt-pkg depend directly on libappstream.

feedback

Talk to me on IRC, comment on the Mastodon thread, or send me an email if there’s anything you think I’m missing or should be looking at.

Architecture

The basic architecture chosen for encrypted storage is that every incoming email is delivered to postfix via LMTP, and then postfix runs a sieve script that invokes a filter that encrypts the email with PGP/MIME using a user-specific key, before processing it further. Or short:

postfix --ltmp--> dovecot --sieve--> filter --> gpg --> inbox

Security analysis: This means that the message will be on the system unencrypted as long as it is in a Postfix queue. This further means that the message plain text should be recoverable for quite some time after Postfix deleted it, by investigating in the file system. However, given enough time, the probability of being able to recover the messages should reduce substantially. Not sure how to improve this much.

And yes, if the email is already encrypted we’re going to encrypt it a second time, because we can nest encryption and signature as much as we want! Makes the code easier.

Encrypting an email with PGP/MIME

PGP/MIME is a trivial way to encrypt an email. Basically, we take the entire email message, armor-encrypt it with GPG, and stuff it into a multipart mime message with the same headers as the second attachment; the first attachment is a control information.

Technically, this means that we keep headers twice, once encrypted and once decrypted. But the advantage compared to doing it more like most normal clients is clear: The code is a lot easier, and we can reverse the encryption and get back the original!

And when I say easy, I mean easy - the function to encrypt the email is just a few lines long:

def encrypt(message: email.message.Message, recipients: typing.List[str]) -> str:
    """Encrypt given message"""
    encrypted_content = gnupg.GPG().encrypt(message.as_string(), recipients)
    if not encrypted_content:
        raise ValueError(encrypted_content.status)

    # Build the parts
    enc = email.mime.application.MIMEApplication(
        _data=str(encrypted_content).encode(),
        _subtype='octet-stream',
        _encoder=email.encoders.encode_7or8bit)

    control = email.mime.application.MIMEApplication(
        _data=b'Version: 1\n',
        _subtype='pgp-encrypted; name="msg.asc"',
        _encoder=email.encoders.encode_7or8bit)
    control['Content-Disposition'] = 'inline; filename="msg.asc"'

    # Put the parts together
    encmsg = email.mime.multipart.MIMEMultipart(
        'encrypted',
        protocol='application/pgp-encrypted')
    encmsg.attach(control)
    encmsg.attach(enc)

    # Copy headers
    headers_not_to_override = {key.lower() for key in encmsg.keys()}

    for key, value in message.items():
        if key.lower() not in headers_not_to_override:
            encmsg[key] = value

    return encmsg.as_string()

Decypting the email is even easier: Just pass the entire thing to GPG, it will decrypt the encrypted part, which, as mentioned, contains the entire original email with all headers :)

def decrypt(message: email.message.Message) -> str:
    """Decrypt the given message"""
    return str(gnupg.GPG().decrypt(message.as_string()))

(now, not sure if it’s a feature that GPG.decrypt ignores any unencrypted data in the input, but well, that’s GPG for you).

Of course, if you don’t actually need IMAP access, you could drop PGP/MIME and just pipe emails through gpg --encrypt --armor before dropping them somewhere on the filesystem, and then sync them via ssh somehow (e.g. patching maildirsync to encrypt emails it uploads to the server, and decrypting emails it downloads).

Pretty Easy privacy (p≡p)

Now, we almost have a file conforming to draft-marques-pep-email-02, the Pretty Easy privacy (p≡p) format, version 2. That format allows us to encrypt headers, thus preventing people from snooping on our metadata!

Basically it relies on the fact that we have all the headers in the inner (encrypted) message. To mark an email as conforming to that format we just have to set the subject to p≡p and add a header describing the format version:

       Subject: =?utf-8?Q?p=E2=89=A1p?=
       X-Pep-Version: 2.0

A client conforming to p≡p will, when seeing this email, read any headers from the inner (encrypted) message.

We also might want to change the code to only copy a limited amount of headers, instead of basically every header, but I’m going to leave that as an exercise for the reader.

Putting it together

Assume we have a Postfix and a Dovecot configured, and a script gpgmymail written using the function above, like this:

def main() -> None:
    """Program entry"""
    parser = argparse.ArgumentParser(
        description="Encrypt/Decrypt mail using GPG/MIME")
    parser.add_argument('-d', '--decrypt', action="store_true",
                        help="Decrypt rather than encrypt")
    parser.add_argument('recipient', nargs='*',
                        help="key id or email of keys to encrypt for")
    args = parser.parse_args()
    msg = email.message_from_file(sys.stdin)

    if args.decrypt:
        sys.stdout.write(decrypt(msg))
    else:
        sys.stdout.write(encrypt(msg, args.recipient))


if __name__ == '__main__':
    main()

(don’t forget to add missing imports, or see the end of the blog post for links to full source code)

Then, all we have to is edit our .dovecot.sieve to add

filter "gpgmymail" "myemail@myserver.example";

and all incoming emails are automatically encrypted.

Outgoing emails

To handle outgoing emails, do not store them via IMAP, but instead configure your client to add a Bcc to yourself, and then filter that somehow in sieve. You probably want to set Bcc to something like myemail+sent@myserver.example, and then filter on the detail (the sent).

Encrypt or not Encrypt?

Now do you actually want to encrypt? The disadvantages are clear:

• Server-side search becomes useless, especially if you use p≡p with encrypted Subject.

  Such a shame, you could have built your own GMail by writing a notmuch FTS plugin for dovecot!

• You can’t train your spam filter via IMAP, because the spam trainer won’t be able to decrypt the email it is supposed to learn from

There are probably other things I have not thought about, so let me know on mastodon, email, or IRC!

More source code

You can find the source code of the script, and the setup for dovecot in my git repository.

As my old shared hosting account expired on January 1, I had to move mail forwarding duties over to the new server. Yes forwarding - I do plan to move hosting the actual email too, but at the moment it’s “just” forwarding to gmail.

The Software

As you might know from the web server story, my server runs on Ubuntu 18.04. I set up a mail server on this system using

• Postfix for SMTP duties (warning, they oddly do not have an https page)
• rspamd for spam filtering, and signing DKIM / ARC
• bind9 for DNS resolving
• postsrsd for SRS

You might wonder why bind9 is in there. It turns out that DNS blacklists used by spam filters block the caching DNS servers you usually use, so you have to use your own recursive DNS server. Ubuntu offers you the choice between bind9 and dnsmasq in main, and it seems like bind9 is more appropriate here than dnsmasq.

Setting up postfix

Most of the postfix configuration is fairly standard. So, let’s skip TLS configuration and outbound SMTP setups (this is email, and while they support TLS, it’s all optional, so let’s not bother that much here).

The most important part is restrictions in main.cf.

First of all, relay restrictions prevent us from relaying emails to weird domains:

# Relay Restrictions
smtpd_relay_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain permit_mynetworks permit_sasl_authenticated defer_unauth_destination

We also only accept mails from hosts that know their own full qualified name:

# Helo restrictions (hosts not having a proper fqdn)
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname

We also don’t like clients (other servers) that send data too early, or have an unknown hostname:

smtpd_data_restrictions = reject_unauth_pipelining
smtpd_client_restrictions = permit_mynetworks reject_unknown_client_hostname

I also set up a custom apparmor profile that’s pretty lose, I plan to migrate to the one in the apparmor git eventually but it needs more testing and some cleanup.

Sender rewriting scheme

For SRS using postsrsd, we define the SRS_DOMAIN in /etc/default/postsrsd and then configure postfix to talk to it:

# Handle SRS for forwarding
recipient_canonical_maps = tcp:localhost:10002
recipient_canonical_classes= envelope_recipient,header_recipient

sender_canonical_maps = tcp:localhost:10001
sender_canonical_classes = envelope_sender

This has a minor issue that it also rewrites the Return-Path when it delivers emails locally, but as I am only forwarding, I’m worrying about that later.

rspamd basics

rspamd is a great spam filtering system. It uses a small core written in C and a bunch of Lua plugins, such as:

• IP score, which keeps track of how good a specific server was in the past
• Replies, which can check whether an email is a reply to another one
• DNS blacklisting
• DKIM and ARC validation and signing
• DMARC validation
• SPF validation

It also has a nice web UI:

Setting up rspamd is quite easy. You basically just drop a bunch of configuration overrides into /etc/rspamd/local.d and you’re done. Heck, it mostly works out of the box. There’s a fancy rspamadm configwizard too.

What you do want for rspamd is a redis server. redis is needed in many places, such as rate limiting, greylisting, dmarc, reply tracking, ip scoring, neural networks.

I made a few changes to the defaults:

• I enabled subject rewriting instead of adding headers, so spam mail subjects get [SPAM] prepended, in local.d/actions.conf:

  reject = 15;
  rewrite_subject = 6;
  add_header = 6;
  greylist = 4;
  subject = "[SPAM] %s";
  

• I set autolearn = true; in local.d/classifier-bayes.conf to make it learn that an email that has a score of at least 15 (those that are rejected) is spam, and emails with negative scores are ham.

• I set extended_spam_headers = true; in local.d/milter_headers.conf to get a report from rspamd in the header seeing the score and how the score came to be.

ARC setup

ARC is the ‘Authenticated Received Chain’ and is currently a DMARC working group work item. It allows forwarders / mailing lists to authenticate their forwarding of the emails and the checks they have performed.

rspamd is capable of validating and signing emails with ARC, but I’m not sure how much influence ARC has on gmail at the moment, for example.

There are three parts to setting up ARC:

1. Generate a DKIM key pair (use rspamadm dkim_keygen)
2. Setup rspamd to sign incoming emails using the private key
3. Add a DKIM TXT record for the public key. rspamadm helpfully tells you how it looks like.

For step two, what we need to do is configure local.d/arc.conf. You can basically use the example configuration from the rspamd page, the key point for signing incoming email is to specifiy sign_incoming = true; and use_domain_sign_inbound = "recipient"; (FWIW, none of these options are documented, they are fairly new, and nobody updated the documentation for them).

My configuration looks like this at the moment:

# If false, messages with empty envelope from are not signed
allow_envfrom_empty = true;
# If true, envelope/header domain mismatch is ignored
allow_hdrfrom_mismatch = true;
# If true, multiple from headers are allowed (but only first is used)
allow_hdrfrom_multiple = false;
# If true, username does not need to contain matching domain
allow_username_mismatch = false;
# If false, messages from authenticated users are not selected for signing
auth_only = true;
# Default path to key, can include '$domain' and '$selector' variables
path = "${DBDIR}/arc/$domain.$selector.key";
# Default selector to use
selector = "arc";
# If false, messages from local networks are not selected for signing
sign_local = true;
#
sign_inbound = true;
# Symbol to add when message is signed
symbol_signed = "ARC_SIGNED";
# Whether to fallback to global config
try_fallback = true;
# Domain to use for ARC signing: can be "header" or "envelope"
use_domain = "header";
use_domain_sign_inbound = "recipient";
# Whether to normalise domains to eSLD
use_esld = true;
# Whether to get keys from Redis
use_redis = false;
# Hash for ARC keys in Redis
key_prefix = "ARC_KEYS";

This would also sign any outgoing email, but I’m not sure that’s necessary - my understanding is that we only care about ARC when forwarding/receiving incoming emails, not when sending them (at least that’s what gmail does).

Other Issues

There are few other things to keep in mind when running your own mail server. I probably don’t know them all yet, but here we go:

• You must have a fully qualified hostname resolving to a public IP address

• Your public IP address must resolve back to the fully qualified host name

• Again, you should run a recursive DNS resolver so your DNS blacklists work (thanks waldi for pointing that out)

• Setup an SPF record. Mine looks like this:

  jak-linux.org. 3600 IN TXT "v=spf1 +mx ~all"

  this states that all my mail servers may send email, but others probably should not (a softfail). Not having an SPF record can punish you; for example, rspamd gives missing SPF and DKIM a score of 1.

• All of that software is sandboxed using AppArmor. Makes you question its security a bit less!

Source code, outlook

As always, you can find the Ansible roles on GitHub. Feel free to point out bugs! 😉

In the next installment of this series, we will be looking at setting up Dovecot, and configuring DKIM. We probably also want to figure out how to run notmuch on the server, keep messages in matching maildirs, and have my laptop synchronize the maildir and notmuch state with the server. Ugh, sounds like a lot of work.

Go is an imperative programming language for concurrent programming created at and mainly developed by Google, initially mostly by Robert Griesemer, Rob Pike, and Ken Thompson. Design of the language started in 2007, and an initial version was released in 2009; with the first stable version, 1.0 released in 2012 ¹.

Go has a C-like syntax (without a preprocessor), garbage collection, and, like its predecessors devloped at Bell Labs – Newsqueak (Rob Pike), Alef (Phil Winterbottom), and Inferno (Pike, Ritchie, et al.) – provides built-in support for concurrency using so-called goroutines and channels, a form of co-routines, based on the idea of Hoare’s ‘Communicating Sequential Processes’ ².

Go programs are organised in packages. A package is essentially a directory containing Go files. All files in a package share the same namespace, and there are two visibilities for symbols in a package: Symbols starting with an upper case character are visible to other packages, others are private to the package:

func PublicFunction() {
    fmt.Println("Hello world")
}

func privateFunction() {
    fmt.Println("Hello package")
}

Types

Go has a fairly simple type system: There is no subtyping (but there are conversions), no generics, no polymorphic functions, and there are only a few basic categories of types:

1. base types: int, int64, int8, uint, float32, float64, etc.

2. struct

3. interface - a set of methods

4. map[K, V] - a map from a key type to a value type

5. [number]Type - an array of some element type

6. []Type - a slice (pointer to array with length and capability) of some type

7. chan Type - a thread-safe queue

8. pointer *T to some other type

9. functions

10. named type - aliases for other types that may have associated methods:

    type T struct { foo int }
    type T *T
    type T OtherNamedType
    

    Named types are mostly distinct from their underlying types, so you cannot assign them to each other, but some operators like + do work on objects of named types with an underlying numerical type (so you could add two T in the example above).

Maps, slices, and channels are reference-like types - they essentially are structs containing pointers. Other types are passed by value (copied), including arrays (which have a fixed length and are copied).

Conversions

Conversions are the similar to casts in C and other languages. They are written like this:

    TypeName(value)

Constants

Go has “untyped” literals and constants.

    1    // untyped integer literal
    const foo = 1 // untyped integer constant
    const foo int = 1 // int constant

Untyped values are classified into the following categories: UntypedBool, UntypedInt, UntypedRune, UntypedFloat, UntypedComplex, UntypedString, and UntypedNil (Go calls them basic kinds, other basic kinds are available for the concrete types like uint8). An untyped value can be assigned to a named type derived from a base type; for example:

type someType int

const untyped = 2             // UntypedInt
const bar someType = untyped  // OK: untyped can be assigned to someType
const typed int = 2           // int
const bar2 someType = typed   // error: int cannot be assigned to someType

Interfaces and ‘objects’

As mentioned before, interfaces are a set of methods. Go is not an object-oriented language per se, but it has some support for associating methods with named types: When declaring a function, a receiver can be provided - a receiver is an additional function argument that is passed before the function and involved in the function lookup, like this:

type SomeType struct { ... }

func (s *SomeType) MyMethod() {
}

func main() {
    var s SomeType
    s.MyMethod()
}

An object implements an interface if it implements all methods; for example, the following interface MyMethoder is implemented by *SomeType (note the pointer), and values of *SomeType can thus be used as values of MyMethoder. The most basic interface is interface{}, that is an interface with an empty method set - any object satisfies that interface.

type MyMethoder interface {
    MyMethod()
}

There are some restrictions on valid receiver types; for example, while a named type could be a pointer (for example, type MyIntPointer *int), such a type is not a valid receiver type.

Control flow

Go provides three primary statements for control flow: if, switch, and for. The statements are fairly similar to their equivalent in other C-like languages, with some exceptions:

• There are no parentheses around conditions, so it is if a == b {}, not if (a == b) {}. The braces are mandatory.

• All of them can have initialisers, like this

  if result, err := someFunction(); err == nil { // use result }

• The switch statement can use arbitrary expressions in cases

• The switch statement can switch over nothing (equals switching over true)

• Cases do not fall through by default (no break needed), use fallthrough at the end of a block to fall through.

• The for loop can loop over ranges: for key, val := range map { do something }

Goroutines

The keyword go spawns a new goroutine, a concurrently executed function. It can be used with any function call, even a function literal:

func main() {
    ...
    go func() {
        ...
    }()

    go some_function(some_argument)
}

Channels

Goroutines are often combined with channels to provide an extended form of Communicating Sequential Processes ². A channel is a concurrent-safe queue, and can be buffered or unbuffered:

var unbuffered = make(chan int)  // sending blocks until value has been read
var buffered = make(chan int, 5) // may have up to 5 unread values queued

The <- operator is used to communicate with a single channel.

valueReadFromChannel := <- channel
otherChannel <- valueToSend

The select statement allows communication with multiple channels:

select {
    case incoming := <- inboundChannel:
        // A new message for me
    case outgoingChannel <- outgoing:
        // Could send a message, yay!
}

The defer statement

Go provides a defer statement that allows a function call to be scheduled for execution when the function exits. It can be used for resource clean-up, for example:

func myFunc(someFile io.ReadCloser) {
    defer someFile.close()
    /* Do stuff with file */
}

It is of course possible to use function literals as the function to call, and any variables can be used as usual when writing the call.

Error handling

Go does not provide exceptions or structured error handling. Instead, it handles errors by returning them in a second or later return value:

func Read(p []byte) (n int, err error)

// Built-in type:
type error interface {
        Error() string
}

Errors have to be checked in the code, or can be assigned to _:

n0, _ := Read(Buffer)   // ignore error
n, err := Read(buffer)
if err != nil {
    return err
}

There are two functions to quickly unwind and recover the call stack, though: panic() and recover(). When panic() is called, the call stack is unwound, and any deferred functions are run as usual. When a deferred function invokes recover(), the unwinding stops, and the value given to panic() is returned. If we are unwinding normally and not due to a panic, recover() simply returns nil. In the example below, a function is deferred and any error value that is given to panic() will be recovered and stored in an error return value. Libraries sometimes use that approach to make highly recursive code like parsers more readable, while still maintaining the usual error return value for public functions.

func Function() (err error) {
    defer func() {
        s := recover()
        switch s := s.(type) {  // type switch
            case error:
                err = s         // s has type error now
            default:
                panic(s)
        }
    }
}

Arrays and slices

As mentioned before, an array is a value type and a slice is a pointer into an array, created either by slicing an existing array or by using make() to create a slice, which will create an anonymous array to hold the elements.

slice1 := make([]int, 2, 5) // 5 elements allocated, 2 initialized to 0
slice2 := array[:]          // sliced entire array
slice3 := array[1:]         // slice of array without first element

There are some more possible combinations for the slicing operator than mentioned above, but this should give a good first impression.

A slice can be used as a dynamically growing array, using the append() function.

slice = append(slice, value1, value2)
slice = append(slice, arrayOrSlice...)

Slices are also used internally to represent variable parameters in variable length functions.

Maps

Maps are simple key-value stores and support indexing and assigning. They are not thread-safe.

someValue := someMap[someKey]
someValue, ok := someMap[someKey]   // ok is false if key not in someMap
someMap[someKey] = someValue

• jak-linux.org
• dep.debian.net redirector
• mirror.fail

Rationale

Uberspace runs CentOS 6. This was causing more and more issues for me, as I was trying to run up-to-date weechat binaries. In the final stages, I ran weechat and tmux inside a debian proot. It certainly beat compiling half a system with linuxbrew.

The web performance was suboptimal. Webpages are served with Pound and Apache, TLS connection overhead was just huge, there was only HTTP/1.1, and no keep-alive.

Security-wise things were interesting: Everything ran as my user, obviously, whether that’s scripts, weechat, or mail delivery helpers. Ugh. There was also only a single certificate, meaning that all domains shared it, even if they were completely distinct like jak-linux.org and dep.debian.net

Enter Hetzner VPS

I launched a VPS at hetzner and configured it with Ubuntu 18.04, the latest Ubuntu LTS. It is a CX21, so it has 2 vcores, 4 GB RAM, 40 GB SSD storage, and 20 TB of traffic. For 5.83€/mo, you can’t complain.

I went on to build a repository of ansible roles (see repo on github.com), that configured the system with a few key characteristics:

• http is served by nginx
• certificates are per logical domain - each domain has a canonical name and a set of aliases; and the certificate is generated for them all
• HTTPS is configured according to Mozilla’s modern profile, meaning TLSv1.2-only, and a very restricted list of ciphers. I can revisit that if it’s causing problems, but I’ve not seen huge issues.
• Log files are anonymized to 24 bits for IPv4 addresses, and 32 bit for IPv6 addresses, which should allow me to identify an ISP, but not an individual user.

I don’t think the roles are particularly reusable for others, but it’s nice to have a central repository containing all the configuration for the server.

Go server to serve comments

When I started self-hosting the blog and added commenting via mastodon, it was via a third-party PHP script. This has been replaced by a Go program (GitHub repo). The new Go program scales a lot better than a PHP script, and provides better security properties due to AppArmor and systemd-based sandboxing; it even uses systemd’s DynamicUser.

Special care has been taken to have time outs for talking to upstream servers, so the program cannot hang with open connections and will respond eventually.

The Go binary is connected to nginx via a UNIX domain socket that serves FastCGI. The service is activated via systemd socket activation, allowing it to be owned by www-data, while the binary runs as a dynamic user. Nginx’s native fastcgi caching mechanism is enabled so the Go process is only contacted every 10 minutes at the most (for a given post). Nice!

Performance

Performance is a lot better than the old shared server. Pages load in up to half the time of the old one. Scalability also seems better: I tried various benchmarks, and achieved consistently higher concurrency ratings. A simple curl via https now takes 100ms instead of 200ms.

Performance is still suboptimal from the west coast of the US or other places far away from Germany, but got a lot better than before: Measuring from Oregon using webpagetest, it took 1.5s for a page to fully render vs ~3.4s before. A CDN would surely be faster, but would lose the end-to-end encryption.

Upcoming mail server

The next step is to enable email. Setting up postfix with dovecot is quite easy it turns out. Install them, tweak a few settings, setup SPF, DKIM, DMARC, and a PTR record, and off you go.

I mostly expect to read my email by tagging it on the server using notmuch somehow, and then syncing it to my laptop using muchsync. The IMAP access should allow some notifications or reading on the phone.

Spam filtering will be handled with rspamd. It seems to be the hot new thing on the market, is integrated with postfix as a milter, and handles a lot of stuff, such as:

• greylisting
• IP scoring
• DKIM verification and signing
• ARC verification
• SPF verification
• DNS lists
• Rate limiting

It also has fancy stuff like neural networks. Woohoo!

As another bonus point: It’s trivial to confine with AppArmor, which I really love. Postfix and Dovecot are a mess to confine with their hundreds of different binaries.

I found it via uberspace, which plan on using it for their next uberspace7 generation. It is also used by some large installations like rambler.ru and locaweb.com.br.

I plan to migrate mail from uberspace in the upcoming weeks, and will post more details about it.

Today, I finished converting it to Hugo.

Why?

I did not really have a huge problem with ikiwiki, but I recently converted my blog from wordpress to hugo and it seemed to make sense to have one technology for both, especially since I don’t update the website very often and forget ikiwiki’s special things.

One thing that was somewhat annoying is that I built a custom ikiwiki plugin for the menu in my template, so I had to clone it’s repository into ~/.ikiwiki every time, rather than having a self-contained website. Well, it was a submodule of my dotfiles repo.

Another thing was that ikiwiki had a lot of git integration, and when you build your site it tries to push things to git repositories and all sorts of weird stuff – Hugo just does one thing: It builds your page.

One thing that Hugo does a lot better than ikiwiki is the built-in server which allows you to run `hugo server´ and get a local http URL you can open in the browser with live-reload as you save files. Super convenient to check changes (and of course, for writing this blog post)!

Also, in general, Hugo feels a lot more modern. ikiwiki is from 2006, Hugo is from 2013. Especially recent Hugo versions added quite a few features for asset management.

• Fingerprinting of assets like css (inserting hash into filename) - ikiwiki just contains its style in style.css (and your templates in other statically named files), so if you switch theming details, you could break things because the CSS the browser has cached does not match the CSS the page expects.
• Asset minification - Hugo can minimize CSS and JavaScript for you. This means browers have to fetch less data.
• Asset concatenation - Hugo can concatenate CSS and JavaScript. This allows you to serve only one file per type, reducing the number of round trips a client has to make.

There’s also proper theming support, so you can easily clone a theme into the themes/ directory, or add it as a submodule like I do for my blog. But I don’t use it for the website yet.

Oh, and Hugo automatically generates sitemap.xml files for your website, teaching search engines which pages exist and when they have been modified.

I also like that it’s written in Go vs in Perl, but I think that’s just another more modern type of thing. Gotta keep up with the world!

Basic conversion

The first part to the conversion was to split the repository of the website: ikiwiki puts templates into a templates/ subdirectory of the repository and mixes all other content. Hugo on the other hand splits things into content/ (where pages go), layouts (page templates), and static/ (other files).

The second part was to inject the frontmatter into the markdown files. See, ikiwiki uses shortcuts like this to set up the title, and gets its dates from git:

[[!meta title="My page title"]]

on the other hand, Hugo uses frontmatter - some YAML at the beginning of the markdown, and specifies the creation date in there:

---
title: "My page title"
date: Thu, 18 Oct 2018 21:36:18 +0200
---

You can also have lastmod in there when modifying it, but I set enableGitInfo = true in config.toml so Hugo picks up the mtime from the git repo.

I wrote a small script to automatize those steps, but it was obviously not perfect (also, it inserted lastmod, which it should not have).

One thing it took me some time to figure out was that index.mdown needs to become _index.md in the content/ directory of Hugo, otherwise no pages below it are rendered - not entirely obvious.

The theme

Converting the template was surprisingly easy, it was just a matter of replacing <TMPL_VAR BASEURL> and friends with { .Site.BaseURL } and friends - the names are basically the same, just sometimes there’s .Site at the front of it.

Then I had to take care of the menu generation loop. I had my bootmenu plugin for ikiwiki which allowed me to generate menus from the configuration file. The template for it looked like this:

<TMPL_LOOP BOOTMENU>
    <TMPL_IF FIRSTNAV>
        <li <TMPL_IF ACTIVE>class="active"</TMPL_IF>><a href="<TMPL_VAR URL>"><TMPL_VAR PAGE></a></li>
    </TMPL_IF>
</TMPL_LOOP>

I converted this to:

{{ $currentPage := . }}
{{ range .Site.Menus.main }}
    <li class="{{ if $currentPage.IsMenuCurrent "main" . }}active{{ end }}">
        <a href="{{ .URL }}">
            {{ .Pre | safeHTML }}
            <span>{{ .Name }}</span>
        </a>
        {{ .Post }}
    </li>
{{ end }}

this allowed me to configure my menu in config.toml like this:

[menu]

  [[menu.main]]
    name = "dh-autoreconf"
    url = "/projects/dh-autoreconf"
    weight = -110

I can also specify pre and post parts and a right menu, and I use pre and post in the right menu to render a few icons before and after items, for example:

  [[menu.right]]
    pre = "<i class='fab fa-mastodon'></i>"
    post = "<i class='fas fa-external-link-alt'></i>"
    url = "https://mastodon.social/@juliank"
    name = "Mastodon"
    weight = -70

Setting class="active" on the menu item does not seem to work yet, though; I think I need to find out the right code for that…

Fixing up the details

Once I was done with that steps, the next stage was to convert ikiwiki shortcodes to something hugo understands. This took 4 parts:

The first part was converting tables. In ikiwiki, tables look like this:

[[!table format=dsv data="""
Status|License|Language|Reference
Active|GPL-3+|Java|[github](https://github.com/julian-klode/dns66)
"""]]

The generated HTML table had the class="table" set, which the bootstrap framework needs to render a nice table. Converting that to a straightforward markdown hugo table did not work: Hugo did not add the class, so I had to convert pages with tables in them to the mmark variant of markdown, which allows classes to be set like this {.table}, so the end result then looked like this:

{.table}
Status|License|Language|Reference
------|-------|--------|---------
Active|GPL-3+|Java|[github](https://github.com/julian-klode/dns66)

I’ll be able to get rid of this in the future by using the bootstrap sources and then having table inherit .table properties, but this requires saas or less, and I only have the CSS at the moment, so using mmark was slightly easier.

The second part was converting ikiwiki links like [[MyPage]] and [[my title|MyPage]] to Markdown links. This was quite easy, the first one became [MyPage](MyPage) and the second one [my title](my page).

The third part was converting custom shortcuts: I had [[!lp <number>]] to generate a link LP: #<number> to the corresponding launchpad bug, and [[!Closes <number>]] to generate Closes: #<number> links to the Debian bug tracker. I converted those to normal markdown links, but I could have converted them to Hugo shortcodes. But meh.

The fourth part was about converting some directory indexes I had. For example, [[!map pages="projects/dir2ogg/0.12/* and ! projects/dir2ogg/0.12/*/*"]] generated a list of all files in projects/dir2ogg/0.12. There was a very useful shortcode for that posted on the Hugo documentation, I used a variant of it and then converted pages like this to {{< directoryindex path="/static/projects/dir2ogg/0.12" pathURL="/projects/dir2ogg/0.12" >}}. As a bonus, the new directory index also generates SHA256 hashes for all files!

Further work

The website is using an old version of bootstrap, and the theme is not split out yet. I’m not sure if I want to keep a bootstrap theme for the website, seeing as the blog theme is Bulma-based - it would be easier to have both use bulma.

I also might want to update both the website and the blog by pushing to GitHub and then using CI to build and push it. That would allow me to write blog posts when I don’t have my laptop with me. But I’m not sure, I might lose control if there’s a breach at travis.

Let’s talk details: In spring, I shutdown my wordpress.com hosted blog, due to concerns about GDPR implications with comment hosting and ads and stuff. I’d like to apologize for using that, back when I did this (in 2007), it was the easiest way to get into blogging. Please forgive me for subjecting you to that!

Recently, Google announced the end of Google+. As some of you might know, I posted a lot of medium-long posts there, rather than doing blog posts; especially after I disabled the wordpress site.

With the end of Google+, I want to try something new: I’ll host longer pieces on this blog, and post shorter messages on @juliank@mastodon.social. If you follow the Mastodon account, you will see toots for each new blog post as well, linking to the blog post.

Mastodon integration and privacy

Now comes the interesting part: If you reply to the toot, your reply will be shown on the blog itself. This works with a tiny bit of JavaScript that talks to a simple server-side script that finds toots from me mentioning the blog post, and then replies to that.

This protects your privacy, because mastodon.social does not see which blog post you are looking at, because it is contacted by the server, not by you. Rendering avatars requires loading images from mastodon.social’s file server, however - to improve your privacy, all avatars are loaded with referrerpolicy='no-referrer', so assuming your browser is half-way sane, it should not be telling mastodon.social which post you visited either. In fact, the entire domain also sets Referrer-Policy: no-referrer as an http header, so any link you follow will not have a referrer set.

The integration was originally written by @bjoern@mastodon.social – I have done some moderate improvements to adapt it to my theme, make it more reusable, and replace and extend the caching done in a JSON file with a Redis database.

Source code

This blog is free software; generated by the Hugo snap. All source code for it is available:

• The blog posts are provided at github.com/julian-klode/blog.jak-linux.org. under the terms of the CC BY-SA 4.0 license.

• The theme is called Ernest, and is a fork of the Hemingway theme. You can find it at github.com/julian-klode/ernest. It is licensed under the MIT license, with server-side scripts under the AGPL, and some vendorized parts under different licenses.

(Yes I am aware that hosting the repositories on GitHub is a bit ironic given the whole focus on privacy and self-hosting).

The theme makes use of Hugo pipes to minify and fingerprint JavaScript, and vendorizes all dependencies instead of embedding CDN links, to, again, protect your privacy.

Future work

I think I want to make the theme dark, to be more friendly to the eyes. I also might want to make the mastodon integration a bit more friendly to use. And I want to get rid of jQuery, it’s only used for a handful of calls in the Mastodon integration JavaScript.

If you have any other idea for improvements, feel free to join the conversation in the mastodon toot, send me an email, or open an issue at the github projects.

Closing thoughts

I think the end of Google+ will be an interesting time, requring a lot of people in the open source world to replace one of their main communication channels with a different approach.

Mastodon and Diaspora are both in the race, and I fear the community will split or everyone will have two accounts in the end. I personally think that Mastodon + syndicated blogs provide a good balance: You can quickly write short posts (up to 500 characters), and you can host long articles on your own and link to them.

I hope that one day diaspora* and mastodon federate together. If we end up with one federated network that would be the best outcome.

I released apt 1.5 this year, and started 1.6 with seccomp sandboxing for methods.

I went to DebConf17 in Montreal. I unfortunately did not make it to DebCamp, nor the first day, but I at least made the rest of the conference. There, I gave a talk about APT development in the past year, and had a few interesting discussions. One thing that directly resulted from such a discusssion was a new proposal for delta upgrades, with a very simple delta format based on a variant of bsdiff (with external compression, streamable patches, and constant memory use rather than linear). I hope we can implement this - the savings are enormous with practically no slowdown (there is no reconstruction phase, upgrades are streamed directly to the file system), which is especially relevant for people with slow or data capped connections.

This month, I’ve been buying a few “toys”: I got a pair of speakers (JBL LSR 305), and I got a noise cancelling headphone (a Sony WH-1000XM2). Nice stuff. Been wearing the headphones most of today, and they’re quite comfortable and really make things quite, except for their own noise ;) Well, both the headphone and the speakers have a white noise issue, but oh well, the prices were good.

This time of the year is not only a time to look back at the past year, but also to look forward to the year ahead. In one week, I’ll be joining Canonical to work on Ubuntu foundation stuff. It’s going to be interesting. I’ll also be moving places shortly, having partially lived in student housing for 6 years (one room, and a shared kitchen), I’ll be moving to a complete apartement.

On the APT front, I plan to introduce a few interesting changes. One of them involves automatic removal of unused packages: This should be happening automatically during install, upgrade, and whatever. Maybe not for all packages, though - we might have a list of “safe” autoremovals. I’d also be interested in adding metadata for transitions: Like if libfoo1 replaces libfoo0, we can safely remove libfoo0 if nothing depends on it anymore. Maybe not for all “garbage” either. It might make sense to restrict it to new garbage - that is packages that become unused as part of the operation. This is important for safe handling of existing setups with automatically removable packages: We don’t suddenly want to remove them all when you run upgrade.

The other change is about sandboxing. You might have noticed that sometimes, sandboxing is disabled with a warning because the method would not be able access the source or the target. The goal is to open these files in the main program and send file descriptors to the methods via a socket. This way, we can avoid permission problems, and we can also make the sandbox stronger - for example, by not giving it access to the partial/ directory anymore.

Another change we need to work on is standardising the Important field, which is sort of like essential - it marks an installed package as extra-hard to remove (but unlike Essential, does not cause apt to install it automatically). The latest draft calls it Protected, but I don’t think we have a consensus on that yet.

I also need to get happy eyeballs done - fast fallback from IPv6 to IPv4. I had a completely working solution some months ago, but it did not pass CI, so I decided to start from scratch with a cleaner design to figure out if I went wrong somewhere. Testing this is kind of hard, as it basically requires a broken IPv6 setup (well, unreachable IPv6 servers).

Oh well, 2018 has begun, so I’m going to stop now. Let’s all do our best to make it awesome!

This was a real problem, because the http method did in fact execute programs - there is this small option called ProxyAutoDetect or Proxy-Auto-Detect where you can specify a script to run for an URL and the script outputs a (list of) proxies. In order to be able to seccomp the http method, I moved the invocation of the script to the parent process. The parent process now executes the script within the sandbox user, but without seccomp (obviously).

I tested the code on amd64, ppc64el, s390x, arm64, mipsel, i386, and armhf. I hope it works on all other architectures libseccomp is currently built for in Debian, but I did not check that, so your apt might be broken now if you use powerpc, powerpcspe, armel, mips, mips64el, hhpa, or x32 (I don’t think you can even really use x32).

Also, apt-transport-https is gone for good now. When installing the new apt release, any installed apt-transport-https package is removed (apt breaks apt-transport-https now, but it also provides it versioned, so any dependencies should still be satisfiable).

David also did a few cool bug fixes again, finally teaching apt-key to ignore unsupported GPG key files instead of causing weird errors :)

This release series moves https support from apt-transport-https into apt proper, bringing with it support for https:// proxies, and support for autodetectproxy scripts that return http, https, and socks5h proxies for both http and https.

Unattended updates and upgrades now work better: The dependency on network-online was removed and we introduced a meta wait-online helper with support for NetworkManager, systemd-networkd, and connman that allows us to wait for network even if we want to run updates directly after a resume (which might or might not have worked before, depending on whether update ran before or after network was back up again). This also improves a boot performance regression for systems with rc.local files:

The rc.local.service unit specified After=network-online.target, and login stuff was After=rc.local.service, and apt-daily.timer was Wants=network-online.target, causing network-online.target to be pulled into the boot and the rc.local.service ordering dependency to take effect, significantly slowing down the boot.

An earlier less intrusive variant of that fix is in 1.4.8: It just moves the network-online.target Want/After from apt-daily.timer to apt-daily.service so most boots are uncoupled now. I hope we get the full solution into stretch in a later point release, but we should gather some experience first before discussing this with the release time.

Balint Reczey also provided a patch to increase the time out before killing the daily upgrade service to 15 minutes, to actually give unattended-upgrades some time to finish an in-progress update. Honestly, I’d have though the machine hung up and force rebooted it after 5 seconds already. (this patch is also in 1.4.8)

We also made sure that unreadable config files no longer cause an error, but only a warning, as that was sort of a regression from previous releases; and we added documentation for /etc/apt/auth.conf, so people actually know the preferred way to place sensitive data like passwords (and can make their sources.list files world-readable again).

We also fixed apt-cdrom to support discs without MD5 hashes for Sources (the Files field), and re-enabled support for udev-based detection of cdrom devices which was accidentally broken for 4 years, as it was trying to load libudev.so.0 at runtime, but that library had an SONAME change to libudev.so.1 - we now link against it normally.

Furthermore, if certain information in Release files change, like the codename, apt will now request confirmation from the user, avoiding a scenario where a user has stable in their sources.list and accidentally upgrades to the next release when it becomes stable.

Paul Wise contributed patches to allow configuring the apt-daily intervals more easily - apt-daily is invoked twice a day by systemd but has more fine-grained internal timestamp files. You can now specify the intervals in seconds, minutes, hours, and day units, or specify “always” to always run (that is, up to twice a day on systemd, once per day on non-systemd platforms).

Development for the 1.6 series has started, and I intent to upload a first alpha to unstable in about a week, removing the apt-transport-https package and enabling compressed index files by default (save space, a lot of space, at not much performance cost thanks to lz4). There will also be some small clean ups in there, but I don’t expect any life-changing changes for now.

I think our new approach of uploading development releases directly to unstable instead of parking them in experimental is working out well. Some people are confused why alpha releases appear in unstable, but let me just say one thing: These labels basically just indicate feature-completeness, and not stability. An alpha is just very likely to get a lot more features, a beta is less likely (all the big stuff is in), and the release candidates just fix bugs.

Also, we now have 3 active stable series: The 1.2 LTS series, 1.4 medium LTS, and 1.5. 1.2 receives updates as part of Ubuntu 16.04 (xenial), 1.4 as part of Debian 9.0 (stretch) and Ubuntu 17.04 (zesty); whereas 1.5 will only be supported for 9 months (as part of Ubuntu 17.10). I think the stable release series are working well, although 1.4 is a bit tricky being shared by stretch and zesty right now (but zesty is history soon, so …).

TUF divides signing responsibilities into roles: A root role, a targets rule (signing stuff to download), a snapshots rule (signing meta data), and a time stamp rule (signing a time stamp file). There also is a mirror role for signing a list of mirrors, but we can ignore that for now. It strongly recommends that all keys except for timestamp and mirrors are kept offline, which is not applicable for APT repositories - Ubuntu updates the repository every 30 minutes, imagine doing that with offline keys. An insane proposal.

In APT repositories, we effectively only have a snapshots rule - the only thing we sign are Release files, and trust is then chained down by hashes (Release files hashes Packages index files, and they have hashes of individual packages). The keys used to sign repositories are online keys, after all, all the metadata files change every 30 minutes (Ubuntu) or 6 hours (Debian) - it’s impossible to sign them by hand. The timestamp role is replaced by a field in the Release file specifying until when the Release file is considered valid.

Let’s check the attacks TUF protects again:

• Arbitrary installation attacks. - We protect against that with the outer signature and hashes
• Endless data attacks. - Yes, we impose a limit on Release files (the sizes of other files are specified in there and this file is signed)
• Extraneous dependencies attacks - That’s verified by the signed hashes of Packages files
• Fast-forward attacks - same
• Indefinite freeze attacks - APT has a Valid-Until field that can be used to specify a maximum life time of a release file
• Malicious mirrors preventing updates. - Well, the user configures the mirror, so usually not applicable. if the user has multiple mirrors, APT deals with that fine
• Mix-and-match attacks - Again, signed Release file and hashes of other files
• Rollback attacks - We do not allow Date fields in Release files to go backwards
• Slow retrieval attacks - TUF cannot protect against that either. APT has very high timeouts, and there is no reasonable answer to that.
• Vulnerability to key compromises - For our purposes where we need all repository signing keys to be online, as we need to sign new releases and metadata fairly often, it does not make it less vulnerable to require a threshold of keys (APT allows repositories to specify concrete key ids they may be signed with though, that has the same effect)
• Wrong software installation. - Does not happen, the .deb files are hashed in the Packages files which are signed by the release file

As we can see, APT addresses all attacks TUF addresses.

But both do not handle key revocation. So, if a key & mirror gets compromised (or just key and the mirror is MITMed), we cannot inform the user that the key has been compromised and block updates from the compromised repository.

I just wrote up a proposal to allow APT to query for revoked keys from a different host with a key revocation list (KRL) file that is signed by different keys than the repository. This would solve the problem of key revocation easily - even if the repository host is MITMed or compromised, we can still revoke the keys signing the repository from a different location.

I requested the domain transfer from STRATO on Monday at 16:23, received the auth codes at 20:10 and the .de domain was transferred completely on 20:36 (about 20 minutes if you count my overhead). The .org domain I had to ACK, which I did at 20:46 and at 03:00 I received the notification that the transfer was successful (I think there was some registrar ACKing involved there). So the whole transfer took about 10 1/2 hours, or 7 hours since I retrieved the auth code. I think that’s quite a good time :)

And, for those of you who don’t know: uberspace is a shared hoster that basically just gives you an SSH shell account, directories for you to drop files in for the http server, and various tools to add subdomains, certificates, virtual users to the mailserver. You can also run your own custom build software and open ports in their firewall. That’s quite cool.

I’m considering migrating the blog away from wordpress at some point in the future - having a more integrated experience is a bit nicer than having my web presence split over two sites. I’m unsure if I shouldn’t add something like cloudflare there - I don’t want to overload the servers (but I only serve static pages, so how much load is this really going to get?).

in other news: off-site backups

I also recently started doing offsite backups via borg to a server operated by the wonderful rsync.net. For those of you who do not know rsync.net: You basically get SSH to a server where you can upload your backups via common tools like rsync, scp, or you can go crazy and use git-annex, borg, attic; or you could even just plain zfs send your stuff there.

The normal price is $0.08 per GB per month, but there is a special borg price of $0.03 (that price does not include snapshotting or support, really). You can also get a discounted normal account for $0.04 if you find the correct code on Hacker News, or other discounts for open source developers, students, etc. - you just have to send them an email.

Finally, I must say that uberspace and rsync.net feel similar in spirit. Both heavily emphasise the command line, and don’t really have any fancy click stuff. I like that.

Security changes

APT 1.4 by default disables support for repositories signed with SHA1 keys. I announced back in January that it was my intention to do this during the summer for development releases, but I only remembered the Jan 1st deadline for stable releases supporting that (APT 1.2 and 1.3), so better late than never.

Around January 1st, the same or a similar change will occur in the APT 1.2 and 1.3 series in Ubuntu 16.04 and 16.10 (subject to approval by Ubuntu’s release team). This should mean that repository provides had about one year to fix their repositories, and more than 8 months since the release of 16.04. I believe that 8 months is a reasonable time frame to upgrade a repository signing key, and hope that providers who have not updated their repositories yet will do so as soon as possible.

Performance work

APT 1.4 provides a 10-20% performance increase in cache generation (and according to callgrind, we went from approx 6.8 billion to 5.3 billion instructions for my laptop’s configuration, a reduction of more than 21%). The major improvements are:

We switched the parsing of Deb822 files (such as Packages files) to my perfect hash function TrieHash. TrieHash - which generates C code from a set of words - is about equal or twice as fast as the previously used hash function (and two to three times faster than gperf), and we save an additional 50% of that time as we only have to hash once during parsing now, instead of during look up as well. APT 1.4 marks the first time TrieHash is used in any software. I hope that it will spread to dpkg and other software at a later point in time.vendors.

Another important change was to drop normalization of Description-MD5 values, the fields mapping a description in a Packages files to a translated description. We used to parse the hex digits into a native binary stream, and then compared it back to hex digits for comparisons, which cost us about 5% of the run time performance.

We also optimized one of our hash functions - the VersionHash that hashes the important fields of a package to recognize packages with the same version, but different content - to not normalize data to a temporary buffer anymore. This buffer has been the subject of some bugs (overflow, incompleteness) in the recent past, and also caused some slowdown due to the additional writes to the stack. Instead, we now pass the bytes we are interested in directly to our CRC code, one byte at a time.

There were also some other micro-optimisations: For example, the hash tables in the cache used to be ordered by standard compare (alphabetical followed by shortest). It is now ordered by size first, meaning we can avoid data comparisons for strings of different lengths. We also got rid of a std::string that cannot use short string optimisation in a hot path of the code. Finally, we also converted our case-insensitive djb hashes to not use a normal tolower_ascii(), but introduced tolower_ascii_unsafe() which just sets the “lowercase bit” (| 0x20) in the character.

Others

• Sandboxing now removes some environment variables like TMP from the environment.
• Several improvements to installation ordering.
• Support for armored GPG keys in trusted.gpg.d.
• Various other fixes

For a more complete overview of all changes, consult the changelog.

I’m proud (yes, really) to announce DNS66, my host/ad blocker for Android 5.0 and newer. It’s been around since last Thursday on F-Droid, but it never really got a formal announcement.

DNS66 creates a local VPN service on your Android device, and diverts all DNS traffic to it, possibly adding new DNS servers you can configure in its UI. It can use hosts files for blocking whole sets of hosts or you can just give it a domain name to block (or multiple hosts files/hosts). You can also whitelist individual hosts or entire files by adding them to the end of the list. When a host name is looked up, the query goes to the VPN which looks at the packet and responds with NXDOMAIN (non-existing domain) for hosts that are blocked.

You can find DNS66 here:

• on GitHub: https://github.com/julian-klode/dns66
• on F-Droid: https://f-droid.org/app/org.jak_linux.dns66

F-Droid is the recommended source to install from. DNS66 is licensed under the GNU GPL 3, or (mostly) any later version.

Implementation Notes

DNS66’s core logic is based on another project, dbrodie/AdBuster, which arguably has the cooler name. I translated that from Kotlin to Java, and cleaned up the implementation a bit:

All work is done in a single thread by using poll() to detect when to read/write stuff. Each DNS request is sent via a new UDP socket, and poll() polls over all UDP sockets, a Device Socket (for the VPN’s tun device) and a pipe (so we can interrupt the poll at any time by closing the pipe).

We literally redirect your DNS servers. Meaning if your DNS server is 1.2.3.4, all traffic to 1.2.3.4 is routed to the VPN. The VPN only understands DNS traffic, though, so you might have trouble if your DNS server also happens to serve something else. I plan to change that at some point to emulate multiple DNS servers with fake IPs, but this was a first step to get it working with fallback: Android can now transparently fallback to other DNS servers without having to be aware that they are routed via the VPN.

We also need to deal with timing out queries that we received no answer for: DNS66 stores the query into a LinkedHashMap and overrides the removeEldestEntry() method to remove the eldest entry if it is older than 10 seconds or there are more than 1024 pending queries. This means that it only times out up to one request per new request, but it eventually cleans up fine.

I introduce TrieHash an algorithm for constructing perfect hash functions from tries. The generated hash functions are pure C code, minimal, order-preserving and outperform existing alternatives. Together with the generated header files,they can also be used as a generic string to enumeration mapper (enums are created by the tool).

Introduction

APT (and dpkg) spend a lot of time in parsing various files, especially Packages files. APT currently uses a function called AlphaHash which hashes the last 8 bytes of a word in a case-insensitive manner to hash fields in those files (dpkg just compares strings in an array of structs).

There is one obvious drawback to using a normal hash function: When we want to access the data in the hash table, we have to hash the key again, causing us to hash every accessed key at least twice. It turned out that this affects something like 5 to 10% of the cache generation performance.

Enter perfect hash functions: A perfect hash function matches a set of words to constant values without collisions. You can thus just use the index to index into your hash table directly, and do not have to hash again (if you generate the function at compile time and store key constants) or handle collision resolution.

As #debian-apt people know, I happened to play a bit around with tries this week before guillem suggested perfect hashing. Let me tell you one thing: My trie implementation was very naive, that did not really improve things a lot…

Enter TrieHash

Now, how is this related to hashing? The answer is simple: I wrote a perfect hash function generator that is based on tries. You give it a list of words, it puts them in a trie, and generates C code out of it, using recursive switch statements (see code generation below). The function achieves competitive performance with other hash functions, it even usually outperforms them.

Given a dictionary, it generates an enumeration (a C enum or C++ enum class) of all words in the dictionary, with the values corresponding to the order in the dictionary (the order-preserving property), and a function mapping strings to members of that enumeration.

By default, the first word is considered to be 0 and each word increases a counter by one (that is, it generates a minimal hash function). You can tweak that however:

= 0
WordLabel ~ Word
OtherWord = 9

will return 0 for an unknown value, map “Word” to the enum member WordLabel and map OtherWord to 9. That is, the input list functions like the body of a C enumeration. If no label is specified for a word, it will be generated from the word. For more details see the documentation

C code generation

switch(string[0] | 32) {
case 't':
    switch(string[1] | 32) {
    case 'a':
        switch(string[2] | 32) {
        case 'g':
            return Tag;
        }
    }
}
return Unknown;

Yes, really recursive switches - they directly represent the trie. Now, we did not really do a straightforward translation, there are some optimisations to make the whole thing faster and easier to look at:

First of all, the 32 you see is used to make the check case insensitive in case all cases of the switch body are alphabetical characters. If there are non-alphabetical characters, it will generate two cases per character, one upper case and one lowercase (with one break in it). I did not know that lowercase and uppercase characters differed by only one bit before, thanks to the clang compiler for pointing that out in its generated assembler code!

Secondly, we only insert breaks only between cases. Initially, each case ended with a return Unknown, but guillem (the dpkg developer) suggested it might be faster to let them fallthrough where possible. Turns out it was not faster on a good compiler, but it’s still more readable anywhere.

Finally, we build one trie per word length, and switch by the word length first. Like the 32 trick, his gives a huge improvement in performance.

Digging into the assembler code

The whole code translates to roughly 4 instructions per byte:

1. A memory load,
2. an or with 32
3. a comparison, and
4. a conditional jump.

(On x86, the case sensitive version actually only has a cmp-with-memory and a conditional jump).

Due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77729 this may be one instruction more: On some architectures an unneeded zero-extend-byte instruction is inserted - this causes a 20% performance loss.

Performance evaluation

I run the hash against all 82 words understood by APT in Packages and Sources files, 1,000,000 times for each word, and summed up the average run-time:

Suffice to say, GPerf does not really come close.

All hosts except the x230 are Debian porterboxes. The x230 is my laptop with a a Core i5-3320M, barriere has an Opteron 23xx. I included the DJB hash function for another reference.

Source code

The generator is written in Perl, licensed under the MIT license and available from https://github.com/julian-klode/triehash - I initially prototyped it in Python, but guillem complained that this would add new build dependencies to dpkg, so I rewrote it in Perl.

Benchmark is available from https://github.com/julian-klode/hashbench

Usage

See the script for POD documentation.

Today, I wrote sicherboot, a tool to integrate systemd-boot into a Linux distribution in an entirely new way: With secure boot support. To be precise: The use case here is to only run trusted code which then unmounts an otherwise fully encrypted disk, as in my setup:

If you want, sicherboot automatically creates db, KEK, and PK keys, and puts the public keys on your EFI System Partition (ESP) together with the KeyTool tool, so you can enroll the keys in UEFI. You can of course also use other keys, you just need to drop a db.crt and a db.key file into /etc/sicherboot/keys. It would be nice if sicherboot could enroll the keys directly in Linux, but there seems to be a bug in efitools preventing that at the moment. For some background: The Platform Key (PK) signs the Key Exchange Key (KEK) which signs the database key (db). The db key is the one signing binaries.

sicherboot also handles installing new kernels to your ESP. For this, it combines the kernel with its initramfs into one executable UEFI image, and then signs that. Combined with a fully encrypted disk setup, this assures that only you can run UEFI binaries on the system, and attackers cannot boot any other operating system or modify parts of your operating system (except for, well, any block of your encrypted data, as XTS does not authenticate the data; but then you do have to know which blocks are which which is somewhat hard).

sicherboot integrates with various parts of Debian: It can work together by dracut via an evil hack (diverting dracut’s kernel/postinst.d config file, so we can run sicherboot after running dracut), it should support initramfs-tools (untested), and it also integrates with systemd upgrades via triggers on the /usr/lib/systemd/boot/efi directory.

Currently sicherboot only supports Debian-style setups with /boot/vmlinuz-<version> and /boot/initrd.img-<version> files, it cannot automatically create combined boot images from or install boot loader entries for other naming schemes yet. Fixing that should be trivial though, with a configuration setting and some eval magic (or string substitution).

Future planned features include: (1) support for multiple ESP partitions, so you can have a fallback partition on a different drive (think RAID type situation, keep one ESP on each drive, so you can remove a failing one); and (2) a tool to create a self-contained rescue disk image from a directory (which will act as initramfs) and a kernel (falling back to a vmlinuz file )

It might also be interesting to add support for other bootloaders and setups, so you could automatically sign a grub cryptodisk image for example. Not sure how much sense that makes.

I published the source at https://github.com/julian-klode/sicherboot (MIT licensed) and uploaded the package to Debian, it should enter the NEW queue soon (or be in NEW by the time you read this). Give it a try, and let me know what you think.

Prior to 1.3~rc4, acquiring the files for an update worked like this: We create some object for the Release file, once a release file is done we queue any next object (DEP-11 icons, .diff/Index files, etc). There is no prioritizing, so usually we fetch the 5MB+ DEP-11 icons and components files first, and only then start working on other indices which might use Pdiff.

In 1.3~rc4 I changed the queues to be priority queues: Release files and .diff/Index files have the highest priority (once we have them all, we know how much to fetch). The second level of priority goes to the .pdiff files which are later on passed to the rred process to patch an existing Packages, Sources, or Contents file. The third priority level is taken by all other index targets.

Actually, I implemented the priority queues back in Jun. There was just one tiny problem: Pipelining. We might be inserting elements into our fetching queues in order of priority, but with pipelining enabled, stuff of lower priority might already have their HTTP request sent before we even get to queue the higher priority stuff.

Today I had an epiphany: We fill the pipeline up to a number of items (the depth, currently 10). So, let’s just fill the pipeline with items that have the same (or higher) priority than the maximum priority of the already-queued ones; and pretend it is full when we only have lower priority items.

And that works fine: First the Release and .diff/Index stuff is fetched, which means we can start showing accurate progress info from there one. Next, the pdiff files are fetched, meaning that we can apply them in parallel to any targets downloading later in parallel (think DEP-11 icon tarballs).

This has a great effect on performance: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds.

In other news

I recently cleaned up the apt packaging which renamed /usr/share/bug/apt/script to /usr/share/bug/apt. That broke on overlayfs, because dpkg could not rename the old apt directory to a backup name during unpack (only directories purely on the upper layer can be renamed). I reverted that now, so all future updates should be fine.

David re-added the Breaks against apt-utils I recently removed by accident during the cleanup, so no more errors about overriding dump solvers. He also added support for fingerprints in gpgv’s GOODSIG output, which apparently might come at some point.

I Also fixed a few CMake issues, fixed the test suite for gpgv 2.1.15, allow building with a system-wide gtest library (we really ought to add back a pre-built one in Debian), and modified debian/rules to pass -O to make. I wish debhelper would do the latter automatically (there’s a bug for that).

Finally, we fixed some uninitialized variables in the base256 code, out-of-bound reads in the Sources file parser, off-by-one errors in the tagfile comment stripping code[1], and some memcpy() with length 0. Most of these will be cherry-picked into the 1.2 (xenial) and 1.0.9.8 (jessie) branches (releases 1.2.15 and 1.0.9.8.4). If you forked off your version of apt at another point, you might want to do the same.

[1] those were actually causing the failures and segfaults in the unit tests on hurd-i386 buildds. I always thought it was a hurd-specific issue…

PS. Building for Fedora on OBS has a weird socket fd #3 that does not get closed during the test suite despite us setting CLOEXEC on it. Join us in #debian-apt on oftc if you have ideas.

Fast forward 7 years to 2016. A few months ago, we noticed that our build system had trouble with correct dependencies in parallel building. So, in search for a way out, I picked up my CMake branch from 2009 last Thursday and spent the whole weekend working on it, and today I am happy to announce that I merged it into master:

123 files changed, 1674 insertions(+), 3205 deletions(-)

More than 1500 lines less build system code. Quite impressive, eh? This also includes about 200 lines of less code in debian/, as that switched from prehistoric debhelper stuff to modern dh (compat level 9, almost ready for 10).

The annoying Tale of Targets vs Files

Talking about CMake: I don’t really love it. As you might know, CMake differentiates between targets and files. Targets can in some cases depend on files (generated by a command in the same directory), but overall files are not really targets. You also cannot have a target with the same name as a file you are generating in a custom command, you have to rename your target (make is OK with the generated stuff, but ninja complains about cycles because your custom target and your custom command have the same name).

Byproducts for the (time) win

One interesting thing about CMake and Ninja are byproducts. In our tree, we are building C++ files. We also have .pot templates depending on them, and .mo files depending on the templates (we have multiple domains, and merge the per-domain .pot with the all-domain .po file during the build to get a per-domain .mo). Now, if we just let them depend naively, changing a C++ file causes the .pot file to be regenerated which in turns causes us to build .mo files for every freaking language in the package. Even if nothing changed.

Byproducts solve this problem. Instead of just building the .pot file, we also create a stamp file (AKA the witness) and write the .pot file (without a header) into a temporary name and only copy it to its final name if the content changed. The .pot file is declared as a byproduct of the command.

The command doing the .pot->.mo step still depends on the .pot file (the byproduct), but as that only changes now if strings change, the .mo files only get rebuild if I change a translatable string. We still need to ensure that that the .pot file is actually built before we try to use it - the solution here is to specify a custom target depending on the witness and then have the target containing the .mo build commands depend on that target.

Now if you use  make, you might now this trick already. In make, the byproducts remain undeclared, though, while in CMake we can now actually express them, and they are used by the Ninja generator and the Ninja build tool if you chose that over make (try it out, it’s fast).

Further Work

Some command names are hardcoded, I should find_program() them. Also cross-building the package does not yet work successfully, but it only requires a tiny amount of patches in debhelper and/or cmake.

I also tried building the package on a Fedora docker image (with dpkg installed, it’s available in the Fedora sources). While I could eventually get the programs build and most of the integration test suite to pass, there are some minor issues to fix, mostly in the documentation building and GTest department: Fedora ships its docbook stylesheets in a different location, and ships GTest as a pre-compiled library, and not a source tree.

I have not yet tested building on exotic platforms like macOS, or even a BSD. Please do and report back. In Debian, CMake is not up-to.date enough on the non-Linux platforms to build APT due to test suite failures, I hope those can be fixed/disabled soon (it appears to be a timing issue AFAICT).

I hope that we eventually get some non-Debian backends for APT. I’d love that.

I initially tried obnam. Obnam seems like a good tool, but is insanely slow. Unencrypted it can write about 3 MB/s, which is somewhat OK, but even then it can spend hours forgetting generations (1 generation takes probably 2 minutes, and there might be 22 of them). In encrypted mode, the speed reduces a lot, to about 500 KB/s if I recall correctly, which is just unusable.

I found borg backup, a fork of attic. Borg backup achieves speeds of up to 15 MB/s which is really nice. It’s also faster with scanning: I can now run my bihourly backups in about 1 min 30s (usually backs up about 30 to 80 MB - mostly thanks to Chrome I suppose!). And all those speeds are with encryption turned on.

Both borg and obnam use some form of chunks from which they compose files. Obnam stores each chunk in its own file, borg stores multiple chunks (even from different files) in a single pack file which is probably the main reason it is faster.

So how am I backing up: My laptop has an internal SSD and an HDD.  I backup every 2 hours (at 09,11,13,15,17,19,21,23,01:00 hours) using a systemd timer event, from the SSD to the HDD. The backup includes all of $HOME except for Downloads, .cache, the trash, Android SDK, and the eclipse and IntelliJ IDEA IDEs.

Now the magic comes in: The backup repository on the HDD is monitored by git-annex assistant, which automatically encrypts and uploads any new files in there to my 1&1 WebDAV drive and registers them in a git repository hosted on bitbucket. All files are encrypted and checksummed using SHA256, reducing the chance of the backup being corrupted.

I’m not sure how the WebDAV thing will work once I want to prune things, I suspect it will then delete some pack files and repack things into new files which means it will spend more bandwidth than obnam would. I’d also have to convince git-annex to actually drop anything from the WebDAV remote, but that is not really that much of a concern with 1TB storage space in the next 2 years at least…

I also have an external encrypted HDD which I can take backups on, it currently houses a fuller backup of $HOME that also includes Downloads, the Android SDK, and the IDEs for quicker recovery. Downloads changes a lot, and all of them can be fairly easily re-retrieved from the internet as needed, so there’s not much point in painfully uploading them to a WebDAV backup site.

Update: I can’t recommend this. Get a service that has native borg support such as rsync.net or a storage box at hetzner.

Despite of what I wrote earlier, we now print warnings for Release files signed with signatures using SHA1 as the digest algorithm. This involved extending the protocol APT uses to communicate with the methods a bit, by adding a new 104 Warning message type.

W: gpgv:/var/lib/apt/lists/apt.example.com_debian_dists_sid_InRelease: The repository is insufficiently signed by key
1234567890ABCDEF0123456789ABCDEF01234567 (weak digest)

Also note that SHA1 support is not dropped, we merely do not consider it trustworthy. This means that it feels like SHA1 support is dropped, because sources without SHA2 won’t work; but the SHA1 signatures will still be used in addition to the SHA2 ones, so there’s no point removing them (same for MD5Sum fields).

We also fixed some small bugs!

This will mean that starting tomorrow, some third party repositories may stop working, such as the one for the web browser I am writing this with. Debian Derivatives should be mostly safe for that change, if they are registered in the Consensus, as that has checks for that. This is a bit unfortunate, but we have no real choice: Technical restrictions prevent us from just showing a warning in a sensible way.

There is one caveat, however: GPG signatures may still use SHA1. While I have prepared the needed code to reject SHA1-based signatures in APT, a lot of third party repositories still ship Release files signed with signatures using SHA-1 as the digest algorithms. Some repositories even still use 1024-bit DSA keys.

I plan to enforce SHA2 for GPG signatures some time after the release of xenial, and definitely for Ubuntu 16.10, so around June-August (possibly during DebConf). For xenial, I plan to have a SRU (stable release update) in January to do the same (it’s just adding one member to an array). This should give 3rd party providers a reasonable time frame to migrate to a new digest algorithm for their GPG config and possibly a new repository key.

Summary

• Tomorrow: Disabling SHA1 support for Release, Packages, Sources files
• June/July/August: Disabling SHA1 support for GPG signatures (InRelease/Release.gpg) in development releases
• January 2017: Disabling SHA1 support for GPG signatures in Ubuntu 16.04 LTS via a stable-release-update.

Important fix for 1.1.6 regression

Since APT 1.1.6, APT uses the configured xz compression level. Unfortunately, the default was set to 9, which requires 674 MiB of RAM, compared to the 94 MiB required at level 6.

This caused the test suite to fail on the Ubuntu autopkgtest servers, but I thought it was just some temporary hickup on their part, and so did not look into it for the 1.1.7, 1.1.8, and 1.1.9 releases.  When the Ubuntu servers finally failed with 1.1.9 again (they only started building again on Monday it seems), I noticed something was wrong.

Enter git bisect. I created a script that compiles the APT source code and runs a test with ulimit for virtual and resident memory set to 512 (that worked in 1.1.5), and let it ran, and thus found out the reason mentioned above.

The solution: APT now defaults to level 6.

New Features

APT 1.1.8 introduces /usr/lib/apt/apt-helper cat-file which can be used to read files compressed by any compressor understood by APT. It is used in the recent apt-file experimental release, and serves to prepare us for a future in which files on the disk might be compressed with a different compressor (such as LZ4 for Contents files, this will improve rred speed on them by factor 7).

David added a feature that enables servers to advertise that they do not want APT to download and use some Architecture: all contents when they include all in their list of architectures. This is to allow archives to drop Architecture: all packages from the architecture-specific content files, to avoid redundant data and (thus) improve the performance of apt-file.

Buffered writes

APT 1.1.9 introduces buffered writing for rred, reducing the runtime by about 50% on a slowish SSD, and maybe more on HDDs. The 1.1.9 release is a bit buggy and might mess up things when a write syscall is interrupted, this is fixed in 1.1.10.

Cache generation improvements

APT 1.1.9 and APT 1.1.10 improve the cache generation algorithms in several ways: Switching a lookup table from std::map to std::unordered_map, providing an inline isspace_ascii() function, and inlining the tolower_ascii() function which are tiny functions that are called a lot.

APT 1.1.10 also switches the cache’s hash function to the DJB hash function and increases the default hash table sizes to the smallest prime larger than 15000, namely 15013. This reduces the average bucket size from 6.5 to 4.5. We might increase this further in the future.

Checksum for the cache, but no more syncs

Prior to APT 1.1.10 writing the cache was a multi-part process:

  1. Write the the cache to a temporary file with the dirty bit set to true
  2. Call `fsync()` to sync the cache
  3. Write a new header with the dirty bit set to false
  4. Call `fsync()` to sync the new header
  5. (Rename the temporary file to the target name)

The last step was obviously not needed, as we could easily live with an intact cache that has its dirty field set to false, as we can just rebuild it.

But what matters more is step 2. Synchronizing the entire 40 or 50 MB takes some time. On my HDD system, it consumed 56% of the entire cache generation time, and on my SSD system, it consumed 25% of the time.

APT 1.1.10 does not sync the cache at all. It now embeds a hashsum (adler32 for performance reasons) in the cache. This helps ensure that no matter what parts of the cache are written in case of some failure somewhere, we can still detect a failure with reasonable confidence (and even more errors than before).

This means that cache generation is now much faster for a lot of people. On the bad side, commands like apt-cache show that previously took maybe 10 ms to execute can now take about 80 ms.

Please report back on your performance experience with 1.1.10 release, I’m very interested to see if that works reasonably for other people. And if you have any other idea how to solve the issue, I’d be interested to hear them (all data needs to be written before the header with dirty=0 is written, but we don’t want to sync the data).

Future work

We seem to have a lot of temporary (?) std::string objects during the cache generation, accounting for about 10% of the run time. I’m thinking of introducing a string_view class similar to the one proposed for C++17 and make use of that.

I also thought about calling posix_fadvise() before starting to parse files, but the cache generation process does not seem to spend a lot of its time in system calls (even with all caches dropped before the run), so I don’t think this will improve things.

If anyone has some other suggestions or patches for performance stuff, let me know.

Improving performance for uncompressed files

The reason for this is that our I/O is unbuffered, and we were reading one byte at a time in order to read lines. This changed on December 24, by adding read buffering for reading lines, vastly improving the performance of rred.

But it was still slow, so today I profiled - using gperftools - the rred method running on a 430MB uncompressed Contents file with a 75 KB large patch. I noticed that our ReadLine() method was calling some method which took a long time (google-pprof told me it was some _nss method, but that was wrong [thank you, addr2line]).

After some further look into the code, I noticed that we set the length of the buffer using the length of the line. And whenever we moved some data out of the buffer, we called memmove() to move the remaining data to the front of the buffer.

So, I tried to use a fixed buffer size of 4096 (commit). Now memmove() would spend less time moving memory around inside the buffer. This helped a lot, bringing the run time on my example file down from 46 seconds to about 2 seconds.

Later on, I rewrote the code to not use memmove() at all - opting for start and end variables instead; and increasing the start variable when reading from the buffer (commit).

This in turn further improved things, bringing it down to about 1.6 seconds. We could now increase the buffer size again, without any negative effect.

Effects on apt-get update

I measured the run-time of apt-get update, excluding appstream and apt-file files, for the update from todays 07:52 to the 13:52 dinstall run. Configured sources are unstable and experimental with amd64 and i386 architectures. appstream and apt-file indexes are disabled for testing, so only Packages and Sources indexes are fetched.

The results are impressive:

  * For APT 1.1.6, updating with PDiffs enabled took 41 seconds.
  * For APT 1.1.7, updating with PDiffs enabled took 4 seconds.

That’s a tenfold increase in performance. By the way, running without PDiffs took 20 seconds, so there’s no reason not to use them.

Future work

Writes are still unbuffered, and account for about 75% to 80% of our runtime. That’s an obvious area for improvements.

Performance for patching compressed files

Contents files are usually compressed with gzip, and kept compressed locally because they are about 500 MB uncompressed and only 30MB compressed. I profiled this, and it turns out there is not much we can do about it: The majority of the time is spent inside zlib, mostly combining CRC checksums:

Going forward, I think a reasonable option might be to recompress Contents files using lzo - they will be a bit bigger (50MB instead of 30MB), but lzo is about 6 times as fast (compressing a 430MB Contents file took 1 second instead of 6).

https://people.debian.org/~jak/pubkey.gpg

and the keys.gnupg.net key server. A very short transition statement is available at:

https://people.debian.org/~jak/transition-statement.txt

and included below (the http version might get extended over time if needed).

The key consists of one master key and 3 sub keys (signing, encryption, authentication). The sub keys are stored on an OpenPGP v2 Smartcard. That’s really cool, isn’t it?

Somehow it seems that GnuPG 1.4.18 also works with 4096R keys on this smartcard (I accidentally used it instead of gpg2 and it worked fine), although only GPG 2.0.13 and newer is supposed to work.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA512

Because 1024D keys are not deemed secure enough anymore, I switched to
a 4096R one.

The old key will continue to be valid for some time, but i prefer all
future correspondence to come to the new one.  I would also like this
new key to be re-integrated into the web of trust.  This message is
signed by both keys to certify the transition.

the old key was:

pub   1024D/00823EC2 2007-04-12
      Key fingerprint = D9D9 754A 4BBA 2E7D 0A0A  C024 AC2A 5FFE 0082 3EC2

And the new key is:

pub   4096R/6B031B00 2014-10-14 [expires: 2017-10-13]
      Key fingerprint = AEE1 C8AA AAF0 B768 4019  C546 021B 361B 6B03 1B00

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlQ9j+oACgkQrCpf/gCCPsKskgCgiRn7DoP5RASkaZZjpop9P8aG
zhgAnjHeE8BXvTSkr7hccNb2tZsnqlTaiQIcBAEBCgAGBQJUPY/qAAoJENc8OeVl
gLOGZiMP/1MHubKmA8aGDj8Ow5Uo4lkzp+A89vJqgbm9bjVrfjDHZQIdebYfWrjr
RQzXdbIHnILYnUfYaOHUzMxpBHya3rFu6xbfKesR+jzQf8gxFXoBY7OQVL4Ycyss
4Y++g9m4Lqm+IDyIhhDNY6mtFU9e3CkljI52p/CIqM7eUyBfyRJDRfeh6c40Pfx2
AlNyFe+9JzYG1i3YG96Z8bKiVK5GpvyKWiggo08r3oqGvWyROYY9E4nLM9OJu8EL
GuSNDCRJOhfnegWqKq+BRZUXA2wbTG0f8AxAuetdo6MKmVmHGcHxpIGFHqxO1QhV
VM7VpMj+bxcevJ50BO5kylRrptlUugTaJ6il/o5sfgy1FdXGlgWCsIwmja2Z/fQr
ycnqrtMVVYfln9IwDODItHx3hSwRoHnUxLWq8yY8gyx+//geZ0BROonXVy1YEo9a
PDplOF1HKlaFAHv+Zq8wDWT8Lt1H2EecRFN+hov3+lU74ylnogZLS+bA7tqrjig0
bZfCo7i9Z7ag4GvLWY5PvN4fbws/5Yz9L8I4CnrqCUtzJg4vyA44Kpo8iuQsIrhz
CKDnsoehxS95YjiJcbL0Y63Ed4mkSaibUKfoYObv/k61XmBCNkmNAAuRwzV7d5q2
/w3bSTB0O7FHcCxFDnn+tiLwgiTEQDYAP9nN97uibSUCbf98wl3/
=VRZJ
-----END PGP SIGNATURE-----

Keyboard

I think I like the keyboard better now than I used to when I first tried it. It gets nowhere near the ThinkPad X230 one, though; appart from the coating, which my (backlit) X230 unfortunately does not have.

Screen

While the screen appeared very grainy to me on first sight, having only used IPS screens in the past year, I got used to it over the weekend. I now do not notice much graininess anymore. The contrast still seems extremely poor, the colors are not vivid, and the vertical viewing angles are still a disaster, though.

Battery life

I think the battery life is awesome. I have 30% remaining now while I am writing this blog post and Chrome OS tells me I still have 3 hours and 19 minutes remaining. It could probably still be improved though, I notice that Chrome OS uses 7-14% CPU in idle normally (and up to 20% in exceptional cases).

The maximum power usage I measured using the battery’s internal sensor was about 9.2W, that was with 5 Big Buck Bunny 1080p videos played in parallel. Average power consumption is around 3-5W (up to 6.5 with single video playing), depending on brightness, and use.

Performance

While I do notice a performance difference to my much more high-end Ivy Bridge Core i5 laptop, it turns out to be usable enough to not make me want to throw it at a wall. Things take a bit longer than I am used to, but it is still acceptable.

Input: Software Part

The user interface is great. There are a lot of gestures available for navigating between windows, tabs, and in the history. For example, horizontally swiping with two finger moves in history, three fingers moves between tabs; and swiping down (or up for Australian scrolling) gives an overview of all windows (like expose on Mac, GNOME’s activities, or the multi-tasking thing Maemo used to have).

What I miss is a keyboard shortcut like Meta + Left/Right on GNOME which moves the active window to the left/right side of the screen. That would be very useful for mult-tasking situations.

Issues

I noticed some performance issues. For example, I can easily get the Chromebook to use 85% of a CPU by scrolling on a page with the touchpad or 70% for scrolling by keeping a key pressed (crbug.com/420452).

While watching Big Buck Bunny on YouTube, I noticed some (micro) stuttering in the beginning of the film, as well as each time I move in or out of the video area when not in full-screen mode (crbug.com/420582). It also increases CPU usage to about 70%.

Running a “proper” Linux?

Today, I tried to play around a bit with Debian wheezy and Ubuntu trusty systems, in a chroot for now. I was trying to find out if I can get an accelerated X server with the standard ChromeOS kernel. The short answer is: No. I tried two things:

  1. Debian wheezy with the binaries from ChromeOS (they have the same xserver version)
  2. Ubuntu trusty with the Nvidia drivers

Unfortunately, they did not work. Option 1 failed because ChromeOS uses glibc 2.15 whereas wheezy uses 1.13. Option 2 failed because the sysfs interface is different between the ChromeOS and Linux4Tegra kernels.

I guess I’ll have to wait.

I also tried booting a custom kernel from USB, but given that the u-boot always sets console= and there is no non-verified u-boot available yet, I could not see any output on the screen :(  - Maybe I should build a u-boot myself?

This version cannot be ordered currently, only pre-orders were shipped yesterday (at least here in Germany). I cannot even review it on Amazon (despite having it bought there), as they have not enabled reviews for it yet.

The device feels solidly built, and looks good. It comes in all-white matte plastic and is slightly reminiscent of the old white MacBooks. The keyboard is horrible, there’s no well defined pressure point. It feels like your typing on a pillow. The display is OK, an IPS would be a lot nicer to work with, though. Oh, and it could be brighter. I do not think that using it outside on a sunny day would be a good idea. The speakers are loud and clear compared to my ThinkPad X230.

The performance of the device is about acceptable (unfortunately, I do not have any comparison in this device class). Even when typing this blog post in the visual wordpress editor, I notice some sluggishness. Opening the app launcher or loading the new tab page while music is playing makes the music stop for or skip a few ms (20-50ms if I had to guess). Running a benchmark in parallel or browsing does not usually cause this stuttering, though.

There are still some bugs in Chrome OS:  Loading the Play Books library the first time resulted in some rendering issues. The “Browser” process always consumes at least 10% CPU, even when idling, with no page open; this might cause some of the sluggishness I mentioned above. Also watching Flash videos used more CPU than I expected given that it is hardware accelerated.

Finally, Netflix did not work out of the box, despite the Chromebook shipping with a special Netflix plugin. I always get some unexpected issue-type page. Setting the user agent to Chrome 38 from Windows, thus forcing the use of the EME video player instead of the Netflix plugin, makes it work.

I reported these software issues to Google via Alt+Shift+I. The issues appeared on the current version of the stable channel, 37.0.2062.120.

What’s next? I don’t know.

hardlink 0.3 now features support for xattr support, contributed by Tom Keel at Intel. If this does not work correctly, please blame him.

I also added support for a –minimum-size option.

Most of the other code has been tested since the upload of RC1 to experimental in September 2012.

The next major version will split up the code into multiple files and clean it up a bit. It’s getting a bit long now in a single file.

There are three things that we looked at:

  1. Reducing privileges by setting a new user and group
  2. chroot()
  3. seccomp-bpf sandbox

Today, we implemented the first of them. Starting with 1.1~exp3, the APT directories /var/cache/apt/archives and /var/lib/apt/lists are owned by the “_apt” user (username suggested by pabs). The methods switch to that user shortly after the start. The only methods doing this right now are: copy, ftp, gpgv, gzip, http, https.

If privileges cannot be dropped, the methods will fail to start. No fetching will be possible at all.

Known issues:

• We drop all groups except the primary gid of the user
• copy breaks if that group has no read access to the files

We plan to also add chroot() and seccomp sandboxing later on; to reduce the attack surface on untrusted files and protocol parsing.

Basic overview: You configure OpenWRT into station mode (that is, as a WiFi client) and use relayd to relay between the WiFi network and your local network. You also need igmpproxy to proxy multicast packages between those networks, other UPnP stuff won’t work.

I did this on the recent Barrier Braker RC2. It should work on older versions as well, but I cannot promise it (I did not get igmpproxy to work in Attitude Adjustment, but that was probably my fault).

Note: I don’t know if it works with IPv6, I only use IPv4.

You might want to re-start (or start) services after the steps, or reboot the router afterwards.

Configuring WiFi connection to the FRITZ!Box

Add to: /etc/config/network

config interface 'wwan'
	option proto 'dhcp'

(you can use any other name you want instead of wwan, and a static IP. This will be your uplink to the Fritz!Box)

Replace wifi-iface in: /etc/config/wireless:

config wifi-iface
	option device 'radio0'
	option mode 'sta'
	option ssid 'FRITZ!Box 7270'
	option encryption 'psk2'
	option key 'PASSWORD'
	option network 'wwan'

(adjust values as needed for your network)

Setting up the pseudo-bridge

First, add wwan to the list of networks in the lan zone in the firewall. Then add a forward rule for the lan network (not sure if needed). Afterwards, configure a new stabridge network and disable the built-in DHCP server.

Diff for /etc/config/firewall

@@ -10,2 +10,3 @@ config zone
 	list network 'lan'
+	list network 'wwan'
 	option input 'ACCEPT'
@@ -28,2 +29,7 @@ config forwarding
 
+# Not sure if actually needed
+config forwarding
+	option src 'lan'
+	option dest 'lan'
+
 config rule

Add to /etc/config/network

config interface 'stabridge'
	option proto 'relay'
	option network 'lan wwan'
	option ipaddr '192.168.178.26'

(Replace 192.168.178.26 with the IP address your OpenWRT router was given by the Fritz!Box on wlan0)

Also make sure to ignore dhcp on the lan interface, as the DHCP server of the FRITZ!Box will be used:

Diff for /etc/config/dhcp

@@ -24,2 +24,3 @@ config dhcp 'lan'
        option ra 'server'
+       option ignore '1'

Proxying multicast packages

For proxying multicast packages, we need to install igmpproxy and configure it:

Add to: /etc/config/firewall

# Configuration for igmpproxy
config rule
	option src      lan
	option proto    igmp
	option target   ACCEPT

config rule
	option src      lan
	option proto    udp
	option dest     lan
	option target   ACCEPT

(OpenWRT wiki gives a different 2nd rule now, but this is the one I currently use)

Replace /etc/config/igmpproxy with:

config igmpproxy
	option quickleave 1

config phyint
	option network wwan
	option direction upstream
	list altnet 192.168.178.0/24

config phyint
	option network lan
	option direction downstream
	list altnet 192.168.178.0/24

(Assuming Fritz!Box uses the 192.168.178.0/24 network)

Don’t forget to enable the igmpproxy script:

# /etc/init.d/igmpproxy enable

Optional: Repeat the WiFi signal

If you want to repeat your WiFi signal, all you need to do is add a second wifi-iface to your /etc/config/wireless.

config wifi-iface
	option device 'radio0'
	option mode 'ap'
	option network 'lan'
	option encryption 'psk2+tkip+ccmp'
	option key 'PASSWORD'
	option ssid 'YourForwardingSSID'

Known Issues

If I was connected via WiFi to the OpenWRT AP and switch to the FRITZ!Box AP, I cannot connect to the OpenWRT router for some time.

The igmpproxy tool writes to the log about changing routes.

Future Work

I’ll try to get the FRITZ!Box replaced by something that runs OpenWRT as well, and then use OpenWRT’s WDS support for repeating; because the FRITZ!Box 7270v2 is largely crap - loading a page in its web frontend takes 5 (idle) - 20 seconds (1 download), and it’s WiFi speed is limited to about 20 Mbit/s in WiFi-n mode (2.4 GHz (or 5 GHz, does not matter), 40 MHz channel). It seems the 7270 has a really slow CPU.

I rebooted my machine, and saw no logo appearing. Just something like an underscore on a text console. The system appears to boot normally otherwise, and once the i915 module is loaded (and we’re switching away from UEFI’s Graphical Output Protocol [GOP]) the screen works correctly.

So it seems the GOP broke.

What should I do next?

As part of that change, python-apt now only supports Python 2.7, Python 3.3, and newer. I’m using some features only present in 3.3 like Python2 unicode literal syntax in order to keep the code simple.

Here’s how I did it:

I took the Python 2 code and ran 2to3 -f print -x future on it. This turned every print statement in a call to the print function. I then went ahead and added a “from future import print_function” to the top of each module. This was the first commit.

For the second commit, I ran 2to3 -p -x future to convert the remaining stuff to Python 3, and then undid some changes (like unicode literals) and fixed the rest of the changes to work under both Python 2 and 3. Sometimes I added a top-level code like:

if sys.version_info_major >= 3:
    unicode = str

So I could use unicode in the code for the Python 2 cases.

I used various backported modules like io and stuff only available in Python 2.7, so dropped support for Python 2.6.

I put a fresh installation of unstable on it and tweaked it to save more power when on battery (enabled RC6++, and enabled autosuspend and other powertop suggestions with a script in /etc/pm/power.d); and configured hdparm.conf to put my hard disk into standby after 5 seconds (it’s only used for big data anyway, so most of the time it is unused). It now consumes 5W in idle with minimum brightness, and 8-10W with brightness 13 of 15. Consumption when surfing is 10 - 15 watts. Booting from grub to gdm is fast, I did not measure it, but it probably took about 5 seconds.

The IPS screen looks really good. Much much much better than the screen in my 2010 Edge 15 (I reviewed that one in 2010). It seems a bit more glossy than that one, but still matte. Keyboard is good as well. The touch pad is crap however. All hardware seems to work.

Comparison to the X240 for others who think about buying one of them: The X240 is lighter and thinner (it’s an Ultrabook) and has an optional FullHD 12.5 inch screen. It also offers a much larger touchpad and palm rest. But compared to the X230, the X240 lacks many things: No dedicated buttons for the trackpoint (you need to use the click-pad), it’s limited at 8GB RAM, uses a slower low-voltage Haswell CPU, and it uses the new M.2 connector (probably supporting only the shortest cards) instead of mini-PCIe, so it’s not really possible to add an additional SSD currently; as M.2 SSDs do not seem to be available yet. I also do not know whether the X240 offers ThinkLight or LEDs for hard drive activity and similar.

The rewrite is sadly not 100% backwards compatible to the original version; as some option names had to be renamed due to our command-line parser not supporting option names like -nh, and some other options were dropped because they are hard to support (like –status-file and –lists-dir) with our command-line parsing. I also decided not to keep the the -p and -r options, but use the standard APT command-line conventions insteads.

For now, it also cannot show you the distribution names you have specified in your sources.list file, but will always display codenames instead; if available. I hope to fix this in Jessie by extending APT’s cache format a bit.

On the performance side, this program now takes about 0.09s compared to the 1.40 seconds needed by apt-show-versions. The times are taken with all data in caches.

The current version can be found in a git repository, a link to gitweb is:

http://anonscm.debian.org/gitweb/?p=users/jak/apt-show-versions.git

Please also note that support for –allversions is not 100% implemented yet, but it should work for most uses.

Now, go testing and report back!

We should thus write our parser without real recursion. Fortunately, Python offers us a way out: Coroutines, introduced in Python 2.5 as per PEP 342. Using coroutines and a simple trampoline function, we can convert every mutually recursive set of functions into a set of coroutines that require constant stack space.

Let’s say our parser looks like this (tokens being an iterator over characters):

def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        return token
    elif token == '(':
         result = parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...

We now apply the following transformations: return X => yield (X) parse() => (yield parse())

That is, we yield the value we want to return, and we yield a generator whenever we want to call (using a yield expression). Our rewritten parser reads:

def parse(tokens):
    token = next(tokens)
    if token.isalnum():
        yield token
    elif token == '(':
         result = yield parse(tokens)
         if next(tokens) != ')': raise ...
         return result
    else:
         raise ...

We obviously cannot call that generator like the previous function. What we need to introduce is a trampoline. A trampoline is a function that manages that yielded generators, calls them, and passes their result upwards. It looks like this:

def trampoline(generator):
    """Execute the generator using trampolining. """
    queue = collections.deque()

    def schedule(coroutine, stack=(), value=None):
        def resume():
            if 0:
                global prev
                now = stack_len(stack)
                if now < prev:
                    print("Length", now)
                    prev = -1
                elif prev != -1:
                    prev = now
            result = coroutine.send(value)
            if isinstance(result, types.GeneratorType):     # Normal call
                schedule(result, (coroutine, stack))
            elif isinstance(result, Tail):                  # Tail call (if you want to)
                schedule(result.value, stack)
            elif stack:                                     # Return to parent
                schedule(stack[0], stack[1], result)
            else:                                           # Final Return
                return result

        queue.append(resume)

    schedule(generator)

    result = None
    while queue:
        func = queue.popleft()
        result = func()

    return result

This function is based on the code in PEP 342, the difference being that

• we do not correctly propagate exceptions through the stack, but directly unwind to the caller of the parser (we don’t handle exceptions inside our parser generators anyway)
• the code actually compiles (code in PEP used ‘value = coroutine.send(value)’ which does not work)
• the code returns a value (code in PEP was missing a return in schedule)
• we don’t use a class, and allow only one function to be scheduled at once (yes, we could get rid of the queue)
• we allow tail calls [where Tail = namedtuple(“Tail”, [“result”])] to save some more memory.

For a more generic version of that, you might want to re-add exception passing, but the exceptions will then have long tracebacks, so I’m not sure how well they will work if you have deep recursion.

Now, the advantage of that is that our parser now requires constant stack space, because the actual real stack is stored in the heap using tuples which are used like singly-linked lists in scheme here. So, the only recursion limit is available memory.

A disadvantage of that transformation is that the code will run slightly slower for small inputs that could be handled using a normally recursive parser.

Code is provided at http://anonscm.debian.org/gitweb/?p=users/jak/cleanup.git, under the MPL 2.0. You need to have python-apt and clasp installed to use it. There is potential minisat+ support, but it’s currently a bit broken.

To use, run python program_builder.py, and it will tell you which packages are no longer needed on your system. It ignores Suggests, if you want those in, you have to hack the code and replace {“Recommends”} by {“Recommends”, “Suggests”}. You can also turn of such dependencies by setting Program.hard_softdeps to False.

def solve_or(or):
  best_real = None
  best_virtual = None
  for dep in or:
     for target in dep:
        if target.name == dep.name and best_real is None:
           best_real = target
        if target.name != dep.name and best_virtual is None:
           best_virtual = target        
        if target.is_installed():
          return target

  return best_real if best_real is not None else best_virtual

Now, this way of solving dependencies is slightly problematic. Let us consider a package that depends on: a | b, b. APT will likely choose to install ‘a’ to satisfy the first dependency and ‘b’ to satisfy the second. I currently have draft code around for a future version of APT that will cause it to later on revert unneeded changes, which means that APT will then only install ‘b’. This result closely matches the CUDF solvers and cupt’s solver.

On the topic of solving algorithms, we also have the problem that optimizing solvers like the ones used with apt-cudf do not respect the order of dependencies, rather choosing to minimise the number of packages installed. This causes such a solver to often do stuff like selecting an sqlite database as backend for some service rather then a larger SQL server, as that installs fewer packages.

To make such solvers aware of the implicit preferences, we can introduce a new type of dependency category: Weak conflicts, also known as Recommends-Not. If a package P defines a Recommends-Not dependency against a package Q, then this means that Q should not be installed if P is installed. Now, if we have a dependency like:

Depends: a | b | c

we can encode this as:

Recommends-Not: c, c, b

Causing the solver to prefer a, then b, and then c. This should be representable as a pseudo-boolean optimization problem, as is common for the dependency problem, although I have not looked at that yet – it should work by taking the standard representation of conflicts, adding a relaxation variable and then minimising [or maximising] the number of relaxation variables.

It turns out the glibc’s memory allocator is relatively slow, especially if you do loops that create one element and destroy one element at the same time (for example, map() on a list you no longer use after you passed it to map). To fix this problem, I considered various options.

The first option was to add a thread-local cache around malloc(). So whenever we want to free() an object, we place it on a thread-local list instead, and if a malloc() request for an object of that size comes in, we reuse the old object.

This fixes the problem with the lists, but experiments shown another problem with the glibc allocator: Allocating many objects without releasing others (let’s say appending an element to a functional list). I started testing with tcmalloc instead, and noticed that it was several times faster (reducing run time from 6.75 seconds to 1.33 seconds (5 times faster)). As I do not want to depend on a C++ code base, I decided to write a simple allocator that allocates blocks of memory via mmap(), splits those into objects, and puts the objects on a local free list. That allocator performed faster than tcmalloc(), but was also just a simple test case, and not really useable, due to potential fragmentation problems (and because the code was statically linked in, causing thread-local storage to be considerably faster, as it does not need to call a function on every TLS access, but can rather go through a segment register).

I then discovered jemalloc, and tried testing with this. It turned out that jemalloc was even faster than tcmalloc in all test cases, and also required about the same amount of memory as tcmalloc (and slightly more than the custom test code, as that combined reference counting with memory allocator metadata and thus had less meta data overhead). Using jemalloc, the list building test performs in 0.9X seconds now (instead of tcmalloc’s 1.33 or glibc’s 6.75 seconds), requires 0 major page faults (tcmalloc has 2), and uses 5361472 kb memory (tcmalloc uses 5290240, glibc requires 7866608).

Given that jemalloc is written in C, I can only recommend it to everyone in need of a scalable memory allocator. Depending on your workload, it might require less memory than tcmalloc (at least that’s the case at Facebook) and is most likely faster than tcmalloc. It also provides tcmalloc-compatible profiling facilities (as Facebook needed them). Furthermore, jemalloc is also the memory allocator used by FreeBSD, and is used by Mozilla projects, and on several Facebook projects, and should thus be relatively stable and useable.

All tests were run with 4 threads, on a dual-core laptop with hyper-threading, using (e)glibc 2.13, tcmalloc 2.0, and jemalloc 3.0. The tests may not be representative of real world performance, and do not account for fragmentation.

Should I try replacing Python’s custom caching of malloc() calls with simply calling jemalloc, and run a few Python benchmarks? Anyone interested in that?

function f(x) { return g(x+1); }

Now consider that x is a reference counted object and that x + 1 creates a new object. The call to g(x + 1) shall be in tail call position.

In most reference counted languages, the reference to an argument is owned by the caller. That is, f() owns the reference to x + 1. In that case, the call to g() would no longer be in a tail call position, as we still have to decrease the reference count after g() exits.

An alternative would be that the callee owns the reference. This however, will most likely create far more reference count changes than a caller-owns language (increase reference count in caller, decrease reference count in callee). For example, the following function requires an increment before the call to g().

function f1(x) { return g(x); }

Does anyone have any ideas on how to solve the problem of tail calls while avoiding the callee-owns scenario? Something that does not require a (non-reference-counting) garbage collector would be preferably.

One of the questions currently unaddressed is how to handle currying in the presence of mutable parameters. In order to visualise this problem, consider a function

    printLn(mutable IO, String line)

If we bind the the first parameter, what should be the type of the function, and especially important how do we get back the mutable parameter? Consider the partially applied form printLn1:

    printLn1(String line)

The mutability parameter would be lost and we could not do any further I/O (and the curried form would appear as a pure function, so a good compiler would not even emit a call to it) An answer might be to put the thing into the result when currying:

    printLn2(String line) -> mutable IO

But how can we explain this? In the end result, do we maybe have to use a signature like:

    printLn(String, mutable in IO io1, mutable out IO io2)

We could then introduce syntax to call that as

    printLn(string, mutable io)

Where as the “mutable io” argument basically expands to “io” and “io1” for the first call, and for later calls to “io1, io2”, and so on. It can also be easily curried by allowing currying to take place on variables not declared as output parameters. We can then curry the function as:

    printLn3(mutable in IO io1, mutable out IO io2)
    printLn4(mutable out IO io2)

If so, we can rename mutable back to unique and make that easier by introducing the unary operator & for two locations, just like Mercury uses ! for it. We could then write calls looking like this:

    printLn("Hello", &io);
    printLn("Hello", io, io1);

How out parameters are explained is a different topic; we could probably say that an out parameter defines a new variable.

Another option is to forbid currying of mutable parameters entirely. This would have the advantage of maintaining the somewhat simple one parameter style.

The programming language Clean does not provide any special syntactic sugar for having mutable variables. In Clean, the function gets a unique object and returns a unique object (noted by *). For example, the main entry point in a Clean program (with state) looks like this:

    Start:: *World -> *World

In short, the function Start gets a abstract world passed that is unique and at the end returns a new unique world. In Clean syntax, our example function would most likely have the signature:

    printLn :: String *IO -> *IO

You know have to either maintain one additional variable for the new unique object, which gets a bit complicated with time. On the other hand, you can do function composition on this (if you have a function composition operator that preserves uniqueness when available, as should be possible in Clean):

    printTwoLines :: *IO -> *IO 
    printTwoLines = (printLn "World") o (printLn "Hello")

Function composition on mutable things however, does not seem like it is needed often enough in a functional programming language with a C syntax.

People might also ask why monads are not used instead. Simon L Peyton Jones and Philip Wadler described monadic I/O in their paper “Imperative Functional Programming” (http://research.microsoft.com/en-us/um/people/simonpj/papers/imperative.ps.Z) in 1993, and it is used in Haskell (the paper was about the implementation in Haskell anyway), one of the world’s most popular and successful functional programming languages.

While monadic I/O works for the Haskell crowd, and surely some other people, the use of Monads also limits the expressiveness of code, at least as far as I can tell. At least as soon as you want to combine multiple monads, you will have to start lifting stuff from one monad to another (liftIO and friends), or perform all operations in the IO monad, which prevents obvious optimizations (such as parallelizing the creation of two arrays) – in short dependencies between actions are more strict than they have to be. For a functional language targeting imperative programmers, the lifting part seems a bit too complicated.

One of the big advantages of monads is that they are much easier to implement, as they do not require extensions to the type system and the Hindley-Milner type inference algorithm used by the cool functional programming languages. If you want uniqueness typing however, you need to modify the algorithm or infer the basic types first and then infer uniqueness in a second pass (as Clean seems to do it).

module main:
    import std (range)
    import std.io (printf, IO)

    # print the Fahrenheit-Celcius table for fahr = 0, 20, ..., 300
    function main(mutable IO io):
        Int lower = 0    # lower bound
        Int upper = 300  # upper bound
        Int step = 20    # step
        for Int fahr in range(lower, upper, step):
            Double celcius = 5 * (fahr - 32) / 9
            std.io.printf(io, "%3d\t%6.1f\n", fahr, celcius)

It does not really look like it, but this language is purely functional. It represents side effects using unique types. If you declare a mutable parameter, you basically declare a unique input parameter and a unique output parameter.

I’m also giving you a list implementation

module std.container.list:

    ## The standard singly-linked list type
    type List[E]:
        Nil                     ## empty list
        Node:
            E value             ## current value
            List[E] next        ## remaining list

_ _And yes, both languages should be able to be represented using the same abstract syntax tree. The only change is the replacement of the opening curly brace by a colon, the removal of the closing curly bracket and semicolons, the replacement of C-style comments with Python-style comments and the requirement of indentation; oh and the for statement gets a bit lighter as well.

module main {
    import std (range);
    import std.io (printf, IO);
 
    /* print the Fahrenheit-Celcius table
        for fahr = 0, 20, ..., 300 */
    function main(mutable IO io) {
        Int lower = 0;   // lower bound
        Int upper = 300; // upper bound
        Int step = 20;   // step
        for (Int fahr in range(lower, upper, step)) {
            Double celcius = 5 * (fahr - 32) / 9;
            std.io.printf(io, "%3d\t%6.1f\n", fahr, celcius);
        }
    }
}

It does not really look like it, but this language is purely functional. It represents side effects using unique types. If you declare a mutable parameter, you basically declare a unique input parameter and a unique output parameter.

I’m also giving you a list implementation

module std.container.list {

    /** The standard singly-linked list type */
    type List[E] {
        Nil;                    /** empty list */
        Node {
            E value;            /** current value */
            List[E] next;       /** remaining list */
        }
    }
}

Thus are only excerpts from a document with tens of pages and the reference implementation of the standard library. The incomplete working draft for the language is attached: JAK Programming Language Early Working Draft (28 pages).

Update: Fixed the link.

The code should be portable to all UNIX-like platforms supporting nftw(). I have tested the code on Debian, FreeBSD 9, and Minix 3.2. For storing path names, it uses a flexible array member on C99 compilers, a zero-length-array on GNU compilers, and a array of length 1 on all other compilers (which should work everywhere as far as I heard).

The new version may have slightly different behaviour compared to 0.1.2 when regular expressions are specified on the command-line, as a result of the switch from Python’s regular expression engine to PCRE (if compiled with PCRE support, you can also use POSIX extended regular expressions if you like).

The code is significantly faster than the old code (in situations where it’s not I/O bound), and uses much less memory than the old one. Per file, we now store a struct stat, a pointer, an int, a char, and the filename; as well as three pointers for a binary search tree (which uses tsearch()).

It should also be compatible to Red Hat’s original hardlink tool now command-line-wise, there is at least a -c option now. The history and the name conflict are interesting, but probably nothing for this post. We’re even less resistant against changing trees than Fedora’s tool (and derived) currently, but should otherwise be better (and far more complicated and feature-packed). And we don’t require mmap(), but use fread() instead. There was no real performance difference in testing. And we are not GPL-licensed, but use the MIT/Expat license.

The package is currently entering Debian experimental for testing purposes. If you have used hardlink previously, or are just curious, give it try.

And the makefile is now compatible with the various BSD makes out there, if that’s interesting for you.

Link to website: http://jak-linux.org/projects/hardlink/

jak-standard

Standard packages for all systems

jak-desktop

Standard packages for all desktop systems (GNOME 3 if possible, otherwise GNOME 2)

jak-printing

Print support

jak-devel

Development packages

jak-machine-<X>

The meta package defining the computer X

Each computer has a jak-machine-X package installed. This package is marked as manually installed, all other packages are marked as automatically installed.

The machine packages have the attribute XB-Important: yes set in debian/control. This creates an Important: yes field. This field is not official, but APT recognizes it and does not remove those packages (the same field is set for the APT package by APT when building the cache, as APT should not be removed either by APT). It seems to work a bit like Essential, with the exception that non-installed packages are not installed automatically on dist-upgrade.

The meta packages are created using seed files similar to Ubuntu. In contrast to Ubuntu, I’m not using germinate to create the packages from the seeds, but a custom dh_germinate_lite that simply takes a seed file and creates the correct substvars. It’s faster than germinate and really simplistic. It also does not handle Recommends currently.

The whole result can be seen on http://anonscm.debian.org/gitweb/?p=users/jak/jak-meta.git. Maybe that’s useful for some people. And if you happen to find some packages in the seeds that are deprecated, please let me know. Oh, and yes, some packages (such as the letterman one) are internal software not publically available yet [letterman is a simple GUI for creating letters using LaTeX].

While I’m at it, I also built Ubuntu’s version of wine1.2 for i386 squeeze. It can be found in deb [http://people.debian.org/~jak/debian/](http://people.debian.org/~jak/debian/) squeeze main (it still needs a few changes to be correct though, I’ll upload a jak2 build soon). I also built updated sun-java6 packages for my parents (mostly needed due to the plugin, some websites do not work with the IcedTea one), but can’t share the binaries due to licensing requirements. I may push out a source repository, though, so others can build those packages themselves. I’ll let you know once that’s done.

I promise that I will not sell them to others. I’m interested in WebOS, in running Debian and/or Ubuntu on those devices (for the extra fun factor), and lend it to family members for surfing, etc.

I also take other tablets and smart phones and various kinds of ARM and PowerPC hardware (I guess that’s all that’s interesting for me) for free, just send me an email if you have some and want to give them to me. This applies to Intel stuff as well, I’d really like to get some kind of WeTab/ExoPC, but can’t buy one currently (and they’re probably to outdated hardware-wise for buying to make sense).

The reason: Space travel. If we take the term worldwide to mean “everywhere on earth”, the license becomes non-free, as it prohibits the use outside of this planet. Affected by this problem are the patent section of GPL-3, the Apache 2.0 license, the CC licenses, the GFDL, and probably also others.

Now what should be used instead? Universal? No, that wouldn’t work in case there are multiple ones (while travelling between them (if they exist) could be impossible, it would still be a restriction). The correct team would probably be “omniversal” meaning “everywhere in the omniverse”. But really, avoiding locations is probably the best way.

In any case, if you have received software from me under a license that uses the term “worldwide”, you can treat worldwide as everywhere, and are thus free to use it outside of earth (and other planets).

For this new feature, run dh_autoreconf with the –as-needed option. dh_autoreconf will then patch all ltmain.sh equal to the system one (which should be all ltmain.sh files if libtoolize ran before or via dh_autoreconf). On clean, dh_autoreconf_clean reverses the patch again.

So, if your package runs autoreconf, and patches ltmain.sh via a patch you can now do this automatically via dh-autoreconf and be future-proof.

The only problem is that this might break once the patch no longer applies to libtool, at which point I need to update the package to include an updated patch. A solution for this problem would be to include the patch in libtool itself, as I proposed in Bug#347650.

In case this works well, the option could also become the default which would make things even easier.

I also bought a Toshiba AC100 before my birthday, a Tegra 2 based notebook/netbook/“web companion” with 1 GHz dual core ARM Cortex A9 chip and 512 MB RAM. It runs Android by default, and had a price of 160€ which is low compared to anything else with Cortex A9. It currently runs Ubuntu 11.04 with a specialised kernel 2.6.37 from time to time, without sound and accelerated video (and not functioning HDMI). Mostly waiting for Nvidia to release a new binary blob for the video part (And yes, if you just want to build packages, you can probably get happy without those things).

Another thing happening last week is the upload of python-apt 0.8.0 to unstable, marking the beginning (or end) of the API transition I started more than a year ago. Almost all packages not supporting it have proper Breaks in python-apt [most of them already fixed, only 2 packages remaining, one of which is “maintained” (well, not really maintained right now) by me], but there may be some which do not work correctly despite being fixed (or at least thought to be fixed).

If you know any other interesting thing I did last week, leave a comment, I wrote enough now. And yes, Wordpress wants to write a multiplication sign instead of an x, so I had to use &#120 instead.

First of all, I dropped the GVariant-based cache. The format strings were simply getting ugly long and were not very understandable, performance was just much too slow (needing more than a few nanoseconds for a package lookup is obviously too slow for solving dependency problems); furthermore, building the cache was also slow and complicated because we needed all attributes of an object at once to pass them to GVariant, leading to ugly API.

I replaced the GVariant cache with one that can be easily mmap()ed and is described completely in C. It’s derived from APT’s cache design (but more robust, as it includes the size of the cache and we can thus detect to small files, although that’s scheduled for the next ABI break in APT as well), but has fewer duplicate data, and uses arrays where APT uses linked lists. The reason for arrays is simple: They take up less space and can be represented naturally in Python and other languages using array-based lists. The cache also contains a coalesced hash table which does use a linked list, but that one is a bit different, as it is for searching only and not exposed. Everything non-stringy is 64-bit aligned in order to keep things as simple as possible. All integers are fixed size, thus the format is architecture-independent if you fix byte orders. The format is described at http://people.debian.org/~jak/apt2-doc/apt-Cache-Format.html.

I stole one more idea from cupt and changed the configuration system to verify types of variables. APT2’s configuration system knows more types than cupt’s, though, including regular expressions, directory and filenames (i.e. it does not let you store a value /d/ in a file variable), strings (which store everything), unsigned and signed integers, and boolean options; all of which are checked when parsing files (producing warnings) or command-line options (producing errors).

I have also simplified the type world by removing all iterator types except for one, replacing them with get_thing() and n_things() functions in the objects holding the arrays. Makes cool bindings slightly harder, but makes the C API much easier to use from C.

Most things expected from a package manager are still missing, but what is there looks good in most cases (especially AptConfiguration has a nice API, and no complaints from valgrind anywhere). Currently I am working on Python bindings so I can interact with the functions easily and check things in an interactive fashion; and I am also writing a document explaining the concepts behind APT2, drafts at http://people.debian.org/~jak/midlevel.pdf. I also have some more code pending further thoughts (including complete index parsing), but it might still take some time before I have something usable in the wild.

On other package managers: From time to time I also use Cupt, look at Cupt code, hack Cupt code, and report bugs against Cupt. I still do not really understand the (extreme) nesting of directory structures in the source code, why there are so (extremely) many source files split all over them, or the general concepts of Cupt, but I can hack together what I need for my personal testing. I also play with yum whenever I end up on a Fedora system (which happens from time to time).

Yet, they are widely used everywhere. Here are some examples:

  * inclusion guards in GLib: ` __G_VARIANT_H__`
      * internal Python functions: `_PyUnicode_AsString`
      * various macros in APT: `__deprecated`, `__hot`

All of this triggers undefined behavior and is thus uncool. Of course in APT, it’s most stupid, as we do not have any namespace and could thus end up redefining things we should not much more likely then the other two.

But why were those solutions chosen in the first place, and what is the alternative? I cannot answer the first question, but for the second one, the obvious alternative is to use trailing underscores:

  * inclusion guards, defined behavior: ` G_VARIANT_H__`
      * internal functions, defined behavior: `PyUnicode_AsString_`
      * various macros, defined behavior: `deprecated__`, `hot__`

Then there is another class of reserved identifiers with underscores:

Meaning that everything except for parameters, local variables and members of structs/unions that starts with an underscore is reserved. So, if you happen to create a variable _mylibrary_debug_flag, you trigger undefined behavior as well. And while we’re at it, do not think you can create a type ending in _t: POSIX reserves all identifiers ending in _t for its own use.

In summary, whenever you write C and want to be 100% safe of undefined-behavior-because-of-naming, do not start any identifier with an underscore and do not end any identifier with _t.

In #debian-devel, some people (including me and others on the Debian side; and sladen, sabdfl for the Ubuntu side) discussed the Ubuntu font license which is considered non-free by Debian, due to extreme naming restrictions in section 2 (unmodified versions must keep the name, slightly modified versions must keep the name and add something). Some consider those restrictions equivalent to invariant sections. After we discusses the font license, we quickly got to discuss Doctor Who and time travel, as those two are obviously connected.

Some other things happened as well, like closing more bugs, but all in all, the last two weeks where a bit less intensive than the two weeks before them.

On Tuesday, I uploaded python-apt 0.8.0~exp2 to experimental, fixing about 10 bugs reported in Ubuntu and Debian bug trackers. It should know even convert integers correctly on all architectures, previously we could have passed long via varargs where int was expected.

Bugs

Until Thursday, I went through the bug list in Launchpad and closed/fixed/reassigned/merged about 100 bugs in APT and python-apt.

APT & python-apt updates for squeeze

Today, I uploaded updates of apt and python-apt to stable. They include support for xz and parsing multi-arch dependencies, as wanted by ftpmasters.

APT 0.8.14 and wildcards/regular expression pinning

Today, I uploaded apt 0.8.14 to Debian unstable, introducing support for pinning using glob() like Syntax and POSIX extended regular expressions. Let’s say we want to pin all packages starting with gnome or kde to 990. The following example does this, using glob-like patterns for gnome, and a regular expression enclosed in / for kde: Package: gnome* /^kde/ Pin: release a=experimental Pin-Priority: 990

This closes 10-year-old Bug#121132 in Debian. Have fun with this feature, but please note that it may not be the fastest thing on earth, as it checks every package in the cache on initialization of such queries, which may take a few 10 ms.

Since some time already, it’s also possible to use such expressions for the Pin field. Thus users of Ubuntu releases could use the following piece of preferences to pin all packages in archives starting with lucid (e.g. lucid, lucid-updates) to 990: Package: * Pin: release a=lucid* Pin-Priority: 990

Those types of pins do not have the negative performance impact of complex expressions in the Package header, as they are only checked against a smaller set of packages, or if “Package: *”, simply checked against the package files in the cache.

This week was a rather busy week. I’m currently doing a (unpaid) 1 month internship as part of my education. Thanks to Michael Vogt and his boss at Canonical Ltd, this internship takes place in IRC and is dedicated to Debian and Ubuntu stuff, primarily APT-related things.

The first two days were spent on multi-arch support in python-apt: On Monday, I released python-apt 0.7.100.3, introducing initial minimal multi-arch support (just enough to not break anymore, but no really new multi-arch-specific API). This release is also the base for the version going to be shipped in Ubuntu natty, which is one of the reasons to keep the changes such minimal. I also fixed an RC bug related to Python 3.2 modules in python-apt, and implemented nocheck build option and disabled test errors on hurd.

On Tuesday, I released python-apt 0.8.0~exp1 to experimental. This release now has the old-style non-PEP8 API disabled and also introduces improved multi-arch support, by introducing bindings for APT’s GrpIterator class, and supporting indexing the cache by (name, architecture) tuples.

On Wednesday, I noticed a strange bug in APT (via python-apt’s test suite) where what the cache considered the native architecture was not the configured one. David Kalnischkies and I debugged the problem, and he found the source of the problem and implemented a fix in his branch of APT. I also introduced multi-arch support for the aptsources module, fixed all Python 3.2 ResourceWarnings in python-apt, and prepared an NMU for python-debian, to adjust it to python-apt’s new API. I also took over maintenance of software-properties in Debian, and did two uploads there (rebased on the Ubuntu package), both with python-apt 0.8 API support.

On Thursday, I shifted a bit more to the Ubuntu side and fixed several bugs in APT and aptdaemon, resulting in the aptdaemon 0.41+bzr614-0ubuntu2 upload and apt 0.8.13.3ubuntu2. I also fixed software-properties KDE version in Debian, as I broke it the previous day.

Today, on Friday, I fixed one more bug in APT. APT now treats Release files that cannot be verified identical to Release files without signature, that is, they are actually parsed now (no more missing Origin fields) - see LP: #704595.

dh-autoreconf 3

I uploaded dh-autoreconf 3, fixing all bugs in the BTS except for one (if someone knows why autopoint depends on git, please tell me, and I may fix this bug as well). For those who don’t know dh-autoreconf, it is a tool to run autoreconf automatically during the package build, so no need for manual cleanup or autoreconf patches.

I now thought about adding the option to automatically patch ltmain.sh to dh-autoreconf. As many know, ltmain.sh does not work correctly with -Wl,–as-needed. Now, if the libtool maintainer cooperates and provides a patch file in the libtool binary package, dh-autoreconf could automatically apply it during build-time, thus fixing this problem as well.

GNOME 3

I’m now running GNOME 3, or the parts of it we have in Debian.

Next week

We’ll probably see python-apt 0.8.0~exp2 next week with more improved multi-arch support and other fixes.

One thing I had problems with was the time required to install build dependencies due to disk I/O. Given that I have 4G RAM in my computer, I decided to use a tmpfs as the writable overlay. For those of you who want to do the same, putting the following in /etc/fstab makes it work:

overlay                     /var/lib/schroot/union/overlay tmpfs         defaults                    0       0

Build times are much faster this way because all write I/O is directed to the RAM

<span style="color:#008200;">#include <stdio.h></span>

<span style="color:#008200;">#define len 1000000000L</span>

<span style="color:#830000;">unsigned long</span> <span style="color:#010181;">f</span><span style="color:#000000;">(</span><span style="color:#830000;">unsigned long</span> a<span style="color:#000000;">,</span> <span style="color:#830000;">unsigned long</span> b<span style="color:#000000;">)</span> <span style="color:#010181;">__attribute__</span><span style="color:#000000;">((</span>noinline<span style="color:#000000;">));</span>

<span style="color:#830000;">int</span> <span style="color:#010181;">main</span><span style="color:#000000;">()</span>
<span style="color:#000000;">{</span>    
    <span style="color:#010181;">printf</span><span style="color:#000000;">(</span><span style="color:#ff0000;">"%lu</span><span style="color:#ff00ff;">\n</span><span style="color:#ff0000;">"</span><span style="color:#000000;">,</span> <span style="color:#010181;">f</span><span style="color:#000000;">(</span><span style="color:#2928ff;">0</span><span style="color:#000000;">,</span> <span style="color:#2928ff;">2</span><span style="color:#000000;">*</span>len<span style="color:#000000;">));</span>
    <span style="color:#000000;font-weight:bold;">return</span> <span style="color:#2928ff;">0</span><span style="color:#000000;">;</span>
<span style="color:#000000;">}</span>

<span style="color:#830000;">unsigned long</span> <span style="color:#010181;">f</span><span style="color:#000000;">(</span><span style="color:#830000;">unsigned long</span> a<span style="color:#000000;">,</span> <span style="color:#830000;">unsigned long</span> b<span style="color:#000000;">)</span>
<span style="color:#000000;">{</span>
    <span style="color:#830000;">unsigned long</span> sum <span style="color:#000000;">=</span> <span style="color:#2928ff;">0</span><span style="color:#000000;">;</span>
    <span style="color:#000000;font-weight:bold;">for</span> <span style="color:#000000;">(;</span> a <span style="color:#000000;"><</span> b<span style="color:#000000;">;</span> a<span style="color:#000000;">++)</span>
        sum <span style="color:#000000;">+=</span> a<span style="color:#000000;">;</span>
    <span style="color:#000000;font-weight:bold;">return</span> sum<span style="color:#000000;">;</span>
<span style="color:#000000;">}</span>

Now, I would be interested to see what’s happening here. I took a look at the assembler code both compilers create, but the only thing I found out so far is that gcc’s assembly is easier to understand - 50 lines (gcc) vs 134 lines (clang). If someone knows the answer, please tell me.

Also see http://lwn.net/Articles/411776/ for a C++ version that calls f() via boost::thread.

Update: I also reported a bug at http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46186.

Arch Linux’s decision to diverge from the rest of the world that uses python for Python 2.X and python3 for Python 3.X is stupid. And doing this without updating reverse dependencies beforehand and thus breaking packages in their own distribution is insane. In the end this means that if you use Arch Linux, you should consider switching to a distribution that does things the right way: Debian. You can also switch to Ubuntu, that’s normally just a bit less right (= faster). But using a distribution that does those crazy things in such an irresponsible way is really insane.

Really, how can they be so stupid in the Arch world?

1. Local (dpkg, rpm)

The first level of package management is the ’local’ level. This level consists of package management tools that install and/or remove packages via package archives such as .deb or .rpm files and a database of the local’s system state.

2. Connected (APT, YUM, Smart)

The second level of package management is the ‘connected’ level. In this level, tools are fetching packages and information about them from (remote) locations known as ‘repositories’ or ‘sources’. A level 2 package manager is usually built on top of a level 1 package manager, for example, APT uses dpkg - but there are also cases where one tool is both, such as opkg.

3. Abstract (PackageKit, Smart)

A level 3 package manager is a tool that supports different level 1 package managers via one generic interface. It may be implemented on top of a level 2 package manager (PackageKit), or may be implemented directly in level 2 (Smart).

Conclusion: Merge Level 2 and 3

Most level 2 package managers share a great deal of tasks, such as solving dependency problems. Merging level 2 package management and level 3 package management would enable us to reduce the number of dependency problem solvers and combine our efforts into a few package managers. Such an implementation would need to be written in C and support all common level 1 package managers in order to be successful. As some might have guessed, this is what I plan to do with the project codenamed APT2, although work is not progressing very fast.

PS: Back from vacation

I spent the more than the complete last week in (mostly rainy) Corfu, Greece and am now getting back to my usual development stuff. I have already processed my news & blogs backlog (1000+ messages) and will now start with my email backlog, so don’t be angry if the answer to your email takes some time.

<div id="_mcePaste">; <<>> DiG 9.7.1-P2 <<>> @8.8.8.8 wikileaks.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 50227
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;wikileaks.org.			IN	A

;; Query time: 2457 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Aug 27 18:10:43 2010
;; MSG SIZE  rcvd: 31</div>

Update: Sorry Google, for me doubting you. As it turns out, you did no evil, you were just a bit slower than the others.

Update: OK, let’s cancel this rename insanity.

See the announcement for further information.

Android (Google): The situation with Android appears to be even worse. Google keeps an internal development tree and the only time the public sees changes is when a new release is coming and Google does a new code drop. Such a behavior is not acceptable.

Ubuntu (Canonical): Canonical does an outstanding job on openness. Canonical employees develop their software completely in the open, and there are no big code drops. They basically have no competitive advantage and allow you to use their work for your own project before they use it themselves. Canonical employees basically behave like normal free software developers doing stuff in his free time, the only exception being that they are paid for it. Most Canonical employees also know how to behave on mailing lists and do not post HTML emails or do top-posting. There are of course also other people: The design process of the Ubuntu font on the other hand is a disaster, with the font being created using proprietary software running on Windows; for a free software company, that’s not acceptable (such persons should probably not be hired at all). Furthermore, the quality of Ubuntu packages is often bad when compared to the quality of a Debian package (especially in terms of API documentation of Canonical-created software); a result of vastly different development cycles and different priorities.

On a scale from 0 to 15 (15 being the best), Nokia/Intel receive 5 points (“D” in U.S. grades), Google receives 3 points (“F” in U.S. grades), and Canonical receives 12 points (“B” in U.S. grades). Please note that those grades only apply for the named projects, the companies may behave better or worse in other projects.

Final tips for the companies:

  * Nokia/Intel: Teach your employees how to behave in email discussions.
  * Nokia/Intel/Google: Don't do big code drops, develop in the open. If someone else can create something awesome using your work before you can, you are on the right way. Competitive Advantage must not exist.
  * Canonical: Fire those who used Windows to create the Ubuntu font and restart using free tools.
  * All: Document your code.

Autotools: Autotools are ugly, slow, and require an immense amount of code copies in the source tree.

WAF: WAF is not as ugly as autools and it’s faster and does not generate Makefiles or stuff like this. But it has serious issues: It requires one to copy it to the source tarball, has no stable API, and requires Python for building. Furthermore, support for unit testing is broken: It runs the unit tests, but does not abort the build process if the tests fail and does not display why the tests fail.

CMake: The syntax is ugly, it generates Makefiles, and support for pkg-config seems to be very very basic.

So how should a build system look like?

  * It should be installed in the system and not require code copies in the source tree (and thus no pre-build actions)
  * It should not generate Makefiles, but build the project itself.
  * It should not require more than a standard C library.
  * It should support pkg-config out of the box.
  * It should support SONAMEs for libraries.
  * It should detect dependencies on headers automatically.
  * It should support unit testing and abort if the tests fail.
  * It should not require developers to specify how to do things, only what to do.

So, why is there no sane build system?

The laptop features an Intel Core i5 430 M processor with integrated graphics, 4 GiB of RAM, 320 GB hard disk storage, matte screen, and of course a TrackPoint. The machine comes pre-installed with Windows 7 Professional which is not very nice (and Lenovo does not take Windows licenses back). I booted Ubuntu 10.04 from an USB disk which worked a few times, but not always, so I upgraded the BIOS to make USB booting reliable (as Lenovo published multiple BIOS updates already, one fixing this USB booting issue).

The first step was booting Ubuntu 10.04 from an USB hard disk to check the Linux support. Using Ubuntu, everything worked out of the box, including stuff like HDMI audio support or output switching. An exception may be the DVD drive, which only works if you set the SATA mode to compatibility. Non-DVD media also works in AHCI mode, but only if you start with a disc inserted in the drive. Playing DVDs requires setting a region code using setregion(8) [otherwise they do not work at all] and SATA compatibility mode [otherwise they only work partially].

Another issue is the HDMI output, as the TV is not correctly detected and some parts at the top and the bottom of the image are missing (that is, the GNOME panels) (but that happens with different screens and ATI cards as well). Oh, and closing the lid with a screen attached and the internal screen disabled works only as long as you do not open it again, because that causes the GPU to stop.

On the input side, we have one keyboard, one TrackPoint, and a touch pad. All of them work out of the box, but they have a few problems on the hardware side: The cursor keys on the keyboard are incredibly small and the touchpad is very large. The keyboard is different from the one in the HP Compaq: First of all, there is space between the keys (like Apple keyboards); and secondly, you need to apply more pressure in order to use the keys.

The device feels more solid than the HP Compaq 6720s, especially touching the screen does not change the colors on the screen. I don’t know how long the keyboard will last, but I hope that I will still have symbols on them in 5 months (which was not the case for the HP Compaq 6720s keyboard).

Power consumption is another interesting point. When the device is idle and the screen is disabled, the device seems to consume about 10W. With screen enabled, consumption varies between 13W (idle) - 35W (building source code); if the values received via ACPI are correct. The device ships with a 48 Wh battery which after some time was displayed as 53 Wh in gnome-power-statistics (only the ‘when loaded’ state, capacity is displayed as 47.5Wh). According to Lenovo, the laptop lasts 4.5 hours on battery, meaning consumption of 47.5 Wh / 4.5 h = 10.5 Wh which is a bit overly optimistic if you want to work on the device.

After checking everything, I took over my Debian sid installation from my old laptop (which I installed in 2008). It runs happely now, although I still need to get accustomed to the new keyboard and the large touch pad.

All in all, the price of 765€ I paid for the device seems to be OK. Now I only need to remove the Windows 7 sticker and hope that the device lasts long enough until I want to buy a new one (probably 2013, if the aliens don’t invade us 2012).

For now, I downgraded gnome-icon-theme to 2.28 to have a usable UI again. But once there are new icons in gnome-icon-theme I will be lost. Hopefully someone with a sense for UI design will fix this “design hell” created by the idiots those who created this icon theme. We don’t need a 90s icon theme.

Update: Here’s one problem, the network icon just does not fit: (not under the same license as the blog, due to legal reasons).

Update 2: I don’t like changes, but after a months or so I will probably accept it. Anyway, Google did even worse things with reordering the icons in Chromium 5. Previously, all actions were on the left (IIRC); now some are on the left, and some in the input box. Anyway, I replaced “the idiots” with “those” now, as some people felt this was overly offensive.

Update 3: I just filed two bugs at gnome.org: 616324 and 616325. Others will probably follow.

The bug describes that every object stored in a module will not be deallocated if the module is deallocated and it’s m_size = -1 (which it should be if the module has no state). The problem seems to be that Python copies the dictionary of the module but forgets to decrease the reference count of the copy when the module is deallocated. This bug may have serious impact if objects are stored in the module which write status to a file when they are deallocated, because the deallocation functions are never called.

I have no reason to believe that Google Street View is illegal, I see no difference between a photo taken by a company and a photo taken by a single person. If Google Street View were illegal, distribution of every picture taken in the public would have to be considered illegal as well.

And creating laws limiting the height at which the camera is placed to e.g. 1.8 meters sounds like discriminating against persons who are e.g. 1.85 meters (“You’re too tall, you’re not allowed to take a picture”).

More information about this topic can be found on e.g. http://www.spiegel.de/international/germany/0,1518,676616,00.html.

And another question is why yet another name. Moblin was already a well-known name and they shouldn’t have changed the name just because they switch the servers and add some Nokia developers.

Furthermore, does this all mean that there will be no Maemo 6? What will happen to the Maemo users on the N900, will it be possible for them to use MeeGo?

Maybe this is the time to start a new project, Debian Mobile. Debian Mobile would take the MeeGo experience and apply it to a Debian system. This would probably the best mobile operating system ever, being backed by Debian’s large repositories with thousands of software packages.

This uploads brings developers the new API with real classes in apt_pkg (you can now use pydoc to view documentation), C++ bindings for making apt-pkg applications scriptable (although they should be considered experimental), a test suite (although aptsources fails in one test for now) and many new context managers for enhanced Python 3 coding fun. And objects are now freed when their reference count reaches 0. A more complete list of news can be found in the What’s New In python-apt 0.7.100 part of the documentation.

For the next releases until 0.7.100 release, the focus is clearly on fixing bugs and improving the documentation. We need more tests of the Python 3 builds, especially in areas dealing with str and unicode stuff.

Have fun, read the documentation, and code.

In other news, the defect N900 with QWERTZ keyboard is back to Amazon.de; and I have ordered a QWERTY one on expansys.de yesterday. Let’s see when that one arrives.

Appart from that problem, the device works almost perfectly. The internet browsing experience is faboulus, the camera is good, and the device is fast; the screen is great and the speakers are great as well. The keyboard is easy to type with and you can type fast after a short time. The video player does not seem to like my H.264 files (the OpenMAX decoder fails whereas on my PC ffmpeg is able to play it).

That’s all for now, I’ll write again when I have a N900 with working AV output.

First, they stated that I am planning an APT2 release for Christmas. They took the statement

as a proof for this. But in the context of this paragraph, ‘publish’ was not meant in the term of publishing a release, but in the term of publishing the code (of the internal branch) in the public repository. The code is public now and lives in a ’temp’ branch and will be reworked there for inclusion in the master branch.

Secondly, those news sites called me an Ubuntu developer. While I do contribute to Ubuntu, and am an Ubuntu Member, I am NOT an Ubuntu Developer, as I am NOT a member of the relevant ubuntu-dev team at launchpad.

Thirdly, those who stated that speed is a goal (mostly pro-linux, at least from my understanding): It is not, at least not now. It is just a coincidence caused by using a relational database. Furthermore, the test was not really fair, since the other package managers both provide more information than capt; information which has yet to be made accessible in APT2. It was just an initial conclusion that SQLite is quite fast.

Please note that APT2 is a free time project in development, and the programming language used is still in development as well; as well as some other reverse dependencies. I should also state that neither Debian nor Ubuntu have any plans to drop apt at the moment; and APT is still actively developed.

While nothing else happened in the public repository, the internal branch has seen a lot of new code; including SQLite 3 caches; Acquire text progress handling; and capt; the command-line advanced package tool. Most of the code will need to be reworked before it will be published, but I hope to have this completed until Christmas. It will also depend on Vala 0.7.9 or newer, which is yet to be released.

The decision to use SQLite 3 as a backend means that we won’t see the size limitations APT has and that development can be simplified by using SQL queries for filtering requests. It also means that APT2 will be very fast in most actions, like searching; which currently happens in 0.140 seconds (unstable,experimental and more repositories enabled), whereas aptitude takes 1.101 seconds, cupt (which has no on-disk cache) 1.292 seconds, and apt-cache 0.475 seconds. Searching is performed by one SQL query. I also want to thank Jens Georg mail@jensge.org, who wrote Rygel’s Database class which is also used with minor modifications (like defaulting to in-memory journals) in APT2 as well. Rygel.Database is a small wrapper around sqlite3 which makes it easier to program for Vala programmers.

The command-line application ‘capt’ provides a shell based on readline with history (and later on command completion) as well as direct usage like ‘capt config dump’ or ‘capt search python-apt’. Just as with Eugene’s cupt, ‘capt’ will be the only program in the core APT2 distribution and provide the same functionality currently provided by apt-get, apt-config and friends. The name is not perfect and can be easily confused with ‘cupt’, but it was the closest option for now; considering that the name ‘apt’ is already used by Java (for its “Annotation Processing Tool”).

That’s all for now, I’ll tell you once all those features have passed my QA, and there is really something usable in the repository. In the meanwhile, you can discuss external dependency solvers, database layouts and other stuff in their threads on deity@lists.debian.org.

And a ‘screenshot’ from capt:

jak@hp:~/Desktop/APT2:temp$ capt
apt$ help
APT2 0.0.20091213 command-line frontend

Commands:
  config dump               Dump the configuration
  config get OPTION         Get the given option
  config set OPTION VALUE   Set the given option
  search EXPRESSION         Search for the given expression
  show PACKAGE              Show all versions of the given package
  sources list              Print a list of all sources
  version                   Print the version of APT2
apt$ search python-apt
build-depends-python-apt - Dummy package to fulfill package dependencies
python-apt - Python interface to libapt-pkg
python-apt-dbg - Python interface to libapt-pkg (debug extension)
python-apt-dev - Python interface to libapt-pkg (development files)
python-aptdaemon - Python module for the server and client of aptdaemon
python-aptdaemon-gtk - Python GTK+ widgets to run an aptdaemon client
apt$ 

If Oracle closed MySQL, the community would step in. I truly believe that the free software community is strong enough to support MySQL. There already is MariaDB and I believe that this project could serve as the base in case Oracle closes MySQL or stops MySQL completely. The community was strong enough to build complete operating systems and maintain them, so it should not be that hard to support one database.

All in all, I don’t see how Oracle buying Sun (and thus MySQL) could negatively influence competition in the database market, and I would therefore recommend the EC to approve the Oracle-Sun deal as it, without imposing any conditions on Oracle.

Ubuntu Software Center (or just ‘Software Center’) is a new graphical user interface for installing and removing applications; replacing gnome-app-install. Under the hood, it uses aptdaemon which exposes an interface to APT via D-Bus; i.e. something in the direction of PackageKit. At a later stage, the Software Center shall replace Synaptic, Update Manager and various other programs related to package management.

The aptdaemon package is completely compatible to the Ubuntu one, and could thus be synced directly to Ubuntu without any change (if Ubuntu supports “3.0 (quilt)” source packages now, I have not looked into this). The software-center package is based on the latest Ubuntu lucid package; and contains some generalization (e.g. replacing ‘Ubuntu Software Center’ with ‘Software Center’) at some more places. It still needs some work in the documentation and some parts of the program will have to be adjusted for Debian aswell. We also do not have a debianized icon yet; this will be worked on later.

It seems that the PC era is slowly coming to an end, with devices being increasingly connected ’to the cloud’ and people being always online; and storing their data on Google’s servers. We do emails online using Google Mail, we do navigation online using Google Maps, we edit and view our documents using Google Docs, our newspaper is Google News; and when we want entertainment we open the browser and type youtube.com into the URL bar. Even if we were formatting the hard disk and reinstalling the system, most people wouldn’t even notice; because all there data is stored online.

There is also the question of freedom. Free software is not very widespread in the SaaS world. You also lose the control over your data. But RMS can tell you more about it.

So it seems that 2010, Google is the new mainframe and netbooks and smartphones are the new terminals. Whether this is good or not cannot be said. The question you have to ask yourself is whether you can trust Google to keep your data secure or not. I trust them enough to host all of my emails, the RSS feeds I read, my searches.

In other news, the development of Ubuntu 10.04 Lucid Lynx just started. For the first time in Ubuntu’s history, the development will be based on the testing tree of Debian and not on the unstable tree. This is done in order to increase the stability of the distribution, as this release is going to be a long term supported release. Ubuntu will freeze in February, one month before the freeze of Debian Squeeze. This should give us enough room to collaborate, especially on bugfixes. This also means that I will freeze my packages in February, so they will have the same version in Squeeze and Lucid (applying the earliest freeze to both distributions; exceptions where needed).

I’m currently running the release candidate of Ubuntu 9.10 “Karmic Koala” on this system, but I expect to return to my full Debian unstable developer environment during the next week(s). Karmic seems to be pretty stable already, but I have experienced problems with PulseAudio and my PCM control getting set to +13dB which is horrible.

I will probably also pre-order a Nokia N900 soon, which will be my largest investment this year - but I really need a new mobile phone, camera and music player. And not to forget the ability to develop software for it, and testing my software on those less powerful devices (compared to my laptop).

All in all, I’m back and soon ready to hack again.

I also fixed some bugs in APT2, like the missing build-depends on libgee-dev and the configuration parser now accepts ‘.’, ‘_’, ‘+’ in the option name. I also talked with Eugene about some differences in the way cupt and APT2 handle quotes and about some other parts of the configuration format. Seems this was a good day.

apt2-config replicates the functionality of apt-config. It currently do not read the configuration files in the correct order, so don’t expect “apt2-config dump” to produce exactly the same output as “apt-config dump”. To build APT2 yourself, you need a patch for waf from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=548329.

First of all, there is a language issue. Implementing a package manager in Perl has some major drawbacks. One of the features of APT was it being written in a lower-level language (i.e. C++ which really is below Perl), making it possible to write applications like synaptic and python bindings which in turn lead to applications like gnome-app-install or Ubuntu’s new Software Store.

Furthermore, writing a package manager in Perl means that Distributions such as Emdebian might not be able to use it since they have excluded Perl due to its space requirements. This becomes even more important considering that cupt depends on even more perl libraries. This means that cupt will never be able to replace APT.

Secondly, a package manager should not be designed specifically for one distribution. This is another major drawback of cupt and other package managers such as yum or zypper. The smart package manager, written in Python and funded by Canonical Ltd. is an example for a distribution-neutral package manager.

Now let’s take a look at package management in modern distributions. Usually we have two levels of package managers, the first being tools like dpkg and rpm which take care of installing and removing the packages and level-2 package managers implementing dependency resolvers and package retrieval from remote locations. Recently, distributions started to add a third layer named PackageKit, which shall provide distribution-independent package management user interfaces. The project was well-received by RPM-based distributions, but failed in Debian-based distributions due to not supporting debconf. Furthermore, adding a third layer just increases the possibility of problems.

The right way to do package management is a distribution-independent level-2 package manager written in C. The smart project shows us that this is possible although itself fails to meet the lower-level language (C) requirement. That’s why I decided to write a package manager in Vala, a GObject-based language which gets converted to C and then compiled. If successful, this project will be able to replace most of the current level-2 package managers and will also provide the same distribution-independence as provided by a level-3 package manager such as PackageKit. It is also easy to create binding for other programming language such as Python or Perl thus enabling application developers to choose the language they like most.

The core of this project is a vendor-neutral library, temporarily called libapt (as the project is called APT2 for now). This library contains all the code which is not specific to a vendor i.e. file retrieval, dependency resolver, caches, etc and  is then enhanced by several vendor-specific plugins, each implementing a PackageManager (interface to the distribution’s level-1 package manager) and a Repository (well, repositories from which you can download packages) interface.

We could even enhance the vendor-independent interface to include more details of a repository. Most repositories nowadays consist of 4 components: A meta index (Release files for Debian, repomd.xml for Fedora/openSUSE/etc.), a package index (e.g. Packages files for Debian, primary.xml.gz on Fedora, etc. ), a source index (e.g Sources files in Debian) and a files index (e.g. Contents-.gz for Debian). I took a look at the repository formats of Slackware, openSUSE and Fedora and it seems that this concept can be applied to all of them. So maybe all we need are distribution dependent parsers for those files.

One of the most important issues with APT is its use of mmap() for the cache. Using mmap() makes it hard to grow the cache, which is sometimes needed. We see a lot of bug reports from people with too small cache sizes. We can circumvent this problem by utilizing an embedded database like SQLite for this, but we would probably loose some speed and it may be harder to maintain a flexible API. We should see what the best option is here, both ways are possible since Vala 0.7.6 includes my patches for adding mmap(), ftruncate(), mremap() and some other functions. An idea to circumvent the mmap() issue is gathering statistics about the relation between the number of repositories and the size of the cache and then using a value which is slightly above the average statistical value.

The project is not very mature yet, it only includes basic library functions for downloading files and parsing configuration files, etc. You can find the (MIT licensed) code at http://git.debian.org/?p=users/jak/apt2.git. I also have some local code for repository management and multi-threaded file fetching, but it’s just not ready to be merged yet.

I ran the V8 and SunSpider benchmarks to compare Iceweasel 3.5, Midori 0.1.9 (using WebKit GTK+ 1.1.12) and Chromium 4.0.206.0 (r25168). The result was that Iceweasel was 10x times slower than the others in the V8 benchmark and about 5 times slower in the SunSpider benchmark. The others were almost equally fast, but Chromium won the V8 benchmark with 2338 points compared to Midori’s 1666. More details are in the PDF Browser Performance, which should have been an ODP, but uploading ODPs is not allowed on wordpress.com. I also ran the V8 benchmark on Arora some time ago, but it was almost as slow as iceweasel. All tests were done on my laptop running Debian GNU/Linux unstable (amd64 architecture).

I chose Vala for this task because it has an easy syntax and allows me to use the features provided by GLib and Co. really easily. GLib provides many functions needed for APT like file access, unicode handling, hashsums creation. It also provides us with signals which are useful for implementing progress reporting.

Another project, called ‘cupt’ tries to provide an APT replacement in Perl. Apart from the fact that I don’t like Perl, it also has the disadvantage of being too big to ship on embedded devices and is not useable from other languages such as Python or C/C++. Yet another project, ‘smart’ provides a different package manager written in Python. While I like Python, it can’t be used from languages such as Perl or C/C++ and is too large. APT2 will only require GLib, Gee and libarchive, which require about 2MB of space; about 1/10 of Perl’s size.

The core of APT2 is a library called ’libapt’. At the moment, this library provides classes for reading tag files (TagFile&TagSection) and an incomplete configuration class based on libgee’s HashMap (to ease coding, it will probably be changed to a special data structure like those in apt-pkg). The next step will be implementing a parser for configuration files (currently we only support in-memory objects) and writing the Acquire system.

The planned Acquire system uses GModule to modularize the support for different URI schemes. Each module provides a worker class which implements one or more URI schemes. The first of these modules will provide a worker using GIO, which deals with local file access, samba shares, FTP, SFTP, WebDAV and various other protocols, including HTTP until a replacement has been written (since we don’t want to force gvfs-backends). The workers communicate with the parent Acquire object using signals and can cause the whole acquisition to be aborted by emiting an “abort” signal (or similar).

The whole project has just been started and may take some months until it becomes useful (if it becomes useful at all). The code can be found at http://git.debian.org/?p=users/jak/apt2.git and is licensed under the MIT license (see http://www.opensource.org/licenses/mit-license.php). It is a secondary project and will not affect my work on apt (not much yet) and python-apt (0.8 series).

I personally don’t like passive notifications that much, as I loose the control over the stuff on my display because I can’t close the notification once I read it. But I like the design of notify-osd’s notifications.

In other news python-apt 0.7.92 is now available in experimental, accepted by ftpmasters 3 hours after it entered NEW.

I expect this pre-release to be one of the most buggiest in the 0.8 series, because it changes a lot of the core stuff, like memory management. The release is currently waiting in NEW due to the new python-apt-dev package, if you want to test it, you can use the repository at http://people.debian.org/~jak/debian/experimental/, where the source package and the binaries for amd64 are provided. Most applications should continue to work with python-apt 0.7.92, if you find a non-working application, report a bug (unless its an error related to apt_pkg.Version, which has been renamed to apt_pkg.VERSION due to naming conflicts).

The next release should hopefully reach RC-quality. It will primarily focus on documentation updates (there is a lot of stuff missing and parts are wrong currently) and bugfixing, but will also introduce the new Qt4 progress classes. The next release will also feature the ABI and API freezes. It is scheduled for release in September (next month), and 0.8 is scheduled for October.

For further information, you can send me an email or ask in #debian-apt on OFTC.

Consider the releases of Ubuntu 8.04 LTS (April 2008) and Debian GNU/Linux 5.0 (February 2009). Whereas Ubuntu 8.04 provides GNOME 2.22 including a new Nautilus, Debian provides GNOME 2.22 with nautilus 2.20. Ubuntu’s release made at about the same time (Ubuntu 9.04) already included GNOME 2.26.

The reason for this are the different development processes. Whereas a Debian release is based on stable upstream versions, the development of Ubuntu releases happens using newest pre-releases, causing Ubuntu releases to be one generation ahead in most technologies, although released at the same time.

Another difference is the way of freezing and the duration of the total freeze. Ubuntu has a freeze process split into multiple stages. The freeze which is comparable to Debian’s freeze is the feature freeze, which usually happens two months before the release. Debian’s freeze happens 6 month before the release, just about the time when Ubuntu has just defined the features to be included in the next release.

Synchronizing the release cycles of Debian and Ubuntu basically means that Ubuntu will always provide the newer features, better hardware support, etc. Ubuntu is the winner of this decision. It will have less bugs because it inherits from a frozen Debian branch and it can include newer versions where required because it freezes 3 months later.

To synchronize the package versions shipped in Debian and Ubuntu you have to make your release schedules asynchronous, in a way that the Debian release freezes after the Ubuntu release. This basically means that I expect Debian 6.0 (2010/H1) to be more similar to Ubuntu 9.10 than to Ubuntu 10.04. I would have preferred to freeze Squeeze in December 2010 and release in April 2011, and Ubuntu LTS to be re-scheduled for October 2010.

Now let’s say you are a Debian and Ubuntu user (and you prefer none of them) and you want to setup a new system in 2010/H2 (so the systems have a half year to stabilize). This system should be supported for a long time. You have two options: Debian 6.0 and Ubuntu 10.04. Which will you choose? Most people would probably consider Ubuntu now, because it provides better support for their hardware and has the newer features. A (partial) solution to this problem would be to make backports an official part of Debian starting with Debian 6.0.

Furthermore, there is the question of security support. If we want to provide the ability to upgrade directly from Debian 5.0 to Debian 7.0, we will have to support Lenny until 2012/H2 (to give users enough time to upgrade, when we release 7.0 in 2012/H1). This means that we would have to support in 2012: Debian 5.0, 6.0 and 7.0. And another side effect is that Debian 6.0 would have a 3 year security support, just like Ubuntu 10.04.

All in all, the decision means that Debian and Ubuntu LTS releases will occur at about the same time, will be supported for about the same time and that Ubuntu has the newer packages and less bugs to fix.

I can not guarantee that all the names will be kept like they are at the moment (it’s a pre-release), but there should not be many more changes needed. The series will hit Ubuntu Karmic later this month, and the final 0.8.0 release is going to be shipped in the final Karmic release.

If you want to help with python-apt, consider to write some examples of what can be done with python-apt, and some tutorials for the documentation. You can also check for spelling mistakes and alike. If you want, you can also contribute code. See the documentation (in the package, or online) for guidelines on how to contribute.

You can get the 0.7.91 release from Debian experimental, and you can view the documentation online at http://apt.alioth.debian.org/python-apt-doc-exp/.

And of course, a short example:

with cache.actiongroup(): # apt.Cache 'cache'
    for package in my_selected_packages:
        package.mark_install() # New PEP 8 names, previously named markInstall()

Ubuntu One creates a directory named Ubuntu One in your home directory. Within this directory, there are two subdirectories. The first one is “My Files” and the second one is named “Shared With Me”. When you place files in the “My Files” directory, the Ubuntu One client gets notified (using inotify) about the change and begins uploading the file to the Ubuntu one server.

When you access the web interface, which should work in every modern browser, and upload a file there, the next time your local client connects the files are fetched to your local hard disk. This also works when you have two different computers and create a file on the one computer, it will be visible on the second one as soon as it has fetched the new file.

You could also copy your .mozilla directory into the synced directory, and create a symlink from your home directory to it. I have not tried it myself, but in theory this would allow you to have your profile synced on all your computers.

And the best thing about Ubuntu One is that the client is completely free software and written in Python. This makes it possible to package the client for other distributions, like Debian. Packaging it for RPM-based distribution such as Fedora should also be doable, but may require some more time.

There seems to have been some criticism that the server side is not free software. While that may not be the good, it’s certainly better than other services where even the client is proprietary. And there still is the possibility to write your own server as the protocol is available.

Ubuntu One is currently in private beta, if you want to try it out, you need an invitation (visit ubuntuone.com for further information).

PHP is crap

Many PHP applications are low quality applications. The problem is that PHP makes it hard for developers to write good applications. PHP developers tend to use many global variables, global code, and not many classes or functions. And when writing applications for the web, PHP makes it easy to mix PHP and HTML, in my opinion even too easy.

Well-written applications follow some basic rules. First, separate your code into functions and classes, and (2) split your code over multiple files. And there is one rule which is especially important for non-commandline applications: (3) seperate logic and presentation. This means if you write a web application, use a template engine.

PHP error and exception handling is also an interesting topic. The problem here lies in the distinction between the both. In my opinion, there should only be one way to handle such stuff: Using exceptions. Exceptions are a very widespread way to program today, and are used in languages like Java and Python for cases where PHP uses errors. The concept of PHP’s errors results in code like @do_something or die("blah"); being written in libraries which is a problem as it clearly violates rule (3) I wrote above, because there is no way to catch this die() error.

Furthemore, PHP has no clear concept of namespaces in the current stable versions (< 5.3). PHP 5.3 finally gets namespaces, but using a backslash character, which looks really bad and makes everyone think of eithers Windows systems or escape characters.

There are even more problems with PHP. From time to time, PHP just crashes with a segmentation fault.

MySQL vs. PostgreSQL

MySQL seems to have many problems with checking data, which can confuse me. It got better in MySQL 5 with the introduction of the new sql_modes like STRICT_ALL_TABLES and TRADITIONAL. And using storage engines like InnoDB also allows you to take advantage of stuff like foreign keys and transactions. But MySQL is still problematic.

For example, in an application I wrote I had multiple entries identified by a numeric id (an INTEGER). One day, I wanted to check whether my error handling was OK and requested an entry with the invalid ID ‘1X’, which resulted in SELECT * from A where id='1X';. Now I expected to get an error or at least no result, but MySQL happily proceeded with the query and returned the entry with the ID 1. I tried the same thing on PostgreSQL and got an error message that ‘1X’ is not integer, as I expected it.

As another example, when I run the same PHP application using PDO on a MySQL database and a PostgreSQL database, I always get strings from the MySQL database whereas I get integers from PostgreSQL if the columns are integers.

Another thing which PostgreSQL does better than MySQL is password encryption. MySQL only provides access to the standard crypt() function and by default only generates DES passwords. It allows other methods to be used (although not all are available everywhere and therefore not documented), but it provides no way to create a salt for them, which makes them not very useful. PostgreSQL in contrast provides less encryption methods but it provides the gen_salt() function which can be used to generate salts for blowfish encryption (gen_salt('bf')).

Another problem is that MySQL simply ignores references when they are added to the column definition instead of a separate ‘FOREIGN KEY’. Because MySQL gives no errors when a feature like REFERENCES is not supported, developers may think that the feature is supported and write code using this feature just to realize later that what they wrote does not what they want.

And you can’t write AGPL applications using MySQL client libraries, because MySQL is GPL-2 which is incompatible with the AGPL. You might think that the “FOSS License Exception” helps here, but this only lists (among others) the GPL-3. Or am I wrong?

But PostgreSQL has other problems, at least when it comes to the performance of the information_schema.columns view. This may be related to the fact that PostgreSQL has > 1000 rows whereas MySQL only has 250. And PostgreSQL may be a bit harder to configure.

Final words

All in all, PHP and MySQL are not the best choices for writing applications. They are to tolerant, and do not treat errors in an acceptable way – by aborting the current action. I can get around some problems in PHP by using error handlers which raise exceptions and setting error_reporting to E_ALL, but in fact this should be the default. MySQL problems can be solved to a large extend by setting sql_mode to TRADTIONAL and using InnoDB databases, which should in my opinion be the default and not an option.

My recommendation is to use PostgreSQL servers and for the applications: use frameworks (if you are allowed to, which may not be always the case [e.g. when you are learning SQL, using a ORM is clearly the wrong way]), document your code, separate your code into classes and functions, and make sure you are not using to many global variables. And use a SCM to track your changes.

And this is not the only change. Memory usage has been decreased by 10MB by creating the Package() objects on the fly instead of pre-creating all 25000 ones. All classes which previously supported the has_key() method now support contains, which allows you to write ‘key in mapping’.

But that’s not all. The Package class now gained support for setting the candidate version of a package, in the same way you get it, using the property: mypkg.candidate = max(mypkg.versions). And you can now pass file descriptors to functions previously only working with file() objects.

But we’re not done yet. The complete API change is pending, renaming everything to PEP-8 compliant names and deprecating the old ones, which will be removed at a later time (after the release of Squeeze). The module apt.debfile is not affected by this change, because it already used the correct names. But for Python 3, we will not keep backward compatibility, in order to make coding easier and to make it clear that applications should be ported to the new API.

And if you want to get your feature into python-apt now is your time. If you want something merged, provide the code, the documentation (with example) and a list of use cases; and report a wishlist bug in the Debian BTS.

And if you want to write a tutorial about python-apt, write it in reStructuredText and I will add it to the documentation. The documentation is currently basically an API reference and there could be much more text.

Python-apt 0.7.90 is currently uploading. The whole development will result in python-apt 0.8.

In Python 2.5, everything works perfectly. In Python 3.1, the same code produces an error. An example is apt_pkg.GetCache().Packages. I defined the slots tp_getattro and tp_methods. In Python 3.1, tp_getattro seems to be ignored.

If you want to help, http://bzr.debian.org/loggerhead/users/jak/python-apt/py3k/changes for browsing the branch and http://bzr.debian.org/users/jak/python-apt/py3k/ for branching it.

This branch contains the current state and allows you to build a python-apt package shipping with Python 3.1 modules and extensions. But as I wrote above, not everything works yet. But its far enough for about 5 hours of work.

Update 2009-04-12 00:17 CEST: I just fixed a bunch of problems, including the one listed above. Most of python-apt should work now, including the ‘apt’ package.

This release is mainly a bugfix release, but also brings new features like apt.package.Version.uri and apt.package.Version.fetch_binary().I also added a Breaks: debdelta (« 0.28~) because debdelta 0.27 is not working anymore since python-apt 0.7.9 and I expect that this problem will be fixed in 0.28, which could then use the new apt.package.Version.uri API to fetch the uris of the packages).

Here is the changelog:

python-apt (0.7.10) unstable; urgency=low

  * Build-Depend on python-debian, use it to get version number from changelog
  * Depend on libjs-jquery, and remove internal copy (Closes: #521532)
  * apt/package.py:

    *  Introduce Version.{uri,uris,fetch_binary()}


  * debian/control:

    * Remove mdz from Uploaders (Closes: #521477), add myself.
    * Update Standards-Version to 3.8.1
    * Use ${binary:Version} instead of ${Source-Version}
    * Fix spelling error: python -> Python


  * debian/copyright: Switch to machine-interpretable copyright
  * Fix documentation building

    * doc/source/conf.py: Only include directories for current python version.
    * debian/control: Build-Depend on python-gtk2, python-vte.
    * setup.py: If pygtk can not be imported, do not build the documentation.


  * Breaks: debdelta (<< 0.28~) to avoid more problems due to the internal API changes from 0.7.9.

– Julian Andres Klode jak@debian.org Wed, 01 Apr 2009 15:24:29 +0200

Have fun.

1. Introduction of the documentation

As you may know, python-apt 0.7.8’s only documentation was available from the source code and to some parts from the docstrings. This changed in python-apt 0.7.9exp2, when I introduced a complete documentation written in reStructuredText, and generated using Sphinx. This documentation has even improved in the final 0.7.9 release, and I will push it to the online mirror at http://apt.alioth.debian.org/python-apt-doc/ soon (it currently still has 0.7.9exp2).

2. Graphical Progress

The module apt.progress.gtk2 provides widgets for Python GTK+ coders, allowing them to easily build graphical applications interacting directly with apt. I am still looking for someone to write a apt.progress.qt4 module, if no one wants to do this, I will do it myself.

3. Enhanced apt.debfile module

A lot of code has been merged from gdebi, now allowing to do a large set of operations on local .deb files. For example, you can now install these packages. This module is also the first module to follow PEP8 naming conventions (lowercase_with_underscores), the others will be adapted at a later point.

4. Complete code cleanup

I have worked on cleaning up the whole code, fixing white space problems, and more. Ben Finney was very helpful, I merged some of his patches for whitespace. I also fixed several other problems related to PEP8 conformance. Previously the PEP8 checking tool (see documentation) found about possible 964 issues, this has been reduced to 7 (-957), but these 7 ‘issues’ are simply the usage of has_key() at locations where it can’t be avoided [interfacing with TagSection objects], i.e. they are false positives.

5. Introduction of apt.package.Version

In python-apt 0.7.9 I am introducing a new class apt.package.Version. This class provides access to various attributes of a version, like the record, the version number, the origin and even allows you to fetch the corresponding source code. But one of the best things about it is that it is sortable, allowing you to write:

print 'Highest bash version:', max(apt.Cache()['bash'].versions)

This makes it very clear that you can build really cool applications with it. The various candidate*() and installed*() methods of apt.Package objects haven been deprecated, and you should use the Version() objects available via Package.installed or Package.candidate properties.

Summary

All in all, the python-apt 0.7.9 is a big step forward. It is one of the largest releases in the history of python-apt and provides various new features to make developing easier. With all the code cleanups, and the documentation, you should be able to find everything you need; if not please report a bug. When we are asked “Can we do this with python-apt?”, we can finally say: “Yes we can!”

Already done

New package: metatheme-gilouche

Today I uploaded the new package metatheme-gilouche, which builds the binary package gnome-theme-gilouche. The Gilouche theme has been created by openSUSE and is used there as the default theme in GNOME. For users of gnome-app-install, this theme provides a “better” style for your main menu. It contains icons (replacing industrial-icon-theme), GTK+, and Metacity themes. The package is currently NEW, once it is in the archive, I will request removal of industrial-icon-theme. And once this package reaches Ubuntu, Ubuntu’s Bug#96042 is closed.

python-apt

Two weeks ago, Sunday (on 2009-03-08), I coded a bit in python-apt, and introduced a new class apt.package.Version, which contains information about a specific version of a package; and an interface to fetch the source package for that version. You can now simply call something like package.candidate.get_source() to fetch it.

The Package class now provides three new properties: Package.candidate, Package.installed and Package.version. The first two return a Version() object for the candidate/installed version or None, if no one is available (which only happens for installed; because there should always be a candidate [because Package() objects are only created for packages where at least one source is available]). These properties replace those like Package.candidateOrigin, which now is Package.candidate.origins [which also fixes the naming]. The old ones are still available, but accessing them results in a DeprecationWarning.

The whole thing should be hitting unstable soon as 0.7.9.

Other stuff

Less interesting, but also true - I have uploaded a new version of ndiswrapper (1.54), and the last snapshot of aufs1 (0+20090302-1). I have also renamed the method debimg.core.resolver.Resolver.add_task() to debimg.core.resolver.Resolver.add_tasks(), but this is a really boring change.

Planned changes

Planned changes in aufs packaging: switching to aufs2

This is a bit complicated, as aufs2 seems to require a large amount of new exports in the kernel. But we need to switch, because the old version of aufs is not developed anymore upstream. With the switch to aufs2, we will loose functionality like exporting the filesystem via NFS and other ones. The code will be split into two source packages, aufs-utils and aufs.

Planned changes in gnome-app-install

First of all, I need to merge the new version of gnome-app-install from Ubuntu. Afterwards, I will start working on various packaging related issues. I will switch the package from CDBS to debhelper 7; install the modules to /usr/share/gnome-app-install/, not systemwide; and also try to create a single package which works on Debian and Ubuntu, so we can have a shared package in future.

Planned changes in command-not-found

First of all, I need to merge the new version from Ubuntu. The version of command-not-found in Debian has various differences from the one shipped by Ubuntu. It uses the files from apt-file to create our command list, it is installed in /usr/share/command-not-found, not /usr/lib/command-not-found, and more.

My plans for the next upload include the inclusion of a pre-generated command-list, for those who do not want to use the apt-file based method; and because it will be available out of the box.

Planned changes in python-apt

First of all, I am still working on improving the documentation. While everything should be listed already, there are still some problems with their descriptions, and some are missing them. I intent to fix this in python-apt 0.7.10 (or whatever version it will be). Everyone can help, for example by sending new documentation texts or more examples (see http://apt.alioth.debian.org/python-apt-doc/coding.html for further information).

Secondly, I will work on a QT4 progress module, in order to equally support GTK+ and QT4 toolkits. This is not really an important task, and if someone else wants to do this, I would be happy too.

Thirdly, I will work on the complete deprecation of mixedCase naming conventions, and switch everything to lowercase_with_underscores; while still providing backward compatibility.

Fourthly, I will complete my work on restructuring the progress module (now a package), to introduce a generic naming scheme, and modules like apt.progress.text who will implement them.

That’s all

If you are a developing for Python, be sure to checkout python2.6 in experimental and thank doko for uploading it. I really hope that python3.1 leaves NEW soon, so I can “play with it”.

If you know something else I did or will do in near future, please leave a comment. ;-)

This is what happens:

  * Check file system for errors (30min)
  * Resize the file system (30min?)
  * Resize the partition
  * Check file system for errors (30min)
  * Move the file system

    * Read-only Simulation (!!!!!!!!!!!!!)
    * Do it (~5 hours)

It really should tell me that it takes such a long time. I mean, why does it have to copy all 87GB, when all I want to do is get 5GB moved to the front? It should free these 5GB on the filesystem, move them to the end of the filesystem, decrease the partition size and finish.

It should not copy 87 GB 2 times, and should not run 2 filesystem checks without informing me before how long this will take. Now I wonder what happens when it resizes my 110 GB extended partition to 105 GB, and moves it?

Bad software, bad day.

In my opinion, all packages which are located on a disk X, should only depend on a disk N (N<X). This means that the package gnome would be moved to disk 2, or its dependencies to disk 1.

This is exactly the way debimg works. Debimg uses so-called package groups, which simply represents a set of packages with cyclic dependencies. These package groups are returned in a specific order, so that a group N only requires a group <N. This order is kept when the groups are added to the disk. Furthermore, we treat the groups as one when adding them to the disk, ie. we check whether the whole group fits on the disk (and add it) or not (create a new disk). This way, we ensure that all dependencies can be satisfied.

Today, I decided to try it out and therefore wrote a small script reading the debian-cd task files and comparing the file list (of the packages added to the disk) with the file list of the official KDE disk.

The results look very good, debimg adds 11 packages and removes 26 which is not that much, and mostly caused by wrong size limits, etc. You can look at the results your self, and regenerate them using:

  * [http://debimg.alioth.debian.org/tests/debian-500-amd64-kde-CD-1.diff](http://debimg.alioth.debian.org/tests/debian-500-amd64-kde-CD-1.diff) - The results
  * [http://debimg.alioth.debian.org/tests/debian-cd.py](http://debimg.alioth.debian.org/tests/debian-cd.py) - The script used.
  * [http://debimg.alioth.debian.org/tests/debimg.tar.gz](http://debimg.alioth.debian.org/tests/debimg.tar.gz) - The version of debimg used (current HEAD)

Anyway, debimg still needs real configuration and handling of the debian installer and source packages, but this should give you an idea of how debimg works. And a tarball, for those who don’t want to use git all the time.

BTW, I have now enabled comment threading here, a new Wordpress feature. I hope that it works, as I have not tried it out yet.

First of all, the repository module has been merged into the master branch. This was the first step towards the creation of the image building, which happened today by introducing the ‘image’ module.

The code should be treated as Beta quality, but the project as a whole is Alpha, because the application utilizing debimg.core is still missing. As always, I hereby encourage to try out debimg, have a look at the examples, and help to develop it.

Patches should be sent to the mailing list, created with git format-patch, and its usual settings (eg. prefixed with eg. ‘[PATCH 1/2]’). The patches should be inline and validate in pyflakes and using pep8.py. See the README for some more recommendations.

Because NEW seems to be really busy at the moment (and doko uploaded python2.6 and python3.1), I’m not uploading the current state as 0.1.0a1 to experimental, but wait until NEW is smaller (maybe it will be 0.1.0b1 then).

What is wanted/planned?

  * Compatibility configuration formats

    * (debimg.config.simple_cdd) A reimplementation of simple-cdd using debimg. This probably needs to wait until there is a module for dealing with the installer, but I want to implement an application to support simple-cdd configuration files.
    * (debimg.config.debian_cd) There could be an implementation of debian-cd's configuration format. This would allow people to easily try out debimg. We can only support a subset, though.


  * New configuration formats

    * (debimg.config.yaml) Debimg's native configuration format. This is the only one supported by the graphical frontends. Where possible, configuration between compatibility formats and this format will be provided.


  * Graphical Frontends:

    * (debimg.frontend.gtk2) The graphical GTK+ frontend for inexperienced users who just want to create their own disk.
    * (debimg.frontend.qt4) The graphical frontend written in QT4.


  * Text frontends

    * (debimg.frontend.text) The basic command-line frontend.
    * (debimg.ubuntu.frontend.text) A script to build Ubuntu images, which could be used to build the official Ubuntu Images. This requires interaction with the germinate tool for dependency resolution, and more. It's also not flexible enough for building custom images.

Requirements for 0.1 Beta

  * Implement at least one configuration format and the build frontend.
  * Have at least one external contributor.

More

  * Mailing-List: [debimg-devel@lists.alioth.debian.org ](mailto:debimg-devel@lists.alioth.debian.org) (Archive at [http://lists.alioth.debian.org/pipermail/debimg-devel/](http://lists.alioth.debian.org/pipermail/debimg-devel/))
  * Wiki: [http://wiki.debian.org/DebImg](http://wiki.debian.org/DebImg)
  * VCS-Git: git://git.debian.org/debimg/debimg.git
  * VCS-Browser: [http://git.debian.org/?p=debimg/debimg.git](http://git.debian.org/?p=debimg/debimg.git)

Changes

Julian Andres Klode (6):

  * debimg/core/files: When creating a file object allow filename as source
  * debimg/core/resolver.py: Introduce Package.fullname, Package.component
  * debimg/core/repository.py: Merge the repository module.
  * debimg/core/resolver.py: Improve handling of certain dependency types
  * debimg/core/repository.py: Allow Repository.add_group to take a 'distro' parameter
  * debimg/core/image.py: Introduce the image module.

You can read more about this in the Debian Bug Tracker in Bug#516190. I expect that this should be forwarded to upstream but I haven’t checked their bug tracker yet.

Next would be the review of OpenSolaris and other posts, but they have less than 1000 views and are therefore not included in this post. So if you want to get many visitors, just review 2 distributions, do a benchmark of SCMs, or something like this. If you don’t want many visitors, don’t write about such topics.

Copyright statements / Comments

MIT license: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - If you had a python module released under the MIT license, and this is in the comment of the module and you somehow ship only pyc or pyo files, you would be violating the license by not including the copyright notice, because these files do not contain the modules. This is also true for many other licenses, but this seems to be the best example.

If you include this in the docstrings, you would only violate such license terms if you distribute bytecode created with the -OO option. This also does not apply if the code is a program which prints the license (eg. via a commandline –license option).

GPL vs LGPL

Is there any difference at all? The LGPL requires you to publish all changes you make to the code, while the GPL also requires you to publish source files you have created. This also means that you can’t link a non-free program to a GPL library, but you can link it to a LGPL library.

Because Python modules are not linked to each other, everything you do is normally considered a use of the module. Therefore, if there is a module G released under the GPL, and a module X released under a different, incompatible license X, you would still be able to use the facilities provided by the module G. This also effects subclassing classes of G in X.

Due to the enormous flexibility provided by Python you can easily break the intented rules of the GPL. Instead of editing the class definition you subclass the class and edit it. You can also replace stuff inside the module G during run-time, simply by setting the relevant attributes.

In summary, Python makes it very easy to work around the restrictions of the GPL, therefore, using the LGPL instead of the GPL makes no sense. You can’t give others more rights than they already have. You would just make it easier for others in case they want to write new code and want to copy some of yours.

What about the AGPL?

The AGPL exposes (compared to the GPL3) further restrictions on using the software on eg. websites. It is intended for programs which may be used by SaaS providers. Like with the GPL, the enormous flexibility of Python compensates most of the restrictions the license.

BTW, which license to choose?

I normally choose to release my programs and modules, etc. under the terms of the GNU General Public License, version 3 (or at your option) any later version. But it also depends on the size of the project. When I work on small scripts, like hardlink, I generally choose the MIT license. This is also somehow related to the fact that I don’t want to have a license which takes more than 50% of the size of my project.

This is actually a bit different to what Bruce Perens does. Bruce recommends 3 types of licenses. The first one is what he calls the “gift” license. He recommends the Apache License 2.0, because it provides better “protection from software patent lawsuites”. The MIT license is another example for this type of license. While not providing the patent protection, this is not that critical for persons like me who live in Germany. Furthermore, the number of patents possibly infringed by the code is proportional to the amount of code.

The second type he recommends is a “sharing-with-rules” license, like the GPL 3. Like him, I mostly use this license for my code. Sometimes I also use the GPL 2, but only when I am required to do so, or because of tradition. In generally, I only upgrade software from GPL-2+ to GPL-3+ when I introduce new features, not for bug fixes or similar.

The third type he describes is the “in-between license”, like the LGPL. As I pointed out above, this type of license is not much different than the GPL, at least if applied to Python modules. Therefore, I never release any Python module under such a license. Things may be different for C libraries (and others), but I never released one.

Documentation, etc.

Well, I license all my documentation under the same license as the software. This makes it easier for the user because he does not need to read yet another license (at least if he reads all the licenses of the software he uses). If I distribute non-code content independent of code, I generally choose a Creative Commons License (CC-BY-SA 3.0, CC-BY 3.0), Germany.

This also has an effect on this blog. From now on, all content (ever) provided by me via this blog is licensed under the terms of the Creative Commons Attribution-Share Alike 3.0 Germany, unless a different license information is included as part of the post. The design and comments from other persons are not included.

Why I wrote this

Really, I don’t know. Maybe I just want to write something, maybe I want to write these things down, so I can read them. Anyway, please tell me if I my conclusions/ideas are wrong.

Update 1: There was a mistake “should expect the information to be correct”, fixed now: “should not expect […]”. I may be wrong with the GPL vs. LGPL thing, have not completely checked this. (2009-02-18 19:18 CET)

Update 2: Seems the GPL vs LGPL thing is not correct, as written by “Anonymous” and Bruce. (2009-02-18 19:26 CET)

Now I will present you the example. It is not much longer than the pool example I gave on monday, but it does much more - it creates a repository (and it has some commandline options). The concept of the Repository is very simple, too. We have two classes, Repository and Distribution, and a Repository object contains multiple Distribution objects, which are responsible for packages and related stuff.

#!/usr/bin/python
# Copyright (C) 2009 Julian Andres Klode.
#
# Released under the terms of the GNU General Publice License, version 3
# or (at your option) any later version.
#
"""Example to demonstrate the power of debimg's repository management.

This example creates a directory example.debimg, with two subdirectories:
    - pool: This holds debimg's file pool (see debimg.core.files.Pool)
    - repo: This holds the created repository (debimg.core.repository)

It creates a repository of all required packages, using your local apt sources.
"""
from __future__ import with_statement
import operator
import optparse
import sys

from debimg.core.resolver import Resolver
from debimg.core.files import Pool
from debimg.core.repository import Distribution, Repository

def main():
    """Called when the script is executed."""
    parser = optparse.OptionParser()
    parser.add_option('-u', '--user', help='Same as in GPG')
    parser.add_option('-c', '--contents', action="store_true",
                     help='Create Contents-*.gz files')
    opts, args = parser.parse_args()

    # Create a pool, which manages the access to the files.
    pool = Pool('example.debimg/pool')

    # Create a resolver using your local apt configuration.
    pkgs = Resolver()

    # Create a new repository
    repo = Repository(pool, 'example.debimg/repo')

    # Create a new distribution and store it in dist.
    dist = repo.add_distribution('lenny')

    # Add all packages with priority required to the resolver
    pkgs.add_priority('required')

    # Add the packages from the resolver to the distribution
    for group in pkgs.groups():
        for package in group:
            dist.add_package(package)

    pool.compact()   # Optional step
    pool.fetch()     # Fetch all the packages from the mirror
    dist.finalize_files() # Link the files into the repository
    dist.finalize_packages() # Create Packages files (uncompressed and gzip)

    ##
    # If we want to create Contents-*.gz files, we can do this as well:
    # but we don't want to
    if opts.contents:
        dist.finalize_contents()

    # Now create our Release files, and sign them using the key provided on
    # the commandline. If no key has been provided, do not sign it.
    dist.finalize_release(Suite='testing', key=opts.user)

    for file in sorted(pool._files, key=operator.attrgetter('uri')):
        print file   # Print information about every file, sorted by URI.

if __name__ == '__main__':
    main()

To try the code, checkout the temp/repository branch at git://git.debian.org/users/jak/debimg.git and execute the script named test-repository, which is the same as printed above. If you have questions, please ask. Also read the docstring of debimg.core.repository to see which changes are not yet done (eg. common base class for Distribution and Repository, and of course adding files manually).

There is also an improved version of this example, which creates a repository, but the needed module (debimg.core.repository) is not public yet, because its far from being finished. I expect to complete repository code on Wednesday.

#!/usr/bin/python
# Copyright (C) 2009 Julian Andres Klode.
#
# Released under the terms of the GNU General Publice License, version 3
# or (at your option) any later version.
#
"""Example to demonstrate the power of debimg's pool

This example creates a directory example.debimg, with two subdirectories:
- pool: This holds debimg's file pool (see debimg.core.files.Pool)
"""
from debimg.core.resolver import Resolver
from debimg.core.files import Pool

def main():
    """Called when the script is executed."""

    # Create a pool, which manages the access to the files.
    pool = Pool('example.debimg/pool')

    # Create a resolver using your local apt configuration.
    pkgs = Resolver()

    # Add all packages with priority required to the resolver
    pkgs.add_priority('required')

    # Add the packages from the resolver to the pool
    for group in pkgs.groups():
        for package in group:
            pool.add_package(package)

    pool.fetch() # Fetch all the packages from the mirror

    for file in sorted(pool._files, key=lambda k: k.uri):
        print file # Print information about every file, sorted by URI.

if __name__ == '__main__':
    main()

Because debimg.core does not depend on any specific configuration format, but is configured solely via parameters, it is more flexible than debimg 0.0. This new code enables people to write their own programs related to Debian images easily. And if you want to, you can use the facilities provided by debimg.configuration and debimg.frontend to write your own program using debimg configuration files (not implemented yet, core needs to be finished first).

Debimg 0.1 will use YAML configuration files, supports multiple repositories, and much more. We could even implement support for pinning packages.

Debimg 0.1 is more than the others. It is a library (actually a python package, debimg.core), it is a program designed for end-users and developers. The enormous flexibility allows us to create applications for almost everyone.

My vision is that someone who needs a custom Debian image simply fires up the debimg GTK+ frontend, selects the packages he/she wants and clicks build. And when someone needs more flexibility, there will be configuration files which lets you configure most aspects. And if this does not suffice, you can write your own application by importing the modules, replacing some functions, methods, etc. and simply call the main() function of the frontend.

The goals:

  * flexibility, high speed, and cool features.
  * be a show case for the features of the low-level python-apt bindings (apt_pkg, apt_inst)
  * high-quality code, no hacks, every single function/method/class/module documented.
  * be a library, and a program.
  * provide a graphical front-end to assist the unexperienced users.
  * provide powerful file-based configuration for advanced developers.
  * And finally, provide a replacement for debian-cd in the near future.

More:

  * Email: [http://lists.debian.org/debian-cd/2009/02/msg00034.html](http://lists.debian.org/debian-cd/2009/02/msg00034.html)
  * Vcs-Git: git://git.debian.org/users/jak/debimg.git
  * Vcs-Browser: [http://git.debian.org/?p=users/jak/debimg.git;a=summary](http://git.debian.org/?p=users/jak/debimg.git;a=summary)

If you want to target Lenny and you don’t have Etch in your sources.list, pin stable at a higher priority than testing. This means that the pin automatically migrates once Lenny becomes stable (which happens hopefully in 2 weeks).

I wrote** 2599 lines** of documentation. More detailed I wrote 2599 lines of documentation from Thursday to Sunday.

In total, the jak branch of python-apt now has a diffstat (compared to debian-sid) of:

Changes: 113 files changed, 5845 insertions (+), 2763 deletions(-)

Partly responsible is Ben Finney, whose patches I merged. I also closed 7 bugs not relevant anymore in current versions of python-apt.

This is the changelog to from debian-sid to jak:

python-apt (0.7.9~exp2) experimental; urgency=low

  * apt/*.py:
    - Almost complete cleanup of the code
    - Remove inconsistent use of tabs and spaces (Closes: #505443)
    - Improved documentation
  * apt/debfile.py:
    - Drop get*() methods, as they are deprecated and were
      never in a stable release
    - Make DscSrcPackage working
  * apt/gtk/widgets.py:
    - Fix the code and document the signals
  * Introduce new documentation build with Sphinx
    - Contains style Guide (Closes: #481562)
    - debian/rules: Build the documentation here
    - setup.py: Remove pydoc building and add new docs.
    - debian/examples: Include examples from documentation
    - debian/python-apt.docs:
      + Change html/ to build/doc/html.
      + Add build/doc/text for the text-only documentation
  * setup.py:
    - Only create build/data when building, not all the time
    - Remove build/mo and build/data on clean -a
  * debian/control:
    - Remove the Conflicts on python2.3-apt, python2.4-apt, as
      they are only needed for oldstable (sarge)
    - Build-Depend on python-sphinx (>= 0.5)
  * aptsources/distinfo.py:
    - Allow @ in mirror urls (Closes: #478171) (LP: #223097)
  * Merge Ben Finney's whitespace changes (Closes: #481563)
  * Merge Ben Finney's do not use has_key() (Closes: #481878 )
  * Do not use deprecated form of raise statement (Closes: #494259)
  * Add support for PkgRecords.SHA256Hash (Closes: #456113)

 -- Julian Andres Klode   Sun, 11 Jan 2009 20:01:59 +0100

…

This will again close 7 bugs. But the most important part is to actually have the complete documentation (OK, some descriptions are missing, but all classes,functions,methods,attributes,data should be there now).

I still need to rearrange parts of the documentation and add some descriptions to some pieces, but all in all I am satisfied with what I have done this week.

Take a look at the documentation: http://people.debian.org/~jak/python-apt-doc/apt/, it received documentation for 9 new classes today.

I also added an entry to the DeveloperNews page on wiki.debian.org.

introduced again some new documentation.

doc/source/apt/debfile.rst | 6 doc/source/apt_pkg/cache.rst | 540 +++++++++++++++++++++++++++++++++- doc/source/apt_pkg/index.rst | 24 + doc/source/conf.py | 5 doc/source/examples/cache-packages.py | 22 + doc/source/examples/cache-pkgfile.py | 29 + doc/source/examples/missing-deps.py | 51 +++

7 files changed, 659 insertions(+), 18 deletions(-)

You can see them at http://people.debian.org/~jak/python-apt-doc/

Missing seem to be:

• AcquireFile
• AcquireItem
• ActionGroup
• Configuration
• MetaIndex
• PackageIndexFile
• PkgManager
• PkgRecords
• PkgSourceList
• PkgSrcRecords
• ProblemResolver
• TagFile
• TagSection

But the weekend is not over yet. I will blog tomorrow again, and once I’m finished with everything (which may be tomorrow, too).

Have fun, read the documentation, find the mistakes. Some things are currently written as ‘???’, if you know what fits there, please tell me.

After Sandro Tosi told me in a comment in my last post that the real big problem with python-apt is a lack of documentation, I immediately started writing it. UsingreStructuredText and Sphinx, we now have a really cool and much more detailed documentation. (Although it is not really finished yet [it contains everything, but there is still room to improve]).

The whole documentation is available at http://people.debian.org/~jak/python-apt-doc/, and the source is in my branch at http://bzr.debian.org/users/jak/python-apt/jak", which can be browsed via Loggerhead at: http://bzr.debian.org/loggerhead/users/jak/python-apt/jak/changes

It also contains a lot of cleanup, whitespace removal (bundled in one commit), and improved docstrings. And apt.debfile and apt.gtk.widgets should work completely now. Oh, and apt.cdrom now supports sources.list.d.

Hello,

while hacking a bit on python-apt in the last days, I formulated some proposals which I want to present to you.

I have already implemented proposals I and VII, started with the implementation of proposal II, and have partially implemented proposal III, all in my local branch. I could implement the rest next week, and this could result in python-apt 0.7.9~exp2 (at least this is my wish, I do not have commit access to the python-apt repository, and this mail is not discussed yet:) ).

My local branch is not online yet, but I’m working on it. It is based off debian-sid revision 202 (which is 0.7.9~exp1).

I would have also suggest to (at least partially) follow PEP 7 (C code), but re-indenting the whole extension code is not a good idea.

Before I go on, I would like to know your opinions about these proposals, what you like and what not, your suggestions, etc, etc…

List of Proposals

 I   Deprecation of mixedCase naming conventions in apt module
 II  Re-organisation of the apt.progress module
 III Language updates
 IV  Unification of testing code
 V   Cleanup of the code
 VI  Using unique variable names throughout the modules
 VII Some other changes

I. Deprecation of mixedCase naming conventions in apt module

In order to comply with PEP 8, we should start to change our names from mixedCase to lowercase_with_underscores style. This style is more common in the Python world, and used eg. in the Python library.

I hereby propose that:

1. a new module apt.deprecate is introduced, which provides functions for compatibility and a class named DeprecateMixedCase
2. Most classes are changed to subclass DeprecateMixedCase instead of object.
3. All methods, attributes and variable names are changed from the mixedCase style to the lowercase_with_underscores style.
4. apt_pkg will be changed to call the new methods on progress objects, and fall back to the old names if methods with new names are not available.

How backward compatibility is implemented:

The DeprecatedMixedName class provides the methods getattr() and setattr(). These classes will forward all access to attributes with mixedNames to the ones with lowercase_with_underscores names.

In case of methods, the getattr() method returns a helper function which fixes all keyword arguments with mixedCase names and than calls the right function.

As this only works for methods with mixedCase names, methods which have mixedCase-named args, get the argument **kwds, and its content will be checked for mixedCase-named arguments. [On the other side, given that the API is marked as not stable, we could also drop that level of compatibility.]

Schedule:

1. Implement all the changes in an experimental version of python-apt as soon as possible.
2. (A) Once all packages have been adjusted to use the new names, remove the old ones (about 50 binary packages, and I would write the needed patches). This would Breaks: about 50 packages. or (B) After the release of Debian Squeeze, remove the compatibility functions (and the deprecate module from apt). This is the same as (A), but does not require Breaks on 50 packages.

Known Issues:

1. The methods will not be available using their oldName at class level, only at object level. This means that some code subclassing apt.* classes may fail. Most code will not break, however.

II. Re-organisation of the apt.progress module

The apt.progress module is not very consistent in the way classes are named. For example, there is OpTextProgress but TextFetchProgress.

Therefore, I hereby propose that:

1. For each of the for types of progress (‘Cdrom’, ‘Fetch’, ‘Install’, ‘Op’), a new class is introduced with the name: Abstract%sProgress % progress
2. For each type of progress a class Text%sprogress % progress is introduced, which subclasses Abstract%sProgress and provides textual output
3. For further classes, a general naming scheme is introduced, which is “%(output_type)s%(progress_type)sProgress” [whereas (1) and (2) implement this naming scheme]

Schedule:

Same as in proposal I.

III. Language updates

The apt package has been written some time ago, and has not been updated to make use of the latest language features.

Therefore, I propose that:

1. all code using old language features is adjusted to use language features as of Python 2.5. 1.1 Examples include the use of decorators, and the use of str methods instead of functions of the string module.
2. The with statement will be used where appropriate, and that apt_pkg.*Lock() gain support for being used as context managers.

Schedule:

Implement as soon as possible.

IV. Unification of testing code

Most modules ship test code which will be called when the module is run as a script.

Therefore, I propose that:

1. All testing code in the apt module is removed
2. All the tests in the ’tests’ directory of python-apt are converted to a unified testing suite.
3. Testing code which has been removed as part of (1) is reintroduced in the relevant test module, if appropriate.

Schedule:

Implement as soon as possible.

V. Cleanup of the code

In accordance to PEP 8, which is also named as a rule in apt/README.apt, code will be reformatted where needed.

This includes 2 lines between module-level definitions, spaces after ‘,’, comparing to None using ‘is’ instead of ‘=’, and more.

Schedule:

Implement as soon as possible.

VI. Using unique variable names throughout the modules

I hereby propose the following names:

 cache    - An apt.cache.Cache() object
 pkg      - An object returned by cache[str()] (in special cases)
 pkg      - An apt.package.Package() object
 cand     - An object returned by depcache.GetCandidateVer()

This would remove some confusion for developers, especially if you see that apt.Package() objects are named cand in debfile, but pkg everywhere else.

Schedule:

Implement as soon as possible.

VII. Some other changes

Here is a list of other changes I have already made in my branch:

• Remove apt.package.Record(), it behaves exactly like the object it is wrapping (apt_pkg.ParseTagFile() result)
• Move the Origin class out of the Package() class and at the module level.
• If a package should be marked for upgrade, but is not upgradable raise an exception (which will be apt.package.NotUpgradableError)

Schedule:

Implement as soon as possible.

File systems have problems. I can’t use ext3 on my laptop, because it would check the whole disk (120GB) every 30 boots. And this check takes a very long time (15 or 30 minutes, I do not know it exactly). XFS is better, but also problematic because deleting a big directory with a deep hierarchy takes a very long time like 1.5 minutes , whereas ext3 needs a few seconds. Also I still wonder why XFS needs no checks.

Therefore I might go with ext4. The time needed to check the partition seemed to be much shorter and the speed is about the same as with ext3. The installation will probably be: Install Lenny on a ext3 partition, install kernel with ext4 support (which is kernel 2.6.28), add ext4 to /etc/initramfs-tools/modules, tar the partition, format it as ext4, and untar the contents again; because converting does not bring the advantags of ext4 to already existing files.

BTW, I did some benchmarking with bonnie++, and XFS had about 20,000 operations per second, whereas ext3 had 60,000. ext4dev from Kernel 2.6.27 was a bit slower than ext3

I also tested BTRFS, but BTRFS is 1) not stable enough 2) as slow as XFS. It also shows horrible write performance of 19MB/s using dd, whereas XFS reached 42MB/s. BTW, benchmarking btrfs with bonnie++ only works using “-s 0”, i.e. without I/O performance test, else you receive a kernel bug. I subscribed to the btrfs mailing list, and sent my bug report email 4 times, but it never reached the list it seems. This bug also happens in other cases.

Creation of bootable USB stick: For both distributions, I created a bootable USB stick because it is faster to install from USB than from CD/DVD. The creation was easy in both cases. Fedora ships a shell script and Ubuntu a graphical program for this task. Both very equally easy to use.

**Booting from USB: **Both distributions booted perfectly from USB. But there was a big difference in boot speed: Fedora 10 booted in about the same time Ubuntu needed to find the USB stick.

Installation from USB to HDD: Installation of both systems went very well, without any problems. Fedora’s installer is far more powerful than Ubuntu’s, e.g. it allows to use LVM or encrypted partitions. To setup LVM and encrypted volumes with Ubuntu, you need to use the text installer on Ubuntu’s alternate disk (It’s debian-installer). Ubuntu’s graphical installer received a new partioning screen.

Boot: Fedora’s boot time seems to have improved a lot since the last release. Ubuntu’s boot time seems to be roughly the same as in 8.04. Both distributions provide a progress bar, whereas Ubuntu’s usplash looks better. But this may change once kernel-based modesetting works on Intel cards in Fedora, because Fedora’s splash program needs it for graphics. Else, it only displays ASCII progress bar.

Login: Fedora 10 shipped with GDM 2.24, which means that there are no themes yet. But this was no problem, as the non-themed design of GDM has improved a lot. I even like it more than any themed GDM I have seen. It reminds me a bit of OSX.

Desktop: Both distributions ship with GNOME 2.24. The default themes look good, whereas Fedora’s uses more GNOME-like icons, whereas Ubuntu uses Human, which is based on Tango. And that is important for me, since I normally do not change any aspect of my desktop, except for adding cpu frequency and tomboy applets to the panel.

Hardware support: Both distributions provide good hardware support. Ubuntu leads here, at the cost of having non-free drivers included on the disk.For users needing windows wireless LAN drivers, Ubuntu is the right choice, as it ships ndiswrapper and the graphical frontend (ndisgtk) on the installation medium. Fedora does not include ndiswrapper, because of what it is used for (installing non-free Windows drivers). Both distributions ship firmware, and consider firmware images as free enough, as they do not run on the CPU.

EXT4: Fedora 10 ships with patches for EXT4 backported from Kernel 2.6.28. It also supports the installation on ext4 partitions, but only using the DVD and if booted with the ext4 option. The Live discs do not support installation to ext4.

Package management: Both distributions ship with good package managment software. YUM really improved since the early times, where it downloaded an extra metadata file for every package. Nowadays, yum and apt seem to be on a nearly equal level, but aptitude lets Ubuntu win, because it can handle problems much better  (ie. it proposes ways to solve problems) than YUM.

Graphical package management: Ubuntu wins here. Fedora only ships PackageKit for package managment, which, in my opinion, is not able to really compete with Synaptic and gnome-app-install.

Conclusion: Both distributions are very solid. Ubuntu still has the better package managment, and a bigger selection of packages. This has only been possible due to Debian, which is the base of Ubuntu. I tried to write this blog post not as an Ubuntu member. I always liked Red Hat and Fedora, and one of my first Linux distributions was Red Hat 9.

Links:

  * [http://www.heise-online.co.uk/open/What-s-new-in-Fedora-10--/features/112093](http://www.heise-online.co.uk/open/What-s-new-in-Fedora-10--/features/112093)
  * [http://www.heise-online.co.uk/open/Ubuntu-8-10-first-tryout--/features/111823](http://www.heise-online.co.uk/open/Ubuntu-8-10-first-tryout--/features/111823)
  * [http://fedoraproject.org/](http://fedoraproject.org/)
  * [http://www.ubuntu.com/](http://www.ubuntu.com/)

OpenSolaris seems to have big problems in terms of wireless LAN. I am using a WPA/WPA2 encrypted wireless LAN, and it asked to me to enter my passphrase. I entered it, and nothing happened. I was able to get wireless working via commandline, but this lead me to the problem that domain names were not resolved. I could solve this, too. But none of this was permanently, and it should work the graphical way or tell me it does not work. Maybe this whole thing happened because I have a space in the name of my Wi-Fi network.

Another problem is suspend/resume, which is not yet working for most laptops; whereas recent Linux distributions support suspend/resume out-of-the-box.

OpenSolaris ships non-free components, e.g. NVIDIA display drivers, or Java 6 Update 10. Hopefully, there will be an option in future to install it without this stuff.

Talking about the desktop, there is GNOME 2.24 with a custom theme. In my opinion, the theme looks quite good, although I do not like the colours of the minimize,maximize,close buttons of the windows sometimes.

One feature introduced with this release is **Time Slider. **Time slider worked very well, although for  reasons I do not know, I do not really like it. The integration with nautilus is good, but I would still like to see an interface using a calender widget instead of this slider. I remember that the slider sometimes did not work as I expected, e.g. it jumped from left to right.

The package management is interesting, but it is to hard to use in my opinion. Debian and derived distributions provide far better package managment in my opinion. Especially the graphical part needs a lot of work in order to compete with Debian.

The overall speed was quiet good, and memory usage was 512MB. There is one problem with measuring memory consumption in gnome-system-monitor: I installed netbeans 6.5 and started it, and gnome-system-monitor showed a memory usage of around 20MB, while the program used more than 200MB (according to top).

All in all, OpenSolaris gets closer to modern Linux distributions like Fedora 10 or Ubuntu 8.10, but it is not there yet. But with improvements in the area of Wi-Fi and Suspend/Resume,  future releases of OpenSolaris might become a good alternative to Linux distributions, at least if you do not really care about the ideas of free software.

**Links: **

  * [http://www.heise-online.co.uk/open/OpenSolaris-2008-11-tested--/features/112274](http://www.heise-online.co.uk/open/OpenSolaris-2008-11-tested--/features/112274)
  * [http://www.opensolaris.org/](http://www.opensolaris.org/)

The URL of the feed is http://juliank.wordpress.com/feed/atom/, replace juliank with the name of your blog.

First of all, this release includes patches written by Jeff Mahoney jeffm@suse.com. These patches enable the usage of NFSv2 and NFSv3 file systems as branches, even on the standard Debian kernel. The package does not support NFSv4 yet, but will support it at a later point (but only with a patched kernel) - the code exists, it’s just not enabled yet.

This release also removes support for pre-lenny Kernels (<2.6.26) and uses quilt instead of dpatch. It also removes any AppArmor support for the moment, but this may be added back later.

On Mon, Oct 13, 2008 at 10:51:43PM +0000, Martin Zobel-Helas wrote:

[ This is a long mail with important information, so please read it all    carefully. ]

Dear Julian Andres Klode!

Your account ‘jak’ has just been created in the central LDAP database of the Debian project.  […]

Thank you everyone (in chronological order):

  * Daniel Baumann (daniel) - Sponsored my first packages
  * Joerg Jaspert (joerg) - The first DD who signed my key
  * Niv Sardi-Altivanik (xaiki) - Advocated me for NM
  * Martin Zobel-Helas (zobel) - My first AM
  * Alexander Reichle-Schmehl (tolimar) - The second DD who signed my key (at CeBIT)
  * Bernd Zeimetz (bzed) - Took over in June, because zobel was very busy
  * Christoph Berg (myon) - For checking the AM report and requesting the account creation
  * Jonathan McDowell (noodles) - For adding me to the keyring
  * Martin Zobel-Helas (zobel) - This time for creating my account

And many thanks to Google - for helping me to find answers to important questions. Without you, I would know nothing. And of course all the others who helped to make this possible.

The path chosen by binary distributions like Debian is much better. Packages are compiled centrally and users download and install these pre-compiled packages. But it is also much less flexible in terms of which functionality is supported. We, as package maintainers, have to decide which functionality is likely to be used by the users. In Contrast, Gentoo has USE flags, which allows the user to enable the specific functionality he/she wants.

Should we combine the advantages of both technologies? Provide a normal binary distribution as we do now, but also provide a framework for users to easily recompile their packages, adding new functionality.

Imagine a combination of all these technologies:  User wants to install software X with feature F. The package maintainer has not enabled F in the binary. The user opens the website for his personal archive, selects the F flag and the software and clicks on build. The server builds the software and notifies the user once the binary is ready.

At Ubuntu, the first step has been made into this direction. While the package format is still the same, Launchpad allows users to create so-called PPAs (Personal Package Archives), where users can upload their source packages, which are then compiled and made available to other users. Let’s say Project A releases version X of their software. They can create a source package for it, upload it to the PPA, and provide the binary packages to the users. Prior to PPAs, most users would have used an old version, compile their own version or use tarballs containing pre-compiled software (mozilla does this).

And sorry for the title. I respect the Gentoo community and the Gentoo developers, they are doing great work there. And ebuilds and Gentoo’s boot system (with this runscript stuff) is really easy.

BTW: I also like the way Gentoo displays its boot messages. It looks much more organized than Debian’s boot process. (In terms of the message logging, where Debian writes “done” when something is done, and Gentoo writes “[OK]” to the right side of the screen, in green letters. (This is partially possible in Debian using lsb-base-logging.sh, but not all init scripts use these functions, or they use the wrong ones.)

  * **10 years: **Google (September), Open Source Initiative (February)
  * **15 years:** Debian, FreeBSD, NetBSD
  * **25 years:** GNU
  * **35 years:** Ian Murdock (April), Mark Shuttleworth (September)
  * **55 years:** Richard Stallman (March)

MJ Ray asked why people do and dont join SPI,  and received answers.

Not knowing SPI is no good reason, at least if you are participating in Debian’s development, and you should really know SPI if you are a Debian Developer.

For me, personally, not joining the SPI would not make any sense. The existance of other organizations is no good reason. Look at me, I am also a Fellow of the Free Software Foundation Europe.

There is only one reason which seems to be a bit right: It’s hard to join SPI. The membership information is located at ‘About -> Membership’ in the menu. Maybe this should be made a top-level menu-item and a big link “JOIN!” should be put on the homepage.

I see no reasons for anybody to not join SPI. All you need to join is a valid email address and interest in free software.

To use it, add the following line to /etc/apt/sources.list and install swfdec-mozilla0.7.

deb <a href="http://ppa.launchpad.net/juliank/ubuntu">http://ppa.launchpad.net/juliank/ubuntu</a> <span>hardy</span> main

Please note that these packages are experimental and are only for experienced users.

This version of dir2ogg will be much much smaller, and is really easy to extend. It will also support multiple output formats and does not need to know about the input files (use -i ‘SHELL PATTERN’ to include other files).

But it will not really replace dir2ogg. The old dir2ogg will still be supported.

Warning: Do not use the mp3 plugin from fluendo.com

swfdec does not always work with the binary mp3 plugin from fluendo.com, at least not in Ubuntu 8.04 64bit. It works with the version shipped in Ubuntu (gstreamer0.10-fluendo-mp3) and with the mad plugin in gstreamer0.10-plugins-ugly. An example is http://youtube.com/watch?v=CSp7jsV7oG0

Installation of missing gstreamer plugins

This works on both, Debian and Ubuntu, provided gnome-app-install is installed (default in Ubuntu). Simply open a youtube video, and you will be asked for searching the missing codecs. This also works in almost any other gstreamer application, like rhythmbox or totem.

Builds of the development branch

I will provide builds of the 0.7.1 git versions of swfdec and swfdec-mozilla in my PPA soon. This version even supports fullscreen mode in youtube! The PPA will build versions for hardy and intrepid.

The release is available in my PPA at:

deb http://ppa.launchpad.net/juliank/ubuntu hardy main
deb-src http://ppa.launchpad.net/juliank/ubuntu hardy main

I will also provide some builds of the development branch (0.7.1) soon.

**Update:**Maybe it was not clear, but the videos crashed the browser. It also didn’t really work in totem and I’m still looking for the source of the problem. I have every available gstreamer plugin package installed. Try it yourself, http://de.youtube.com/watch?v=CSp7jsV7oG0. It works with gnash, but gnash does not work with wordpress stats page.

Update 2: I now found the real problem: I had an binary version of fluendo’s mp3 plugin installed (from fluendo.com), which caused the crash. After removing the file, everything works. (using fluendo-mp3 shipped with Ubuntu).

Update 3: I removed the ffmpeg version from my PPA, as it is not needed.

First of all, debimg’s set support is almost finished. I uploaded a tarball containing the differences between the official lenny weekly build from yesterday and a build created today by debimg, using the tasks of debian-cd 3.0.4 (after manual conversion to a format supported by debimg). Look at http://jak-linux.org/cdimage/tests/ for the tarball.

Secondly, the dependency resolver has been rewritten. It’s a bit slower now (0.72 seconds for main), but creates much better results. Resolving the dependencies of all packages in Debian Lenny i386 in alphabetical order, debimg 0.0.X resolved 206 dependencies differently than apt. Now, these have been decreased to 15 dependencies, whereas 13 dependencies are false-positive (some packages were not installed because they were already installed). This means that only two ones were different, in this caseachims-guestbook and chdrv, which both depend on virtual-only packages (achims-guestbook: apache | httpd, chdrv: console-utilities).

The third big change is the addition of the hooks module. This module allows you to hook in custom functions, which have access to the Configuration object (ConfigObj) and the MediaSet. There are currently three types of hooks: pre_hooks (run before fetching packages, adding files to the disk), mid_hooks (run after the packages have been fetched) and post_hooks (run after the image has been built). Hooks can be added based on project and architecture, using a simple syntax which support shell patterns. (It’s ‘project/arch’). The hooks module uses python decorators to register functions. debimg 0.2 will switch to hooks for internal functions, too, like bootloaders and other stuff.

The code has not been merged into the master branch, but I will hopefully be able to merge it tomorrow. The release of debimg 0.1 is now planned for this weekend.

Support for disk splitting

This release of debimg adds support for creating media sets, i.e. splitting the packages over multiple disks. This is achieved by providing a new class called MediaSet, which passes all calls to methods to all medias.

When a package is added to a MediaSet, MediaSet tries to find the first disk where it can be added, by checking if all dependencies of the package are provided on the disk (or previous ones). This dependency checker does no recursive dependency checks, and can be disabled via OptimizedDiskSplitting = False (or no, n, 0…) in the configuration file.

The MediaSet classes are also lists, so you can access disk one via the index 0. BaseMedia has been modified to also support being accessed via index, using disknumber-1 in order to have both classes share the same API.

New data lists

This release of debimg brings users a more powerful way to include packages. Instead of various keys to include packages, debimg now uses the Include key.

To include packages, you have to use the ‘Include’ field. The value is a list of items, separated by commas. An item may either be the name of a package or a special form written as key:value. In this case, the following keys are supported:

Task/Priority: This includes all packages belonging to the specified task or priority. In case of tasks, debimg first includes all Key packages, then the other packages.

Configuration files are easier

This release of debimg makes it possible to build multiple architectures using the Architectures option. This option replaces the previous Architecture option, which is now set automatically by the script for each architecture.

It also enables the Projects option, in case you want to build multiple projects. (All need to have the same MediaType and same NumberOfDisks). The project may also be cd-set, dvd-set, dvd9-set, netinst or businesscard. In these cases, MediaType is set automatically. [MediaType may also be moved in to data files].

This release also allows you to prefix any configuration value with path: in order to automatically convert the value to an absolute path. This is needed for many file options, but should not be used for some other stuff.

Improved Jigdo Support

debimg 0.1 allows you to define the public path (name) to (of) the image file and the template file that is written to the Jigdo file.

Use urlgrabber for file downloading

Starting with this release, debimg uses urlgrabber to download all kinds of files (except packages). A urlgrab call has been added to BaseMedia.addFile() which now understands http:// and ftp:// urls.

Plugins (maybe)

debimg 0.1 may introduce support for registering custom functions. This feature is low-priority, although required for easy development of Ubuntu-related code.

About debimg

debimg is a GPL-3 licensed software designed to replace debian-cd, a tool to create Debian images. For further information about debimg, visit the Wiki page. For more information about Debian, visit the website at www.debian.org.

Now, some benchmarks (using IPython):

In [1]: a = range(10**6)

In [2]: b = set(a)

In [3]: %timeit 10**6 in a 10 loops, best of 3: 31.8 ms per loop

In [4]: %timeit 10**6 in b 10000000 loops, best of 3: 98.6 ns per loop

1ms are 1 million ns. Therefore, using sets is about 322515 times faster than using lists (or tuples).

debimg can now calculate dependencies in 0.5 seconds again, instead of 1 minute with lists.

It should be noted that the e-mails are fetched with a single account, as they are on an imap server, and because ubuntu.com is a forward address.

Is there any way to make Evolution respond with the e-mail address at which I received an email?

BTW, work on debimg 0.0.4ubuntu1 (previously 0.0.3ubuntu1) has reached the first milestone: Support for selecting packages based on the output of Germanite. This weekend, I will release debimg 0.0.4 and debimg 0.0.4ubuntu1 (or I merge it directly, let’s see).

Update: This was just some mistake on my side. I get emails from an Ubuntu mailing list and want to send emails to it using the ubuntu.com address. It works for normal emails.

  1. Change debimg to use germinate to calculate dependencies (package lists)
  2. Add the additional stuff (live, etc.)

Once we can recreate the Ubuntu hardy i386 and amd64 alternate disks, work starts on the live filesystem and on merging these features back into debimg master, which will also get support for more archs.

debimg uses germinate directly on the Python level.

debimg 0.0.3ubuntu1 is sheduled for this Sunday. This will be more or less really hacks.

  * [master] Move the fetching of packages from packages to media, so we can use it for all files
  * Add libdebimg.germinate as a wrapper around germinate, providing functions to build the disks
  * Modify libdebimg to build all disks in one run, multiple architectures and multiple seeds.

The basic code structure will look like the following:

  1. For each architecture:

    1. seeds = Run germinate
    2. For seed in seeds:

      1. Get the packages
      2. Get extra files
      3. Build the disk

The Debian version may switch to seed files too, in version 0.2.

GNOME 2.22 (Python) / Updated Packages

At the end of march, I updated some GNOME packages. These packages were gnome-python and gnome-python-desktop. The upload of gnome-python-desktop was really important, because the old version depended on libtotem-plparser7, which was not available anymore, and FTBFS because the metacity API changed. This upload made other packages building and running again!

That time, I also uploaded new releases of dir2ogg and ndisgtk, which fixed some bugs.

On the first of April, I updated aufs to a new upstream snapshot, which fixed linux-modules-extra-2.6’s FTBFS on armel, removed bashism in shell scripts, added a hack for limited splice support, and enabled building on -rt kernel (if the required functions are exported).

Packages To-Do

In the next week, I will upload a new aufs snapshot with support for kernel 2.6.25 and re-added support for kernels < 2.6.23. I will also update app-install-data to the current state of the archive.

Other stuff includes my ITA upload of gimmie and the upload of jockey, a tool to install drivers. Jockey will also be modified to provide the functionality from ndisgtk, which development has been discontinued as it is feature-complete (bug fixes will still be provided). Another package will be, of course, debimg 0.1 once it’s released and of course the python-libisofs bindings.

Ubuntu packages

I requested syncs for dir2ogg and ndisgtk (after I uploaded sync’able versions to Debian) and for aria2, which was not installable before. I have also reported some bugs and used 8.04 for some time.

debimg stuff

I released debimg 0.0.3 on 4th April. This is the first release to require Python 2.5 and also the first release which uses the new media module, which provides a generic interface to disk creation.

I actually have not worked on debimg since that day, mainly because I did not have enough time. In May, I hope to add support for more architectures (at least theoretically, by providing a generic way of handling bootloaders and other non-packages and non-dists files) and release 0.1.

debimg 0.1 will not contain any features related to python-libisofs, because the focus is getting the basic functionality.

I have also not uploaded any new netinst build on jak-linux.org, since March.

debimg and the Debian Project News

Debimg will appear in the second issue of the Debian Project News, I’m currently working on the text for it. (BTW, I have also fixed one link in the first issue of the DPN)

debimg and Ubuntu

I have suggested to switch Ubuntu’s image building to debimg, and offered to do everything needed to do this. Given the speed of debimg and that the current features almost match the requirements of Ubuntu, this seems to be a great idea.

python-libisofs

As you may know, I released a first preview of the python-libisofs bindings some days ago, with almost complete support for creating image files. The next steps will be reading and growing images and of course, the bindings for libburn and libisoburn.

Switching from Ubuntu 8.04 to Debian unstable

I am currently working on migrating my laptop from Ubuntu 8.04 to Debian unstable, in order to be able to work better on my packages. Since most of my packages are sync’able now, this enables more efficient development. Another reason is to help the lenny release. This does not mean that I will be less active on the Ubuntu side, at least not much.

Let’s make Debian Lenny and Ubuntu Intrepid the best releases ever.

I tested this on my notebook, a HP Compaq 6720s with Intel X3100 (GM965) graphics controller.

I downloaded the preview live image for x86_64 and booted with the i915.modeset=1 option. The boot was almost normal, except that it was flicker-free. After the system booted I switched the virtual terminal from Xorg to tty1 and back, and it was extremely fast. The terminals all had the same resolution.

This does not mean that everything is perfect. There are a lot of things which do not work. For example, running glxgears and moving the window does not “look good”. Also, speed was a bit low. Normally, in Ubuntu Hardy, I get about 1000 FPS, in Fedora I only got 230 FPS. But the difference was not visible.

Another thing which did not work was suspend & resume, because the graphics card seemed to be not correctly re-initialized. This is worse than Ubuntu & Debian. In Ubuntu, suspend & resume works almost every time. In Debian, it works except that the backlight is disabled in X.

If these problems can be fixed, it would be very interesting to get this feature into Ubuntu Intrepid, maybe not as the default, but as an option for people who want to test it.

Fedora 9 seems to be a fast and stable distribution, with experimental features for experienced users and developers. It also provides a nice application for configuring PulseAudio, like per-application volume settings, a functionality I miss in Ubuntu.

The bindings support the creation of ISO Images and (almost) all options libisofs supports, like Rockridge, Joliet, and much more. Reading and Modifying existing images is not supported yet.

The code is written in Cython and you need cython installed for building from the git branch. It can be installed just like any other Python module/extension/package, using a setup.py.

Browse: http://git.debian.org/?p=users/jak-guest/python-libisofs.git Get: git clone git://git.debian.org/git/users/jak-guest/python-libisofs.git

I’ll package it for Debian within the next weeks after some further tests.

For those of you who don’t know about them, Asia is a band formed in 1981, including former members of Yes, King Crimson, and Emerson, Lake & Palmer, Uriah Heep, U.K., and The Buggles.

The most popular song of Asia is “Heat of the Moment”, from their debut album which spent 9 weeks at number 1 U.S. album chart.

This new album is excellent, featuring songs like the rocker “Never Again” and “Extraordinary Life”. A must-have for all Asia fans out there.

The new album will be released in the United States on April, 15 and in Japan on April, 23. It can be bought from Amazon.com for $9.99 and in Germany from Amazon.de for 15.97€. (See your own Amazon for your local price.)

BTW, the Album is already #20 in Amazon.com Music Bestsellers.

Visit the band’s Website, and buy your own one from your favorite shop.

The third release of debimg is available now: 0.0.3

Get the tarball: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz Verify it: http://alioth.debian.org/~jak-guest/debimg_0.0.3.tar.gz.asc The ChangeLog: http://alioth.debian.org/~jak-guest/ChangeLog-0.0.3

Clone git repo: git://git.debian.org/git/users/jak-guest/debimg.git Browse the repo: http://git.debian.org/?p=users/jak-guest/debimg.git

More Information: http://wiki.debian.org/DebImg Daily images: http://jak-linux.org/cdimage/daily-builds/testing/

About debimg

debimg is a software designed to replace debian-cd, written in Python, and supporting the creation of single disks for the i386 and amd64 architectures.

debimg is of course free software and licensed under the terms of the GNU General Public License 3 or (at your option) any later version.

About “the checksum”

This release is called “the checksum”, because the software knows the MD5SUM, SHA1SUM and SHA256SUM of every file on the disk.

News

  * Introduction of the media module

    * Add md5sum.txt and sha1sum.txt to the image
    * NEW OPTION: JigdoMap, see the config file
    * Lot of code cleanup


  * Support for custom installer images

    * Modify InstallerImages to an url supported by apt


  * Renamed most options in the configuration file for better

readability * debimg requires Python 2.5, as it uses the with statement.

Description of the Release

This release of debimg introduces the media module with its classes MediaFile, BaseMedia and DebianMedia.

The MediaFile class contains information about a file. It contains the absolute path to the file on the filesystem, the path on the media, its size, md5sum, sha1sum and sha256sum.

The BaseMedia class provides methods to add files to the image, open files on it, and creating the final image. It also provides methods to create the files md5sum.txt and sha1sum.txt, and methods to support Jigdo file creation.

The DebianMedia class provides methods to add packages to the image, creating Release and Packages files for dists.

Quick start

To get started with debimg, get the tarball and extract it to some directory.

Now, open the file debimg.cfg and change the option Mirror to the URL of your preferred mirror. This mirror may be any kind of mirror supported by apt, but if you use file:/ they have to be on the same mountpoint, as the files are hardlinked. (use copy:/ to get them copied).

Now, run ./debimg debimg.cfg. This will create an ISO image named debian-lenny-i386-netinst.iso. This image is a normal netinst (except for missing documentation and some other small things), and contains the Lenny d-i.

To change more settings, take a look at debimg.cfg.

Future

  * Cleanup of the configuration format (almost finished)
  * Support for splitting disks

    * Introduction of MediaSet class
    * Changes to the lists required


  * Add documentation to the disks
  * Support for PowerPC
  * Create Debian package (almost finished)

Another interesting feature will be the libisofs [0] support provided by the python-libisofs extension, which is currently under development. [1]

Previous release announcements

  * 0.0.2: [http://lists.debian.org/debian-cd/2008/03/msg00114.html](http://lists.debian.org/debian-cd/2008/03/msg00114.html)
  * 0.0.1: [http://lists.debian.org/debian-cd/2008/03/msg00021.html](http://lists.debian.org/debian-cd/2008/03/msg00021.html)

Links

  * [0] libburnia project: [http://libburnia-project.org/](http://libburnia-project.org/)
  * [1] [http://juliank.wordpress.com/2008/04/02/python-extension-for-libisofs/](http://juliank.wordpress.com/2008/04/02/python-extension-for-libisofs/)

At a later point, this extension will be used to create the ISO Images in debimg. It will be disabled by default, but you can enable it via a configuration option.

In [1]: a = ‘a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z’

In [2]: timeit a.partition(’/’)[0] 1000000 loops, best of 3: 362 ns per loop

In [3]: timeit a[:a.find(’/’)] 1000000 loops, best of 3: 443 ns per loop

In [4]: timeit a.split(’/’, 1)[0] 1000000 loops, best of 3: 697 ns per loop

In [5]: timeit a.split(’/’)[0] 1000000 loops, best of 3: 1.45 µs per loop

These tests were made with IPython 0.8.1 (Python 2.5.1 (r251:54863, Oct 5 2007, 13:50:07) ) on an Ubuntu 64 bit system.

In order to get support for disk splitting, I introduce three new classes: Media, MediaSet and MediaSetCollection (all in libdebimg.media).

The Media class is responsible for adding files to an image (incl. copying/linking), generating extra files (like MD5SUMS or documentation) and for building the image. It knows the space which is free, the size of each file and the md5sum of each file. It also writes all data inside dists/

The MediaSet class contains one or more Media instances. It has the functionality to split the packages into the disks.

MediaSet knows the dependencies of all packages and tries to add it to the disk with smallest number (e.g. 1) if there is enough space left for the package and all dependencies. If a dependency is on disk2, the package will be added to disk 2 or later. (If one dependency is on the last disk, we don’t check where the others are, instead, we put it on the same disk [faster])

The MediaSetCollection class contains multiple MediaSet instances (e.g. for DVDs and CDs), and calls their functions. This makes it possible to build multiple MediaTypes at the same time in an efficient way, because we don’t need to call debimg multiple times or run in multiple loops.

This means that many methods of libdebimg.packages.MirrorBuilder move into the new module. For example, the writePackagesFiles() and writeReleaseFiles() will be part of media.Media. (and can be called from MediaSet [for every Media] and MediaSetCollection [for every MediaSet], too)

From boot, run_genisoimage() will be removed. It is replaced by the createImage() method of media.Media.

In future, debimg will be able to run in three modes: simple, set and sets. The simple mode only creates one disk and is used for netinst and businesscard images. The set is used when only one MediaType has to be generated. The recommended mode for building e.g. weekly-builds is sets, as it has support for multiple MediaTypes.

Another change is the change to the format of data/*.list. I remove the keys called Priority and Tasks. Instead of these, I will add support for these directly into Depends. This means that you will be able to use “Priority:standard” just like “pkgname”. This makes sorted lists possible, which are needed for disk splitting. I will also rename the package list to match the names used in debian-cd (mostly) and add a new field called Single-Disk: yes, which disables sorting of the list, and is a bit faster than the other one.

This feature code is unreleased at the moment, but will hopefully be ready next week. Please note that I will never build complete sets with debimg. Therefore, I added a mode to create a list of all packages and build the image without them. (I have no local mirror)

Next week will also be the start of the first daily-builds of PowerPC images and businesscard images available from jak-linux.org/cdimage.

If you want to use the current release (0.0.2), a good point to start is a mail I wrote:

http://lists.debian.org/debian-cd/2008/03/msg00116.html

With these new features, debimg is now able to produce netinst images in less than 4 seconds.

BTW, you can easily build your own disk. Simply unpack the tarball, change the mirror to your one, and run ./debimg debimg.cfg. This should give you an i386 netinst disk.

Get debimg 0.0.2 from http://users.alioth.debian.org/~jak-guest/debimg_0.0.2.tar.gz

Fetch daily-built images at http://jak-linux.org/cdimage/daily-builds/testing/.

For more information, see http://lists.debian.org/debian-cd/2008/03/msg00114.html

deb <a href="http://ppa.launchpad.net/juliank/ubuntu">http://ppa.launchpad.net/juliank/ubuntu</a> <span>gutsy</span> main

I wanted to try it, but I had no time to compile it, so I uploaded it to the PPA. The next day, I had a compiled backport.

I don’t use it actively because it does not support some sites I visit.

Another news is the recent addition of ndisgtk to the ship and ship-live seeds, which means it will be available on the disk. http://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/ubuntu.hardy/revision/1223.

BTW, recent Ubuntu Hardy images (including Alpha 6) have support for using aufs instead of unionfs, simply add union=aufs to the kernel options. It may help if you have problems with unionfs and should be faster and more stable.

debimg is a new software which aims to replace debian-cd. No code has been released yet, but will be in the next days.

Please note that I can only provide jigdo images, because of bandwith and storage.

I will soon add build logs, and files listing the differences compared to original netinst images. I will also add support for more architectures.

debimg homepage at jak-linux.org

Currently, gimmie uses Apps/Tools, but what should I use now? It does not fit into the other sections. Recommendations?

Other important changes:

  * I am the new maintainer
  * Gmenu and sexy extensions are not built, instead the python-gmenu and python-sexy packages are used
  * Gimmie's extensions are compiled for Python 2.4 and Python 2.5
  * debian/copyright uses the format defined at [http://wiki.debian.org/Proposals/CopyrightFormat](http://wiki.debian.org/Proposals/CopyrightFormat)
  * The suspend and hibernations buttons are back again ([Commit](http://git.debian.org/?p=collab-maint/gimmie.git;a=commit;h=485f334993753f9afb98ed11e686abbe08abff36))

The package is maintained in a git repo in the collab-maint project, and can be viewed online using Gitweb. You can get the source from its repo at git://git.debian.org/git/collab-maint/gimmie.git

To create the orig.tar.gz tarballs, you need to have pristine-tar installed. To build the package, install git-buildpackage (from unstable) and pristine-tar and simply run ‘git-buildpackage –git-pristine-tar’

The same instructions also apply to building readahead-list, which can be found at git://git.debian.org/git/collab-maint/readahead-list.git

I am currently working on a pristine-deb program which will workaround all these issues. It will be able to work with git repositories like pristine-tar and uses pristine-tar for control.tar.gz and data.tar.gz. I may also add a function to decompress all compressed files (like changelog.gz) and uses pristine-gz to compress this file again before adding it to the tarball. This may save some additional space.

BTW, git-buildpackage now supports pristine-tar. Simply enable it with “pristine-tar = True” in gbp.conf. Thank you Guido Guenther, for adding these patches.

debimg’s main features are

Speed

debimg should be able to build netinst disks in less than 20 seconds

Free Software

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Easy Configuration

  * Packages can be included based on their priority, tasks or name (no manually created task files)
  * All packages are defined in one file,which is based on the format on debian/control. You can even use the "package [architectures]" syntax from Build-Depends

Based on python-apt

  * Highspeed dependency solver written in C (or C++ ?)
  * Used for downloading all kind of data in debimg
  * Packages{,.gz} files are created from the apt cache instead of scanning the disk

many features are missing at the moment, and no code has been released. I will send an email containing more details, like configuration file examples, to the MLs in a few days.

debimg homepage at jak-linux.org

Another upload today was ndisgtk 0.8.1-1ubuntu1, bringing Ubuntu up-to-date with Debian and closing 3 bugs.

BTW, I requested to include ndisgtk in main, see https://wiki.ubuntu.com/MainInclusionReportNdisgtk. Having ndisgtk on the Ubuntu disks would be very useful for users without linux network drivers available and without enough experiences to use ndiswrapper from the commandline.

I currently have some small issues with the website: After using –rebuild, tarballs have a different modification time and are therefore reuploaded everytime, though the content is actually the same. I would also need a plugin to create directory indexes with sha1sums, but I haven’t found one yet. (If you know one, please let me know).

If you have never heard of ikiwiki: ikiwiki is a wiki system used and developed by Debian Developer Joey Hess and many contributors. In contrast to ftpsync, a python script I wrote in 2007, it has an active upstream with multiple contributors, and supports multiple markup languages (e.g. Markdown).

This switch to ikiwiki is also a switch back to the design used at mentors.debian.net, which had been replaced in December 2007 with a (CC-licensed) design created by styleshout.com. This template just looks better and has the better license (GPL-2).

In the next days, I will cleanup and extend the website and the template and release both under the terms of the GNU GPL version 2. I will alsoI will also try to write a plugin for directory indexes, write a plugin to create directory indexes (although I have almost no experience with perl), and will release it under the GPL 2.

Thank you, Joey and all contributors for creating this software.

As some of you may know, the readahead-list package is now maintained in a git repo in the collab-maint project. I decided to use git instead of bzr (which I used for everything before), because of its speed and because I wanted to learn more about git, how it works.

I used git-import-dsc to import the first revision, and used debdiffs from 1 to 2 and from 2 to 3 to import the next revisions. Afterwards, I run git-import-orig on the new upstream tarball, which I downloaded and recompressed. Then, I did the packaging changes, added them using ‘git add changed-file’ and committed them using git commit.

After I had done this, I read Planet Debian and saw Joey Hess’s post about the new features of pristine-tar 0.5, i.e. the integration with git. Running Ubuntu at the moment, I fetched the source package, built it and installed it.

I then opened a shell in my git repo and ran pristine-tar commit path-to-orig upstream/0.20050517.0220 to import the delta for the first tarball. Afterwards I did it for the second tarball.

Because I use git-buildpackage to build the package and Joey said he would like to see support for pristine-tar in git-buildpackage, I then wrote a patch for the programs in git-buildpackage to import and export the orig.tar.gz when needed. The patch can be seen in gitweb, and the maintainer responded in Bug#463580 will integrate the patch with some minor modifications.

Both git and pristine-tar are great works, and it makes it so easy to maintain the readahead-list package. The combination of git, git-buildpackage and pristine-tar is the most powerful I ever used to maintain a Debian package, especially when you are not upstream.

One of them was ndisgtk 0.8.1, which adds more translations and fixes some problems. I intent to create a 0.9 release soon, with support for PolicyKit. I will upload the release candidate of this version to Debian experimental and Ubuntu Hardy.

The other upload was readahead-list 1:1.20060421.1016-1, which I found on the Gentoo mirrors, but not where previous releases were located. This is OK, because the upstream author is a Gentoo developer and uploaded this tarball himself. I already requested to sync this release into Ubuntu Hardy, to reduce the diff between both distributions.

In both uploads, I changed the format of debian/copyright to match http://wiki.debian.org/Proposals/CopyrightFormat. I will also change my other packages in their next uploads, and hope that other maintainers also use the new format, especially for NEW packages. I some cases (especially when you are upstream + packager), this format is also shorter.

For more information about the Concept of Debian Maintainers, see http://wiki.debian.org/Maintainers

I’m very happy about this, as it makes my life easier. But this does not mean that I don’t want to be a DD anymore. A DD is able to vote, to sponsor packages, to upload NEW packages, and many more things. (BTW: I’m currently at T&S step)

But this is not the only thing. 5 patches from the aufs source have been commited to Ubuntu Hardy’s kernel repo.

  1. [Export __lookup_hash](https://lists.ubuntu.com/archives/kernel-team/2008-January/002036.html): and is needed for NFS
  2. [Export put_filp](https://lists.ubuntu.com/archives/kernel-team/2008-January/002037.html): This patch exports put_filp and is needed for NFS.
  3. [Export do_splice_from and do_splice_to](https://lists.ubuntu.com/archives/kernel-team/2008-January/002038.html):
  4. [Export security_inode_permission()](https://lists.ubuntu.com/archives/kernel-team/2008-January/002039.html), also used by UnionFS, where it is currently disabled
  5. [Export deny_write_access()](https://lists.ubuntu.com/archives/kernel-team/2008-January/002040.html):

The aufs source package will see an update in Debian soon, which will be sync’ed into Ubuntu Hardy. This update uses the same code as the aufs code in linux-ubuntu-modules. This new code has improved XFS support, improved TMPFS support, improved support for lhash and put_filp patches. It’s also the first revision to support kernel 2.6.24.

In the next days, we may see first disks using aufs. I’ll provide patches for the software tomorrow. If nothing goes wrong, Ubuntu Hardy disks may use aufs instead of UnionFS, which may still be provided as a fallback.

vcs-performance.pdf

Some notes:

  1. The unit in the first chart is seconds
  2. I haven't run git-gc before taking the directory sizes. Running git-gc saves about 1083 kilobytes
  3. This PDF file is licensed under the MIT License, see [http://opensource.org/licenses/mit-license.php](http://opensource.org/licenses/mit-license.php)

A few months ago, I bought an HP Compaq 6720s (GR 644ET), installed Ubuntu 7.10 64-bit on it, and everything worked out of the box (except eth0, due to problems in Ubuntu’s kernel). I have absolutely no problems with my laptop, even the built-in card reader works out of the box.

BTW, HP has sponsored several Developer machines (including Gluck) and was one of the sponsors of the debconf7.

Maybe the following page at the Ubuntu wiki helps you with your laptop: https://wiki.ubuntu.com/hp_dv6000_series_(dv6116eu)

The next time you buy an HP laptop, buy one of their business laptops and make sure that a version with pre-installed FreeDOS is available.

I also became involved in the development of Debian GNU/Linux with my first package, aufs, sponsored by Daniel Baumann. This was the starting point for my Debian stuff.

Now I maintain 6 packages in Debian. One of the most interesting packages is gnome-app-install, which will be part of the gnome-desktop task (already added to tasksel’s list). Users will be able to install applications easily. It also activates the automatic codec installation, so users will be able to play most multimedia files easily.

30th December, I finally got assigned my AM, Martin Zobel-Helas (zobel), and I’m happy to be able to continue the NM process now.

On the Ubuntu side, some things have happened, too. I merged several new versions of dir2ogg and a new aufs version, and became an Ubuntu Member on the 29th November.

In 2008, I plan to complete the New Maintainer process and become an Ubuntu MOTU. I will continue my packaging stuff, and will merge new features from Debian to Ubuntu and from Ubuntu to Debian.

A happy new year to everyone and sorry for blogging this such late.

A Debian Maintainer can upload new versions of his packages, if they are not NEW and if the Packages contains a DM-Upload-Allowed field. This would allow me to upload new versions of gnome-app-install or app-install-data, dir2ogg, etc. without needing to ask a sponsor first.

I hope to be accepted as a Debian Maintainer soon.

General Resolution: http://www.debian.org/vote/2007/vote_003

Application: http://lists.debian.org/debian-newmaint/2007/12/msg00056.html

I used to blog on fsfe.org, the website of the Fellowship of the Free Software Foundation Europe. There was one problem that made me switch to wordpress.com: only Fellows can post comments.

I will write here about my activity in free software projects, like Debian and Ubuntu, and my own software.

I am still working on the categories and will maybe change the style.