There is also another good news, apart from the equinox. I have resurrected cflow project. It was initially written in 1997-1999 and fell into oblivion shortly afterwards. Somehow, I got distracted by other things... Now, a recent thread in gnu-devel mailing list has shown that there is a demand for this kind of program and this made me remember my past attemtps. So I have (quite happily) resurrected the package. During this week I have managed to turn it into GNU-compatible project. I have also sent a registration request to RMS.
The question is: where was my brain during these seven years? Another question is: how many useful programs still lie buried in my archives?
The program is compatible with POSIX specs and offers quite a number of improvements (of course it will offer more still:). Here is a sample output tree describing the project itself. The tree was obtained using chart.cflow rule in src/Makefile.am.
How to handle a shrink [of a mailbox]? meaning, the &^$^@%#@^& user start two browsers and deleted emails in one session. My views is that we should scream bloody murder and hunt them with a machette. But for now just play dumb, but maybe the best approach is to pack our things and leave i.e exit()/abort().
Imagine the following synopsis: your imap4d daemon is working with a mailbox. Part of its data is cached into memory, any modifications to the mailbox are not flushed to disk until user explicitely requires it.
Now, the same user launches another mail client that opens the same mailbox, deletes some messages, and closes it.
The user makes some modifications to the mailbox in imap4d, requests EXPUNGE and closes the mailbox. Imap4d performs flush using the data cached in memory and, voilà - the mailbox is trashed.
You'd say the obvious solution is to lock the mailbox while it is being operating upon. But no, that won't work for imap4d, because appending to the mailbox is quite normal, and it should be allowed. Locking the mailbox will cause mail delivery to be blocked during the client's session (and as a sysadmin, I may say that the number of users that keep their imap session open for 24 hours a day is surprisingly big!)
Of course, the mailbox must be locked while it is being written to, and we do this, but that's another story.
In GNU mailutils we use a function called mailbox_is_updated to determine whether the mailbox was modified recently. The function returns 0 if it was modified and 1 otherwise. For plain UNIX mailboxes a simple approach is used: the function checks if the size of the mailbox file has changed.
It is assumed that if it the new size is greater than previously recorded one, than some new message were appended to the mailbox. If the size is less than the recorded one, then, well... see the quote at the start of this article :)
The mailbox can be modified without changing its size (well, yes, it's rather rare, but nevertheless possible);
And finally, the case of mailbox size shrinkage was still not handled in imap4d. Still, because the issue has quite long history: it dates back to April of 2001, when I first joined the Mailutils team.
I noticed that pop3d broke on simultaneous accesses to the mailbox. The story with pop3d is quite different, since it does use mailbox locking (well, that's to say now. Back then it still did not...), anyways:
Nice comment on a correctness of the name mailbox_is_updated. It raised a small discussion (see my next posting and Dawe's answer to it), but did not lead to renaming the function :)
As a side note, the story of linguistic inventions in bug-mailutils list is quite amusing... I certainly should catalogue them :^)
An unrelated issue, but still worth mentioning: on some machines disabling mmap speeds up pop3d. Rationale: mailbox_is_updated indirectly calls msync which makes the daemon spend quite some time in uninterruptible sleep.
The approach used by imap4d is a bit different from what we use in pop3d. Essentially, upon selecting a mailbox imap4d does the following:
Before processing the new command, call imap4d_sync. This function calls mailbox_is_updated and issues RFC2060 untagged responses if the mailbox was modified. Imap4d always did this. The essential change is the following.
If the mailbox was modified and mailbox_corrupt is set, close the mailbox and open in again. This discards all cached data and reads the new mailbox contents.
The drawback of this approach is that all modifications to the mailbox the user might have made during the imap4 session are discarded. In my opinion it is a reasonable price for preserving the mailbox contents. However, this issue can be addressed too. More on this later.
While solving mailbox shrink problem, the method does not solve the described above weaknesses of mailbox_is_updated.
To actually solve the issue mailbox_is_updated must be able to tell that the mailbox was modified, and how exactly it was modified, independently of its size. Unfortunately this is easier said than done. Currently mailutils is able to handle various mailbox formats and the way to do the proper diagnosis will vary with the format.
Let's first see what can be done for plain UNIX mailboxes. Obviously, mailbox_is_updated must first check mtime of the mailbox file. If it did not change, the function can return 1 immediately. Otherwise, it should assume some changes were done to the mailbox. Now, the question is whether these changes affect the cached data. The code in mailbox/mbox/mbox.c keeps offsets of UNIX 'From ' lines in the mailbox file. It will suffice to compare these with real offsets to decide whether the mailbox was modified (in the terms of libmailutils corrupted) or a new message was added to it. If any of the registered offsets differ, we scream bloody murder and raise MU_EVT_MAILBOX_CORRUPT event1) . In any case, we return 0 indicating that the mailbox has been changed.
As a side note, we could also prevent undesirable modifications of the plain UNIX mailbox if we apply kernel exclusive lock on the file from offset 0 up to its current size...
I have implemented new MDA mode for Anubis. In this mode, it acts as a mail delivery agent, receiving incoming mail from the standard input, processing it and piping modified message to the real local mailer.
This allows me to first process incoming mail through Anubis and then through Sieve (invoked by mail.local)!
First, I have finally joined Ukrainian translation team. Fortunately, it did not require any paperwork as our usual disclamer by default covers any other language teams one may want to join. Would it be so with our GNU disclamer as well! I hate waiting for a snail-mail to arrive from Boston to me and for my reply to reach Boston again only in order for my 25-line patch to Emacs being finally applied to CVS (by the way it is a real case, Ted have sent it almost a month ago and I'm still waiting for it to arrive, and the patch to rmail is still waiting in the queue, although everybody knows how many assignments I have already made to GNU project... I wonder why would not RMS create a new assignment form for this and any other project whatsoever I may wish to join or create within GNU or (whatever else) in the future).
Secondly, Mailutils finally has vacation extension action in Sieve. To be precise, we already had it as an extension to Scheme implementation of Sieve. I lacked time (or was too lazy?) to do the same for our Sieve library until Felix Egli sent us his version which didn't quite match our standards and was incompatible with the existed one, so I decided to finally write it:) Massive thanks, Felix!
Another news in Mailutils is ODBC SQL driver, which actually means that from now on Mailutils is able to interact with virtually any existing DBMS (since any serious DBMS has an ODBC interface).
As usual most of the time is taking tar. After 1.15.1 release I have started a major rewrite of buffer handling and incremental archive parts.
Several fixes were applied to Anubis as well. Recent letters from Paolo made me finally gather my wits and set forth a draft describing the proposed method of handling multi-part messages.
This, in its turn, forced me to finally attack and solve the problem of using gendocs.sh script to generate web documentation in our projects. The resulting Makefile.am is very elegant. However, this patch must be applied to the script in order to use it. I'll submit the patch upstream tomorrow.
The only thing left with this approach is the absence of per-chapter html output, which I am going to fix shortly (it will require sending a new assignment form to Boston. At the end, I think, I'd better send quite a bunch of them beforehand).
A courageous decision of the Polish Government prevented European Union from adopting a disastrous "Software Patent Directive"
I have been chasing for bugs in gsasl 0.2.2 for over four hours. Managed to find quite a few. Debugging is probably the most exhaustive kind of work I've ever tried...
Emacs folks have begun arguing about GFDL vs. GPL problem (though, frankly speaking I can't see any problem at all). Debian people evidently become tired of it and their discussion on this topic has faded away. Now it's emacs-devel turn, isn't it? Interestingly, how similar both threads are: the two sides go on spitting different reasons and no one seems to have heard the other, whereas RMS keeps olympic silence and only when really annoyed does he tell: "I believe GFDL does what it should". So spricht Zaratustra...
I baptized the machine pepelatz (a term from Georgi Daneliya's film Kin-Dza-Dza). Wander what's this? Here's the definition:
After a more-than-busy two-week pause I'm back again (I'd better tell some other time what I've been doing during these two weeks). Most important news during this time are: together with Max we have set up our server (Ulysses) and I've finally managed to mirror both ftp.gnu.org and www.gnu.org. Moreover, both mirrors are now accessible as ftp.gnu.org.ua and www.gnu.org.ua.
Another mirror I've been runnng, that of the Unix Heritage Society at tuhs.org.ua, is now hosted on this server too.
The question now is, how much will it cost to maintain all this, given its average daily traffic... We'll see...
Other FS news: a simple script to view po files. Why? I don't know. May prove useful some day... Click here to see it work.
Hacked on GNU mailutils and introduced a universal sql-access layer. The auth library has been rewritten accordingly and mail.local is finally able to retrieve mailbox quotas from SQL database, a feature many of the mailutils users have constantly pinged me about.
While feedling with mailutils I've noticed something that makes me feel a bit uncomfortable: we have no convention on how to write diagnostic messages. Some of them begin with a capital letter, some don't. Some end with a period, some don't. A chaos. We should certainly change it, but I'd better do it after releasing 0.5.1 maintenance release (this change will affect po files, and hence will require waiting for the translators to finish their jobs).
Anubis: a couple of security-related changes proposed by Ulf Härnhammar. Socks.c still waiting to be rewised by Wojtek.
Oh, yes, forgotten to say: among various junk on my machine I've excavated my geek code. Its surprising how little has changed since I wrote it...
Yesterday I released version 1.3 of GNU Radius. I wish Jordi updated Catalan localization. It was very important to me. Pity...
More work on rmail.el. People from emacs-devel have supplied some fresh ideas and RMS have asked me to update documentation as well:
Documentation is the castor oil of programming. Managers know it must be good because the programmers hate it so much.
Well, my log got interrupted for more than two weeks... I have started to learn Arabic, and it is taking now the rest of my time, which was in short supply already. I am growing more and more fascinated by both Arabic script and its grammar...
GNU Radius 1.3 is ready for the release. The current snapshot is marked 1.2.96 and is as usual available from mirddin.farlep.net. I am waiting for Jordi to finish Catalan translation, that's the only thing missing.
Several bugfixes and improvements have accumulated in GNU mailutils which makes me plan to release a maintenance version next week. See NEWS file for a short description of these.
I have submitted a hacked version of rmail.el to emacs-devel. This version makes extensive use of the new features provided by GNU mailutils movemail program. Both modified source and diff file against the latest CVS version 1.393 is available.
The new radius documentation is available. The main page is now xhtml 1.0 (as well as radius homepage)1) . I wish I would be able to convert documentation files to xhtml as well, but that would require either updating texi2html (I wouldn't like to do this) or texinfo (I'd love to, should the time permit)...
Anyway, the package itself is ready for the release. Today I have fixed a small bug that manifested itself on BSD systems, and have given a final polish to the docs. I'm going to release the stuff before Nov 06.