Category: SysAdmin


Recently, I started seeing the following error in my mail logs on RHEL4 servers:

Sep 30 18:36:26 mx01 pop3(woody): file lib.c: line 37 (nearest_power): assertion
 failed: (num <= ((size_t)1 << (BITS_IN_SIZE_T-1)))
Sep 30 18:36:26 mx01 dovecot: child 11238 (pop3) killed with signal 6

This has been happening since dovecot-0.99.11-10.EL4, released on 18 August 2011. The changelog for this release shows:

- fix potential crash when parsing header names that contain NUL characters
 (#728674)

It could be that this update introduced a bug, or it could simply be a coincidence and completely unrelated.

The problem is a malformed multi-part (rfc1341) message. These messages usually look something like this. In the headers, you'll see:

Content-type: multipart/mixed; boundary="M7qxj4rHbpfU"

The body, oversimplified, will look something like this:

Some text.

--M7qxj4rHbpfU
Content-type: foo

foo foo foo
foo foo foo
foo foo foo

--M7qxj4rHbpfU
Content-type: text/plain; charset=us-ascii 

bar bar bar
bar bar bar
bar bar bar

--M7qxj4rHbpfU--

This last line, the closing encapsulation boundary, indicates that there are no further parts:

--M7qxj4rHbpfU--

In cases where this crash happens, this line is missing. I suspect what's happening is that dovecot continue reading the next mail's headers as if it's part of the last part, until it reaches the new (unexpected) Content-type header.

To fix this, you can insert the missing boundary - copy one of the previous ones and add -- at the end. Or delete the offending mail - mutt -f will open the mailbox.

The other day, I encounetered a problem where Plesk would fail to save settings from the Web Hosting Settings page, giving only this error:

Error: Unable to update hosting preferences: hosting update is failed: webstatmng is failed –unset-config –domain-name=Domain name –stat-prog=

Rather uninformative, but it turned out to be fairly simple. The box had been upgraded from Plesk 7.5.4 to 9.5.4, and it seems something went wrong in the psa database:

# mysql -u admin -p`cat /etc/psa/.psa.shadow` psa -e "SELECT d.name, h.webstat FROM hosting h, domains d WHERE dom_id=id AND name = 'example.com';"
+-------------+---------+
| name        | webstat |
+-------------+---------+
| example.com |         | 
+-------------+---------+
1 row in set (0.00 sec)

To remedy this, run the following MySQL query. Replace ‘none’ with ‘webalizer’ or ‘awstats’ where desired:

# mysql -u admin -p`cat /etc/psa/.psa.shadow` psa -e "UPDATE domains d, hosting h set h.webstat='none' WHERE h.dom_id=d.id AND d.name='example.com'"

Or if you have more than one domain that has this problem, run the follwing to change all the empty webstat fields to ‘none’:

# mysql -u admin -p`cat /etc/psa/.psa.shadow` psa -e "UPDATE hosting SET webstat='none' WHERE webstat = '' "

Now, you can go back to Plesk’s Web Hosting Settings page and save your settings.

Every now and then I encounter a server that have multiple external drives attached to it. When the server is rebooted – let’s say we just updated the kernel – it sometimes happens that the external drives don’t get assigned the same device numbers they had before. This, in turn, means that we can’t mount them the old way, i.e. by device number, via /etc/fstab.

Fortunately, there are alternative ways of doing this. Let’s start by finding out the filesytem ID.

For ext2/3/4, do:

# dumpe2fs /dev/sdb1 | grep UUID
dumpe2fs 1.39 (29-May-2006)
Filesystem UUID:          5fbb68f9-86af-4784-bdb3-3ab452a5f263

For reiserfs, do:

# debugreiserfs /dev/sdc1 | grep UUID
debugreiserfs 3.6.21 (2009 www.namesys.com)

UUID: aa3c3244-83c2-491e-bab5-067c5e73ce9c

For xfs, do:

# xfs_admin -u /dev/sdc1 
UUID = 0d4139ef-db18-417c-8dc0-273f94acfa3a

JFS, as far as I can tell, does not automatically generate a UUID when you create the filesystem. You have to do it.

# jfs_tune -U random /dev/sdc1 
jfs_tune version 1.1.12, 24-Aug-2007
UUID updated successfully.

Now, to view the UUID, do:

# jfs_tune -l /dev/sdc1 | grep UUID
File system UUID:       cab1b11d-1f09-4dc6-aec8-b84537ff895f
External log UUID:      b02579b7-0100-0000-0000-000001000000

Now, adjust your /etc/fstab:

#/dev/sdc1                                 /backup   ext3  defaults 0 0 
UUID=b7bcf64a-e172-42ce-8ce7-12365cc72c6c  /backup   ext3  defaults 0 0

This will ensure that each filesystem is mounted in the right place, even if the device names come up in the wrong order.

I had to solve a problem for a customer where they need to delegate management of backup MX servers to their clients. Of course, they don’t want to give their clients root access to their server, but they can write an app that gives users access to certain entries in a database, based on their privileges. This means I can join my two favourite server apps in the world: Postfix + MySQL

First, we need to create a database and some tables. These are quick and dirty and only meant to be a proof of concept. The fields a pretty straight forward: id is just a number, domain is the domain being relayed, and destination is the primary MX or where ever you need the mail to go. The syntax is as per the Postfix transport documentation. :domain.tld will do an mx lookup on domain.tld, and smtp:host.domain.tld will deliver directly to the host specified. My database is called ‘backupmx’

CREATE TABLE `domains` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `domain` varchar(128) NOT NULL DEFAULT '',
  `destination` varchar(128) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  UNIQUE KEY `domain` (`domain`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

CREATE TABLE `recipients` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `address` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  KEY `address` (`address`)
) ENGINE=MyISAM AUTO_INCREMENT=9 DEFAULT CHARSET=utf8;

View full article »

I encountered this today when trying to query a package on a Red Hat server:

rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db-&gt;close: Invalid argument
error: cannot open Name index using db3 – Cannot allocate memory  (12)

This happens when an operation involving the rpm db gets interrupted and the locks not cleared.  The solution is rather simple:

1.   Make a backup of /var/lib/rpm   (because you never know)

2.   rm /var/lib/rpm/__db.00*

3.   rpm --rebuilddb

You should have a working rpm databse now.

I came across an error after upgrading a Plesk server from 8.6 to 9.2.3:

# /usr/local/psa/admin/sbin/mchk --with-spam
==> Checking for: mailsrv_conf_init... ok
==> Checking for: mail_mailbox_restore... ok
==> Checking for: mailsrv_entities_dump... ok
==> Checking for: mail_admin_aliases... ok
==> Checking for: mail_auth_dump... ok
==> Checking for: mailman_lists_dump... ok
==> Checking for: mail_responder_restore... ok
==> Checking for: mail_drweb_restore... ok
==> Checking for: mail_kav_restore... not exsists
==> Checking for: mail_spf_restore... ok
==> Checking for: mail_dk_restore... ok
==> Checking for: mail_grey_restore... ok
awk: cmd. line:50: (END OF FILE)
awk: cmd. line:50: invalid char '�' in expression
unable to process "pop3d"
awk: cmd. line:50: (END OF FILE)
awk: cmd. line:50: invalid char '�' in expression
unable to process "pop3d-ssl"
awk: cmd. line:50: (END OF FILE)
awk: cmd. line:50: invalid char '�' in expression
unable to process "imapd"
awk: cmd. line:50: (END OF FILE)
awk: cmd. line:50: invalid char '�' in expression
unable to process "imapd-ssl" 

View full article »

This is something that comes up all the time when clients are having PCI compliance scans done on their servers – qmail, by default, allows SSLv2 to be used.

To disable it is very easy.  Edit or create /var/qmail/control/tlsserverciphers and add the following line to it:

ALL:!ADH:!LOW:!SSLv2:!EXP:+HIGH:+MEDIUM

Now restart qmail, and you’re done!

Today I got tasked with removing duplicate mails from a mail folder with over 100,000 mails in it.  Doing this from a mail client is so impractical, it’s not even worth giving any thought at all.  Fortunately, the mailbox is on a mail server using Maildir style mailboxes, so I knew this could be done with minimum effort.

I discovered the ‘reformail’ utility, part of courier-imap, and after a few trial runs, I settled on the following:

# cd /path/to/mailbox/Maildir/cur

# for i in `find . -type f`; do reformail -D 10000000 /tmp/duplicates <$i && rm $i; done


-D looks for, and deletes duplicates.

10000000 is the length of the temporary file where a list of message IDs will be written

/tmp/duplicates is the aforementioned temporary file.

The temporary file needs to be big enough to accommodate the message ID of each mail.  In this particular case, I have found the average length to be 54 characters, but I would suggest using around double that to be safe.  So adjust to your needs.

In a big mail folder, and especially on ext3, this will take a long time to complete.

I recently decided to move my enormous mail archive from my trusty Courier-imap/Maildir setup to DBmail. The reason is simple.  I have several mail folders with 100,000+ mails in.  That means several directories with upwards of 100,000 files in.   And that means bad performance.    There is another reason:  I can execute far more powerful searches with an SQL query, than any mail client can allow me to do.

DBmail is a POP3/IMAP server that uses a regular database server (currently MySQL, PostgreSQL or SQLite) for its mail store.  Given the obvious advantages, I’m surprised this isn’t more popular.

There is one gotcha to the setup.  MySQL, being Swedish, has a default collation setting of ‘latin1_swedish_ci’ while DBmail assumes ‘utf8_general_ci’ will be set.  But the DBmail docs, and even the MySQL notes page, does not mention this* at all, and the included create_tables.mysql script does not set the correct collation either.   This results in the following error showing up in the logs:

Sep 14 03:00:01 hermes dbmail/maintenance[16708]: Error: dbmysql.c,db_mysql_check_collations(+138): collation mismatch, your MySQL configuration specifies a different charset than the data currently in your DBMail database.

This is easily fixed.  Assuming your database is called ‘dbmail’ do:

mysql> use information_schema;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select * from SCHEMATA where SCHEMA_NAME = ‘dbmail’;

The ‘DEFAULT_COLLATION_NAME’ column will most likely show ‘latin1_swedish_ci’ – this is the problem.  Run the following:
mysql> alter database `dbmail` collate `utf8_general_ci`;
Query OK, 1 row affected (0.02 sec)

Run the select query again and check if the right collation is showing.  After doing this, dbmail should connect to MySQL without any problems.

From time to time,  Squirrelmail gives this error on a Plesk machine:

Error opening /var/lib/squirrelmail/prefs/default_pref
Could not create initial preference file!
/var/lib/squirrelmail/prefs/ should be writable by user apache
Please contact your system administrator and report this error.

Squirrelmail depends on safe_mode being off.  Let’s see if this is the case:

# grep ^safe_mode /etc/php.ini
safe_mode = On

Since this is a multi-domain system, we want to make changes only to the effected subdomain, in this case the webmail.* subdomain.  But changing the Plesk config won’t help, since Plesk will just overwrite it.   So we create a second file, that will load after, and thus override the Plesk file:

# vi /etc/httpd/conf.d/zz011_squirrelmail_safemode_fix.conf

And add the following:

<Directory "/usr/share/squirrelmail">
     php_admin_flag safe_mode off
</Directory>

Now check your apache config and restart gracefully:

# httpd -t
OK
# apachectl graceful

You may also need to set the following in /usr/share/squirrelmail/config/config_local.php

$default_folder_prefix = 'INBOX.';

Reload Squirrelmail in your browser – it should work now