speed problem for large mails

[Stefan Schmidt <jsj at jsj dot dyndns dot org>]

Hi everyone,

I am evaluating imapsync for a larger migration task from Cyrus IMAP to 
MS Exchange 2007.

Right now I am testing between two Cyrus IMAP servers
Source: openSUSE 11.0, cyrus 2.3.11
Destination: openSUSE 11.1, cyrus 2.3.11
with a test mailbox containing about 400.000 mails with a total size of 
about 2.5GiB.

So far it works good, but there are larger mails the mailbox, where the 
speed drops dramatically.
One mail is 77MB big, and a transfer takes longer than 30 minutes - 
where I first hit a timeout in my destination cyrus ;). OK, timeout 
increased.
Then I had a closer look at the transfer speed. When the tranfer 
started, iftop showed something around 400K/s. This very soon drops 
below 100K/s and is as low as 15K/s around a transferred size of 40GB. 
I strace'd the imapsync process and found, that for almost every 4K 
read a mremap is made, which gets more expensive with the increase of 
the already allocated memory chunk.

I do not know if there is something to be done with any of the command 
line parameters, the buffersize parameter does not change this 
behaviour.
My command line is here:

imapsync --host1 source --user1 jsj --password1  PWCHANGED --authmech1 
PLAIN --ssl1 --fastio1 --host2 destination --user2 jsjneu --password2 
 PWCHANGED --authmech2 PLAIN --ssl2 --fastio2 --buffersize 65536

Nevertheless I am very interested in speeding things up, as the final 
usecase has 320GB in about 2500 mailboxes, and I do now want to wait 
ages to migrate.

What can I do to help?

Greetings,

Stefan
-- 
Stefan Schmidt		jsj at jsj dot dyndns dot org