Re: [imapsync] speed problem for large mails

[Gilles LAMIRAL <gilles dot lamiral at laposte dot net>]

Hello Stefan,

> I am evaluating imapsync for a larger migration task from Cyrus IMAP to
> MS Exchange 2007.

Ok.

> Right now I am testing between two Cyrus IMAP servers
> Source: openSUSE 11.0, cyrus 2.3.11
> Destination: openSUSE 11.1, cyrus 2.3.11

rsync is better with two cyrus imap servers.
Test in the real future situation since you'll encounter
other problems with other environment.

> where the speed drops dramatically.
> One mail is 77MB big, and a transfer takes longer than 30 minutes 

I make regular test with big emails and I detect no problem.
Is there the same issue from localhost to localhost ?

> Then I had a closer look at the transfer speed. When the tranfer
> started, iftop showed something around 400K/s. This very soon drops
> below 100K/s and is as low as 15K/s around a transferred size of 40GB.

What are your kernel, perl release, architecture ? 

> I strace'd the imapsync process and found, that for almost every 4K
> read a mremap is made, which gets more expensive with the increase of
> the already allocated memory chunk.

Since we're in system calls the perl script can't do many
things to solve this problem.

> I do not know if there is something to be done with any of the command
> line parameters, the buffersize parameter does not change this
> behaviour.

Try:
  --split1 100

> My command line is here:
> 
> imapsync --host1 source --user1 jsj --password1  PWCHANGED --authmech1
> PLAIN --ssl1 --fastio1 --host2 destination --user2 jsjneu --password2
 PWCHANGED *censored* --authmech2 PLAIN --ssl2 --fastio2 --buffersize 65536
> 
> Nevertheless I am very interested in speeding things up, as the final
> usecase has 320GB in about 2500 mailboxes, and I do now want to wait
> ages to migrate.

You can try --maxsize in a first run then a second run
for big messages.

Sorry, no good solution.

-- 
Au revoir,                               02 99 64 31 77
Gilles Lamiral. France, Chavagne (35310) 06 20 79 76 06