Re: [imapsync] speed problem for large mails

[Stefan Schmidt <jsj at jsj dot dyndns dot org>]

Hello Gilles,

thank you for your answer.

On Friday 09 January 2009 01:42:45 Gilles LAMIRAL wrote:
> > Right now I am testing between two Cyrus IMAP servers
> > Source: openSUSE 11.0, cyrus 2.3.11
> > Destination: openSUSE 11.1, cyrus 2.3.11
>
> rsync is better with two cyrus imap servers.

I know, but nevertheless in my case it serves as a good (and cheap) 
testbed.

> Test in the real future situation since you'll encounter
> other problems with other environment.

Never denied ;)

> > where the speed drops dramatically.
> > One mail is 77MB big, and a transfer takes longer than 30 minutes
>
> I make regular test with big emails and I detect no problem.
> Is there the same issue from localhost to localhost ?

Yes.
I tested with a Dual PIII 1.5GHz, and a Core2Duo 2.33, where in the 
first test the destination Cyrus was in a VMware guest.

> > Then I had a closer look at the transfer speed. When the tranfer
> > started, iftop showed something around 400K/s. This very soon drops
> > below 100K/s and is as low as 15K/s around a transferred size of
> > 40GB.
>
> What are your kernel, perl release, architecture ?

2.6.18 with perl 5.8.8 for testing purposes with a different perl 
version
2.6.25.18 for the source cyrus 2.3.11
2.6.27.7 for destination cyrus with perl 5.10.0

> > I strace'd the imapsync process and found, that for almost every 4K
> > read a mremap is made, which gets more expensive with the increase
> > of the already allocated memory chunk.
>
> Since we're in system calls the perl script can't do many
> things to solve this problem.

I feared you might say that.

> > I do not know if there is something to be done with any of the
> > command line parameters, the buffersize parameter does not change
> > this behaviour.
>
> Try:
>   --split1 100

This will not help, as the problem is one large mail.

> > My command line is here:
> >
> > imapsync --host1 source --user1 jsj --password1  PWCHANGED
> > --authmech1 PLAIN --ssl1 --fastio1 --host2 destination --user2
> > jsjneu --password2  PWCHANGED --authmech2 PLAIN --ssl2 --fastio2
> > --buffersize 65536
> >
> > Nevertheless I am very interested in speeding things up, as the
> > final usecase has 320GB in about 2500 mailboxes, and I do now want
> > to wait ages to migrate.
>
> You can try --maxsize in a first run then a second run
> for big messages.

Yeah, but I do not know, when I can deliver the large mails to my 
users...

I keep trying.

Greetings,

Stefan
-- 
Stefan Schmidt		jsj at jsj dot dyndns dot org