[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [imapsync] From Exchange to Dovecot. Speed (was How to sync the public folders?)


From Gilles LAMIRAL <gilles dot lamiral at laposte dot net>
Subject Re: [imapsync] From Exchange to Dovecot. Speed (was How to sync the public folders?)
Date Mon, 03 Nov 2014 00:14:59 +0100

Hi,

In this case they want it to be that way: all coworkers should have
access to these folders without manual intervention, as soon as someone
creates a folder in there everyone should see it.

Ok. So it looks like "group folder" instead of "public folders", yes it can be good in many scenari, but let's forget it, everyone finds its own usage.

I have that already ... but it takes days to complete this sync (fast
ethernet connection only, sloooow old exchange hardware) ... so I will
only be able to fully test that tomorrow.

does "–-nofoldersizes --skipsize –-fast" still speed up things?

–-nofoldersizes skips the folder size stats and removes the ETA. --skipsize is on by default. --fast does nothing (just remove a sleep 2)

To avoid stressing servers by getting headers --useuid is good.

imapsync ... --useuid --tmpdir /var/tmp/


You have to find the real bottleneck. It can be hardware on Exchange but it can be something else. For example, there is a throttling mechanism on Exchange 2013 http://www.linux-france.org/prj/imapsync_list/msg01785.html

Throttling is there when transfer starts fast and then goes down.

Then try --maxmessagespersecond 4

imapsync ... --maxmessagespersecond 4

I quote the § on throttling from David Karnowski:

We actually got a helpful response from Microsoft today explaining the throttling
that they enforce on IMAP connections:

    ImapMaxConcurrency                        : 20
    ImapMaxBurst                              : 240000
    ImapRechargeRate                          : 360000
    ImapCutoffBalance                         : 600000


The figures (outside of the concurrency value) are measured in milliseconds and represent the amount of time over a 1 hour period. The max burst value represents the amount of time that can be spent before any throttling is done (microdelays), the recharge rate shows how much budget will be added over a 1 hour period (constant process), and the cutoff balance represents the amount of budget that can be used after which a backoff request (exception) is generated.

    As far as the throttling bit, our Datacenter team confirmed from the logs that
the account is being throttled. They indicated that the disconnects you were
seeing were a result of authentication failures (because the account was
throttled).

    "We recently added an optimization to not cause auth to fail if the account is
throttled but all subsequent commands will fail with invalid state error due to
throttling."

    Let me check to see what we can do in terms of raising the throttling and if we
can exempt the accounts you are using from the throttling policies while the
migration is performed. I'm not sure we will be able to exempt them fully, but I
do believe we can increase the limits.

    The throttling here is occurring because you are using the 3rd party IMAPSync
tool that in essence is acting as an IMAP "client" to import the data. There
isn't specifically a problem with that, but as I surmised early on, this does
result in the User throttling coming into play.

    Also wanted to provide a further explanation to your below question. The way
throttling has been implemented in Exchange 2013 is that each user receives a
MaxBurst value - this is the "budget", or amount of work they can do in a 1 hour
period without being throttled. At the same time, as the budget is being used,
we are "recharging" the budget (Recharge Rate). If you use up your MaxBurst
value, we will begin inserting MicroDelays. Basically this means that we will
begin slowing down our responses to commands. You won't know this is happening,
as it is designed to not be intrusive to the end-user. It is only when you reach
the CutoffBalance that we issue a backoff exception. At this point, you would
see the error responses due to invalid state.


So, we have experienced Microsoft's throttles as: disconnects, failures to authenticate, and also the "BAD Command received in Invalid state." error which is in the log excerpt I provided as well as in Microsoft's explanation of the throttling.


-- Au revoir, 09 51 84 42 42 Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06