Re: [imapsync] Exchange 2k7 success story

[Gilles LAMIRAL <gilles dot lamiral at laposte dot net>]

Hello Stefan,

> Our Exchange admin tells me, we had Exchange 2007 SP1, most probably 
> with Update Rollup 2 installed at that time - about a year ago, between 
> February and March 2009.

Ok.

> I already described our move in my mail from Aug 5th, 2009,
> <200908050952 dot 05139 dot jsj at jsj dot dyndns dot org>.

Ah yes, it is \Flagged in my mailbox.
The strange thing is that the archive prints the whole reply
given by Ralp but truncate yours:
http://www.linux-france.org/prj/imapsync_list/msg00307.html
http://www.linux-france.org/prj/imapsync_list/msg00309.html

Oh I see, the next paragraph begins with \n\nFrom at the beginning
and the archive software takes it for a new message.
I'll take a look at this ugly mbox format bug.

> Again, here is the text:

Thanks, we have it now:
http://www.linux-france.org/prj/imapsync_list/msg00453.html

> ====
> FWIW, I migrated from cyrus to Exchange using the following command 
> line:
> /root/bin/imapsync.1_278 \
>  --host1 $CYRUSSERVER --user1 $CYRUSUSER \
>  --authuser1 cyrusadmin --password1  PWCHANGED \
>  --host2 $EXCHANGESERVER --user2 $EXCHANGEUSER \
>  --authuser2 exchangemigration --password2  PWCHANGED \
>  --useheader 'Message-Id' --useheader 'Message-ID' \
>  --useheader 'Received' \
>  --nofoldersizes \
>  --skipsize \
>  --regexflag 's/(\A[^\\]\w+\s)|(\s[^\\]\w+)//g' \
>  --regexflag 's/(\$No.Junk)//g' \
>  --regexflag 's/(No.Junk)//g' \
>  --regexflag 's/(Seen-handled)/Seen/g' \
>  --regexflag 's/(Junk)//g' \
>  --regexflag 's/(\$)//g' \
>  --regexmess 's/^Message-Id/Message-ID/i' \
>  --regextrans2 's/.Drafts{,1}$/Drafts/' \
>  --regextrans2 's/^INBOX$//' \
>  --regextrans2 's/(.{1,})/\/$1/' \
>  --regextrans2 's/(.*)/INBOX$1/' \
>  --regextrans2 's/\.Draft[s]{0,1}$/Drafts/' \
>  --regextrans2 's/.*\/Drafts/Drafts/' \
>  --regextrans2 's/.*\/.Sent/Sent\ Items/' \
>  --exclude '.Trash$' \
>  --fastio1 --fastio2
>
> So I had a big deal with removing flags Exchange does want to know about

Now imapsync >= 292 only transfers flags supported by --host2
A nice default feature I think, like the RFC recommends it.

> and moving folders to places where an Exchange user expects her/his 
> mails.

This is server specific.
Maybe I couls add --exchange option but we have to deal with
i18n also. I prefer a good FAQ item and sysadmins knowing what
they need and what they do.

> I looped it for 2 weeks in 4 threads over all users with a final run 
> after the deliver switch.
> I do not recall any issues with deleted mails, except omitting the 
> .Trash folder from squirrelmail or TB.

Ok.

> There were warnings and sometimes not transferred mail boxes due to 
> other flags I did not cover here, but that affected only about 10 out of 
> approx 2000 mailboxes.

Ok. Fixed now.

> As written in an other thread here, if there were messages not 
> transferred due to "Keywords are not supported", these messages were 
> successfully transferred during the next loop.

Strange!

> Exchange seem to allow the 6 RFC3501 sec 2.3.2. mentioned flags \Seen, 
> \Answered, \Flagged, \Deleted, \Draft and \Recent, only. So in migrating 
> to Exchange one has to make sure, that only these flags are there and 
> delete all others.

May a negative lookahead pattern could work:

--regexflag 's/(?!(\Seen|\Answered|\Flagged|\Deleted|\Draft))//g'


> Also, as Exchange rewrites the mails upon receiving, for comparison with 
> already existing mails, restrict to use only to the "Received:" and 
> "Message-ID" header.

Ok.

> The "Keywords are not supported" issues were with mails containing 
> square brackets in the "Subject:" header. These were nevertheless 
> transferred successfully.

Strange since square brackets are very common.

> A really important - but not Exchange related - point for me was, to not 
> use ssl if possible, or when you have huge mails, say bigger than 10MB. 
> It slows down the transfer greatly by the horrible memory management in 
> the ssl libraries - allocating space, filling it up, allocating a memory 
> chunk 4096 bytes bigger, memcopying the first buffer into the second, 
> reading 4096 byte, allocation 4096 bytes bigger buffer, memcopying, and 
> so on. This is OK for up until 10MB, but afterwards it takes ages for a 
> single mail - I remember a 75MB mail took more than 5000s on a Xeon 
> X7350 core with 4GB RAM!

Ok. It's architecture dependent. I remember the discussion now.

> HTH, imapsync made the migration (for which we spend top Euros for hired 
> external partners - which failed royally) a - lets say - walk. And: 
> imapsync was only the personal "Plan B". Great piece of software!

Thanks!

--
Au revoir,                               02 99 64 31 77
Gilles Lamiral. France, Chavagne (35310) 06 20 79 76 06