Re: FirstClass to Google

["Lavelle, Adam" <alavelle at uei-schools dot org>]

Some Interesting followup ----

In terms of it missing messages because of FirstClass reporting erroneous RFC822.DATE - running the same command with --nofoldersizes makes a huge difference is the messages that it transfers...  waiting for the first test run to complete, but it's already migrated several thousand messages more then before.

So in terms of the date options - I want to make sure I've chosen the best option....

I'm using --idatefromheader  (rather then the default of --syncinternaldates) since testing suggested that it was the better option, but now on the messages that it's transferring that it was missing without the --nofoldersizes 100 percent of the message arrived dates are the current time.

The following excerpt shows how it's doing the date, which certainly looks correct to me.
This shows up correctly when you open the message in google webmail, but the folder list (message arrived date) show the transferred date today.
Haven't paid attention to what it does with the --syncinternaldates

================================================================================
Sending: 5190 UID FETCH 5649491 BODY.PEEK[HEADER.FIELDS (Date)]
Sent 56 bytes
LITERAL: received literal in line * 3959 FETCH (BODY[HEADER.FIELDS ("Date")] of length 41; attempting to retrieve from the 41 bytes in: Date: Wed, 27 May 2009 21:10:33 -0500
  
  <END_OF_iBuffer>
Read:     * 3959 FETCH (BODY[HEADER.FIELDS ("Date")]    Date: Wed, 27 May 2009 21:10:33 -0500
  
       UID 5649491)
Read:     5190 OK FETCH Completed
found data between fetch headers: 5190 OK FETCH Completed
header date from 1: [Wed, 27 May 2009 21:10:33 -0500]
header date from 1: [Wed, 27 May 2009 21:10:33 -0500] (fixed)
flags & date from: [\Seen][Wed, 27 May 2009 21:10:33 -0500]
Sending literal: 2016 APPEND Transferred-INBOX (\Seen) "Wed, 27 May 2009 21:10:33 -0500" {204976}
      then: Message-id: <fc dot 011edd9000563436011edd9000563436 dot 563453 at MASKED>
  Date: Wed, 27 May 2009 21:10:33 -0500
  Subject: MASKED
  X-Mailer: FirstClass 9.1 (build 9.221)
  X-FC-Form-ID: 141
  X-FC-SERVER-TZ: 10486504
  To: 
  Cc: , 

Thanks,
Adam

On Wed, Aug 11, 2010 at 11:47 AM, Lavelle, Adam <> wrote:

Hello,

Been having a lot of issues attempting to get our users (300) migrated from OpenText Firstclass v9.221 to Google.
Perusing the list I see others have tried, but ran into issues somewhat similar to what I am seeing.

Having used imapsync to do many server migrations over the years, I really appreciate it's capabilities.  Thank you for such great work.
What I'm seeing here is my attempt to deal with the incredibly poor IMAP implementation of the OpenText FirstClass server.

The problem I'm running into is that only about 1/5 of the users inbox gets transferred.
The rest are rejected based on 0 size, or header issues.
I've tried a variety of imapsync versions and perl mail libraries with generally the same results.

Command being used:

$RCSfile: imapsync,v $ $Revision: 1.340 $ $Date: 2010/08/09 00:03:21 $
Here is a [darwin] system (Darwin fc3.uei-schools.org 10.2.0 Darwin Kernel Version 10.2.0: Tue Nov  3 10:35:19 PST 2009; root:xnu-1486.2.11~1/RELEASE_X86_64 x86_64)
With perl 5.10.0 Mail::IMAPClient  3.25
Command line used:
/users/admin/imapsync-1.340/imapsync --skipsize --nosyncacls --allowsizemismatch --idatefromheader --host1 MASKED --user1 tlegaux --password1 MASKED --host2 imap.gmail.com --user2 tlegaux at uei-schools dot org --password2 MASKED --authmech2 LOGIN --ssl2 --folder INBOX --debug --prefix2 Transferred- --debugimap --fast --split1 20 --useheader Message-ID
Temp directory is /var/folders/1U/1U9GuXGlHlOgHQmb5fqTC++++TI/-Tmp-
PID file is /var/folders/1U/1U9GuXGlHlOgHQmb5fqTC++++TI/-Tmp-/imapsync.pid
/var/folders/1U/1U9GuXGlHlOgHQmb5fqTC++++TI/-Tmp-/imapsync.pid already exists, overwriting it
Modules version list:
Mail::IMAPClient     3.25
IO::Socket           1.30_01
IO::Socket::SSL      1.33
Digest::MD5          2.40
Digest::HMAC_MD5     1.01
Term::ReadKey        2.30
Date::Manip          6.11

1st issue seems to revolve around the RFC822.DATE picking up 0 when I'm seeing a different result.  For example debug output shows:

Imapsync:
---
Read:     * 820 FETCH (RFC822.SIZE 0 UID 4117682)

Yet if I do a telnet session to run IMAP, the same command shows:

Telnet
---
? UID FETCH 4117682 full
* 2668 FETCH (FLAGS (\Seen) INTERNALDATE "03-Mar-2009 07:58:07 -0600" RFC822.SIZE 2048 ENVELOPE ("Tue, 03 Mar 2009 13:58:07 GMT" "Accomodations materials"  MASKED
? OK FETCH Completed

? UID FETCH 4117682 body[text]

* 2300 FETCH (BODY[TEXT] {1944}

This is a multi-part message in MIME format.

----=_--004cbaa2.004cba72.c5b8bdde
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit


Imapsync Summary
---
++++ Statistics ++++
Transfer time                    : 1542 sec
Messages transferred             : 0
Messages skipped                 : 11226
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 879
Messages void (noheader) on host1 : 8216
Messages void (noheader) on host2 : 0
Messages deleted on host1         : 0
Messages deleted on host2         : 0
Total bytes transferred           : 0
Total bytes duplicate host1       : 0
Total bytes duplicate host2      : 30724115
Total bytes skipped              : 99313776
Total bytes error                : 72
Message rate                      : 0.0 messages/s
Average bandwidth rate            : 0.0 KiB/s
Reconnections to host1            : 0
Reconnections to host2            : 0
Detected 1 errors


Question being - is there a way to totally ignore the RFC822.SIZE output from host1, and attempt to transfer every message?  I'm using the --skipsize option but that still seems to request the size of all messages.


2.  Less of an issue at the moment, but imapsync appears to run into problems if either host1 or host2 runs into a timeout or unexpected connection loss.  Logs show it attempts to reconnect, but really doesnt.  I assume that the reponse from host1 isn't an accurate response to a logout request?

Read:     * BYE FirstClass IMAP4 server logging out
ERROR: * BYE FirstClass IMAP4 server logging out at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1473
    Mail::IMAPClient::_get_response('Mail::IMAPClient=HASH(0x100d86648)', 301, undef) called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1406
    Mail::IMAPClient::_imap_command_do('Mail::IMAPClient=HASH(0x100d86648)', 'LOGOUT') called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1312
    Mail::IMAPClient::_imap_command('Mail::IMAPClient=HASH(0x100d86648)', 'LOGOUT') called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1925
    Mail::IMAPClient::logout('Mail::IMAPClient=HASH(0x100d86648)') called at ./imapsync line 2083
reconnecting to MASKED, last error: * BYE FirstClass IMAP4 server logging out
Connecting via IO::Socket::INET to MASKED:143 Timeout 600
Connected to MASKED errno(Socket is already connected)
Read:     * OK FirstClass IMAP4rev1 server v9.221 at mail.uei-schools.org ready
Sending: 302 AUTHENTICATE CRAM-MD5
Sent 27 bytes
Read:     + PDIwMTAwODEwMTc0MTQzLjUwMzE5ODUyQHVlaS1zY2hvb2xzLm9yZz4=
Sending: dGxlZ2F1eCBiZGQwNmM3YjI4NTUyNjFmOTA4M2NjZjAyMTcxNzIwOA==
Sent 58 bytes
Read:     302 OK CRAM-MD5 login successful
Sending: 303 SELECT INBOX
Sent 18 bytes
Read:     * 11227 EXISTS
Read:     * 0 RECENT
      * OK [UNSEEN 1529] Message 1529 is first unseen
      * FLAGS (\Deleted \Seen \Flagged \Draft \Recent)
      * OK [PERMANENTFLAGS (\Deleted \Seen \Flagged)] Limited
      * OK [UIDVALIDITY 1] UIDs valid
      303 OK [READ-WRITE] SELECT completed
reconnect successful on try #1
Sending: 304 LOGOUT
Sent 12 bytes
Read:     * BYE FirstClass IMAP4 server logging out
ERROR: * BYE FirstClass IMAP4 server logging out at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1473
    Mail::IMAPClient::_get_response('Mail::IMAPClient=HASH(0x100d86648)', 304, undef) called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1406
    Mail::IMAPClient::_imap_command_do('Mail::IMAPClient=HASH(0x100d86648)', 'LOGOUT') called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1312
    Mail::IMAPClient::_imap_command('Mail::IMAPClient=HASH(0x100d86648)', 'LOGOUT') called at /Library/Perl/5.10.0/Mail/IMAPClient.pm line 1925
    Mail::IMAPClient::logout('Mail::IMAPClient=HASH(0x100d86648)') called at ./imapsync line 2083
reconnecting to MASKED, last error: * BYE FirstClass IMAP4 server logging out
Connecting via IO::Socket::INET to MASKED:143 Timeout 600
Connected to MASKED errno(Socket is already connected)
Read:     * OK FirstClass IMAP4rev1 server v9.221 at mail.uei-schools.org ready
Sending: 305 AUTHENTICATE CRAM-MD5
Sent 27 bytes
Read:     + PDIwMTAwODEwMTc0MTQ3Ljc2MzE4ODI4QHVlaS1zY2hvb2xzLm9yZz4=
Sending: dGxlZ2F1eCA0NGFkNjZhZWFlMTMyMTNiZWZjZWI1MjA2N2I1NTUyYQ==
Sent 58 bytes
Read:     305 OK CRAM-MD5 login successful
Sending: 306 SELECT INBOX
Sent 18 bytes
Read:     * 11227 EXISTS
Read:     * 0 RECENT
      * OK [UNSEEN 1529] Message 1529 is first unseen
      * FLAGS (\Deleted \Seen \Flagged \Draft \Recent)
      * OK [PERMANENTFLAGS (\Deleted \Seen \Flagged)] Limited
      * OK [UIDVALIDITY 1] UIDs valid
      306 OK [READ-WRITE] SELECT completed
reconnect successful on try #2
Sending: 307 LOGOUT

Any help is greatly appreciated,

Thanks
Adam