Updating from SugarCRM 6.5.17 to SuiteCRM 7.11.12

I have been experimenting with the above upgrade on a fairly large installation for a while doing this in a docker environment on a copy of the database and the site before doing a real upgrade. Since there a number of changes, I needed to perform this in a number of steps:

  • The old environment uses MySQL 5.1.73 and php 5.4.16 from which I dumped the database.
  • The docker environment defaults to MySQL 5.5.68 in the container and I stayed with php 5.4.16. As an aside, MySQL 5.1.73 uses utf8 (3-byte) and does not support utf8mb4 whereas 5.5.68 does.
  • After dumping the old database from MySQL 5.1.73 I reloaded it in the container MySQL 5.5.68 and copied the source files to the proper directory.
  • Next I ran the SugarCE-Migration-6.x-to-Suite-7.6.6.
  • The next step was upgrading from php 5.4.16 to 7.0.27.
  • Followed by upgrading using SuiteCRM-Upgrade-7.6.x-to-7.8.31.
  • Next was upgrading using SuiteCRM-Upgrade-7.8.x-to-7.11.12 while still on php 7.0.27 which should work according to the upgrade matrix.

A couple of questions:

  • Since MySQL 5.5.68 defaults to utf8mb4 and the old database used utf8, where would it be optimal to convert tables to utf8mb4?
  • And when converting formats, how do I address issues such as “Index column size too large. The maximum column size is 767 bytes.” and “Specified key was too long; max key length is 1000 bytes”?
  • The UI was familiar until the last upgrade from 7.8.31 to 7.11.12 where it seemed to have lost functionality and required more clicks to do some things. As an example, while I was able to send and receive e-mails at the step when I was still at 7.8.31, the UI is either borked or something is not right when at 7.11.12. Accessing the e-mails has now become much harder and I cannot see how I can simultaneously view multiple accounts like in older versions?
  • Nor does it seem to be simple to move e-mails between folders as it was before.
  • Switching between folders seems cumbersome and in fact I have not gotten that to work since all folders are no longer visible in my default mail account?

So, I could use a recommendation on at what step to convert existing MySQL tables from 3-byte utf8 to full utf8, ie utf8mb4, how to resolve the two types of error messages I listed above, and why e-mail now has become so complicated to use in 7.11.12?

Thanks.

I should also have added that:

  • The Sent e-mails is empty after the upgrade step from 7.8.31 to 7.11.12, as is Archived e-mails. Both showed up at the earlier step.
  • At 7.8.31 it pulled in new e-mails but after 7.11.12 it does not. Nothing else have changed in the docker setup. Has something changed in how the software operates internally between 7.8.31 and 7.11.12 with respect to e-mails?

What could have happened?

As generic advice, have a look at this article, if you haven’t yet

But it seems to me that you are already doing it right so perhaps that won’t add much.

About your questions:

  • I don’t know much about MySQL encodings, I can’t help there

  • there is a new Email module since 7.9. It changes a lot of things and not necessarily for the better. The new module has been, let’s say, problematic. Any way, I think you will have to stick with it, there’s really no way around it. So try to see what works, what doesn’t, and to work around limitations the best you can.

OK, luckily I am doing this on a copy of the database and the site using docker so I can do this over and over again until it works. I think I follow the requirements in the upgrade matrix - UNLESS there are some errors in the upgrade requirements. Trying to follow the requirements is why I have to do it in several steps…

As for the change in MySQL encoding I will tomorrow edit the mysqldump file from MySQL 5.1.73 which does not allow utf8mb4 (ie 4-byte encoding) before importing it into MySQL 5.5.68 which does. It ought to be a simple search-and-replace of both the utf8 encoding and the collation sequence for each table in the file. Should be straightforward (I know, famous last words…)

The troubles with the e-mail module are more worrysome, however. Trying the new migration in docker tomorrow, I will stay with SuiteCRM 7.8.31 which is the last release in the 7.8 branch as far as I can tell. BUT, SuiteCRM seems to be up to 7.11.x and why there should still (?) be problems in the e-mail module I cannot fathom. This is one of the KEY modules in any CRM software, is it not? Any bugs should have been reported a long time ago I would think.

I could tell that in the previous migration attempts where I ended up at 7.11.12 attempts to repair e-mail etc in the Admin module fouled up things royally and I ended up with no working e-mails whatsoever. Although I am pleased to hear it might not be my fault I am very concerned that this is an issue.

You’re right about email.

You seem to be echoing what I proposed to do here:

  • Fix Email settings screens. This isn’t so hard, we can get this right once and for all.
  • Fix Email code. This is hard, but I really don’t know why it keeps getting postponed. Email and Campaigns are essential for SuiteCRM.

But I am afraid that didn’t exactly take off as intended.

I hear and I tried what is recommended at https://docs.suitecrm.com/user/core-modules/emails/#_upgrading, ie “Sync Inbound Email Account Tool” followed by repairing but ended up with a broken system.

But, since SuiteCRM is up to 7.11.18 and you are saying that e-mail functionality is broken as of version 7.9 going forward, how can people still use it? Is that when upgrading from an older version it gets broken and new installations work as expected? Or, is it wider than that? If so, does SuiteCRM have a future?

Tough questions…

There are several levels of “broken”.

Config screens are somewhat broken but usually this can be made to work, eventually, after some pain. For other people it just works ok.

Usage of the email module writing and sending emails. This works, but with the limitations that you see. So you can evaluate the functionality that is there and decide if it’s enough for you.

The same goes for the templating capabilities. You have my add-on if you want better functionality.

About SuiteCRM having a future, I am uncertain. I’d say at the level of operation we’ve seen during the pandemic, things look pretty bad. But we can hope for a post-pandemic world and some return to pre-pandemic levels. I do think the Email module is central to that survival, and I am sure that “more of the same” way of handling that challenge just won’t cut it. A new beginning is necessary in terms of how to approach it - I would put a top-level developer full time on it, until it is fixed, no matter how long that takes. But I don’t make any decisions here.

I think what some companies do with SuiteCRM is actually invest a bit of money with customizing and bug-fixing the functionalities that are critical to them. With all the money they save (no licenses etc) this is a likely a viable approach.

Hm, not so good news (obviously). I was hoping this would be the way forward from SugarCRM after they dropped the CE version.

I am giving my upgrade attempt described above another shot (did not have time yesterday.) Right now I plan to stay at 7.8.31, ie the latest release in 7.8.