Make a quick vote 🗳 ️How do you refer to SuiteCRM extensions: add-ons or plug-ins? Click here to have your say 👈

SuiteCRM stopped working: folders_rel folder filled up the entire disk space

Hello everyone,

I have a self-hosted SuiteCRM 7.11.19 with Email module monitoring several our inbound emails, with one of them having over 14GB of emails (old email box).

SuiteCRM uses a cron job to download new email every 5 minutes, and it seems to be working fine.

However, earlier today I could not longer log into my SuiteCRM, it said my session has expired.

I logged into my system over SSH and found out the disk was full (100GB) and narrowed it down to the overgrown MySQL database with the folders_rel table taking over 70GB).

71G     /var/lib/mysql/suitecrm/folders_rel.ibd

How can I safely clean up the folders_rel table?

Is this happening due to having Auto-Import function enabled on the inbound email boxes?

Any tips would be greatly appreciated.

Does your folders_rel table have many rows with deleted=1 ?

Perhaps you can start by clearing those, it shouldn’t make a difference.

But make sure you create full backups, just in case…

No, not a single one.

I’ll try to figure out why there are so many records. Other people have reported that each individual picture is stored as a note / an extra record in this table. Not sure if that’s at the root of this problem though. I think the issue is with Auto-Importing every single email into the database.

Maybe there is a way to delete all the attached pictures from the database (we certainly don’t need them under SuiteCRM).

Although I don’t use inbound email in SuiteCRM, and so I am not very knowledgeable on this, from what I’ve heard in these forums it’s definitely a good idea to manage the size of your IMAP inboxes in ways that hide the large amounts of email from SuiteCRM, because SuiteCRM will hit performance issues much sooner than any Gmail or Outlook.com.

If you can get a policy of removing or archiving older stuff, or using IMAP labels (folders) to keep a lot of stuff only server-side, and clearing that from SuiteCRM, it should really help.

Try to ensure that only stuff you’re currently working on, or that is relevant to what you’re working on, is stored in SuiteCRM.

Thank you for your input!

I’m currently investigating this issue, and it seems the problem is not with the amount of emails (or the total size of the emailbox), but rather there’s a bug/issue that fill in folders_rel table with junk.

The issue may be related to this one:

since my log file used to fill up with the mentioned error messages.

I am not sure about the purpose of this folders_rel table, but as far as I understand it should not be nearly as large as it is. I think its purpose is to maintain a relationship (a hierarchy) of emailbox folders, so it shouldn’t ever change except for when new inbound email accounts are added to SuiteCRM and/or those accounts get new folders/labels.

It appears that SuiteCRM fails to detect the right folder and attempts to re-create it every time it saves a new email.

Forgot to mention: folders_rel table contains over 125M records!

Also, it may be that the issue occurred after I applied a “hotfix” from this commit:

Either I applied it incorrectly, or the hotfix doesn’t work properly. Will continue investigating.

Maybe the code is fixed now, and has been for a while, but it had previously filled up quite a lot, due to the old bug. Maybe now it just went over a certain threshold because of new emails and became noticeable.

If you try an aggregate query grouping the entries in that row by month or year of creation, you might get insights as the when this was most problematic.

I would also look into orphaned rows on that table. By “orphaned” I mean rows that reference items that have been deleted are no longer present in SuiteCRM. These should be the ones you can safely clear.

So you would need a SELECT joining folders_rel with Emails table, something like this (untested!)

SELECT id 
FROM folders_rel 
JOIN emails ON emails.id = folders_rel.polymorphic_id AND folders_rel.polymorphic_module = 'Emails'
WHERE emails.deleted = 1;

Then when you’re confident you have what you need (and full BACKUPS!) you can consider turning that into a DELETE…

After some more digging, I found that the same problem happened to others as well.

Looks like my issue is related to #9192:

It seems to have been merged into hotfix branch in Nov 2021, so I wouldn’t face this issue if I had updated the system earlier :see_no_evil:

I’m going to remove the duplicated entries from folders_rel and will attempt to upgrade my SuiteCRM to the latest 7.11.x.

Alright, applying the changes from the commits of the hotfix above solved the issue.

So, to summarize, a solution to my issue was the following:

  1. Apply the fix from Fix Issue #6994: Update pollMonitoredInboxesAOP to double check that … by MikeyJC · Pull Request #9246 · salesagility/SuiteCRM · GitHub
    This makes sure the issue will not re-appear after removing the duplicate records.
  2. Delete the duplicate records from folders_rel. I only kept 3 actual records, corresponding to my inbound_email mailboxes and deleted the rest.
  3. [Optional] Run OPTIMIZE TABLE inbound_email; to free up the disk space

Now SuiteCRM is working again, and looks like no data has been lost.

1 Like