Hey all, I normally use Mautic for my outbound email campaigns but I have the occasion to use SuiteCRM for a client. So, I thought I’d brush up on how bounce emails are handled and how the bounce account works.
The documentation is rather sparse in this regard, even going back to SugarCRM CE it’s rather sparse. I also read through a bunch of post here as well and it seems there is a lot of confusion So, here’s what I propose. If people reading this post can add to the knowledge base, I’ll take everything learned, test it and write a comprehensive guide to email bounce handling and post it in the show and tell for everyone to refer to.
So my current understanding is as follows:
The bounce handling account receives bounced emails via the return path in the email header.
SugarCRM will first check if the message is from “MAILER-DAEMON” or “POSTMASTER”, if the message is not from either it will be ignored. This allows SugarCRM to ignore messages that are not bounce notifications. When an email server sends a bounced message it must provide a “From Address”. A common “From Address” used by email servers for bounce notifications is “MAILER-DAEMON@whatever_domain.com” or “POSTMASTER@whatever_domain.com”.
SugarCRM then checks the contents of the bounced message for an identifier key. Campaign emails are required to have an unsubscribe link in the body of the message. In SugarCRM this unsubscribe link contains an identifier key that is unique to the recipient and campaign message; this means that SugarCRM can identify the intended recipient and source campaign message from this single key. An unsubscribe link looks as follows: index.php?entryPoint=removeme&identifier={uniquely generated key} The identifier key is the only way SugarCRM can know the intended recipient of a bounced message. If the key is not present in the bounced message, the message will be ignored. This does pose problems when the original message is not included with the bounced message. Unfortunately, at this time, messages cannot be processed if the original message is not included with the bounced message.
The final step in the process is to lookup the targeted recipient and source campaign based on the identifier key and update the campaign log. Successfully processed bounced messages will appear on the status page of a campaign.
(is this correct???)
What happens when an email is found as bounced? Does it automatically mark the email address as invalid or do I need a workflow for that?
If anyone can add to this thread some best practices, clarifications, and how they administer bounced emails (or links to any relevant content) I would appreciate it and we can together make a great guide!
That text seems to be describing what this function does
If you write a text about this, I suggest adding it directly in the Documentation site, it is an editable wiki. It’s better not to use SugarCRM docs text directly there, we’re not sure if there are licensing implications. So it’s better to just write a new text or edit it significantly.
Another quick question on this topic if anyone has experience (I just don’t use campaigns all that often, so some of the minute details I don’t know for sure)… When a bounce comes back from a receiving server and instead of sending back an email with the body of the original email sent, you get a file attachment with the original email, does it still recognize this as a bounce?
From the documentation, I’m thinking that it wouldn’t because there would be no campaign ID in the body of the email. This is a hard one for me to test, so if anyone knows for sure, please chime in.
Interesting question. I am just guessing, but I think it could work well even when the bounced email comes as an attachment, because the function that looks for the id is a simple regexp match, done on the raw email, which is just text.
Thanks @pgr I know its hard to know exactly without testing it and I have to send it to a server that I know is going to bounce it in that way to confirm. I’ll have to look through some past bounced emails and see if I can find a server that bounces like that and test. I think that’s the only way to know for sure unless someone has experience with this and chimes in.
It really is amazing when you delve into this stuff all the nuances that really should be documented so you can plan a campaign effectively. Hoping to produce a deep dive document specific to this subject when I’m done.
I think SuiteCRM should be excellent (not just good) at email. I don’t think we should even have to use Mautic or other outside tools. We should just keep improving SuiteCRM’s email system.
Apart from all the buzzwords and new tech, having your company email well connected to a centralized corporate database, that’s often the poor man’s “killer app”. Lots of companies could really fly past their competition if they had that.
I really wish there was more development in this area for SuiteCRM. I’m typically up against Salesforce and Hubspot. Salesforce has Pardot (essentially Mautic) and Hubspot has both features combined with more of a focus on the marketing. Both SuiteCRM and Salesforce (without Mautic or Pardot) lack some key marketing features that are more or less standard in other CRM’s like Hubspot. Primarily:
Lead engagement tracking: form fills, website visits and email opens, sends can be tracked by the lead or contact.
Lead scoring: assigning points to engagement activities that can be strategic for the sales team in terms of follow up.
Drip Campaign ability: While SuiteCRM can send emails on a specific date easily, sending emails on a relative date is a bit of a challenge (it’s possible with workflow but a mess if you have dozens of them for different segments).
Segmentation: While SuiteCRM has Target Lists, Mautic/Pardot/Hubspot have segmentation. You define the segment, not the list. People get put in a segment based on criteria, not membership in a list.
Web form builder: Mautic and Hubspot have robust web form builders that seamlessly import or update data from the form.
Landing Page Builders: Every marketing campaign needs both a landing page and contact form. Mautic and Hubspot make this easy.
Intelligent Campaigns. - You can not only send campaigns at a relative date, but design campaigns such that actions like sending emails are based on criteria like “did the contact open the previous email” You can really build some advanced follow up campaigns.
While I’d love to see this in SuiteCRM, it’s a HUGE mountain to climb from where email campaigns are now in SuiteCRM. Mautic is open source as well, and is so far ahead in all these areas I’m wondering if some kind of partnership or working together might be a more optimal solution. SuiteCRM is a World Class CRM that stacks up against the best. Mautic is a World Class marketing platform that stands up against the best. Together, they are really a cutting edge open source sales and marketing stack that’s pretty hard to beat.
That’s my 2 cents on where I’d like to see things go.
If you send through a bulk email sender (like most sensible people would) like Sendgrid, they do not honor the return bounce path. However, you can set the return bounce path in Sendgrid, to send an email to your bounce account. HOWEVER, it comes from Sendgrid, not the receipients domain, it does include the senders email in the body, but not the campaign ID and it does not get marked as bounced by SuiteCRM’s “Run Nightly Bounced Emails” job.
Wow, this is another can of worms! I’m trying to see how ProcessBouncedEmails.php gets executed at all! I have a log entry at the beginning of the function and I have both ran both “Run Nightly Process Bounced Campaign Emails”, but it does not run. It does not run when “Check Inbound Mailboxes” job runs.
The only reference to this file is in InbouncEmail.php So I’m going to have to troubleshoot that.
case 'bounce':
require_once('modules/Campaigns/ProcessBouncedEmails.php');
campaign_process_bounced_emails($email, $header);
break;
There are mechanisms to avoid re-importing already-imported emails, these are typically a nuisance when you’re developing and testing.
Set your breakpoints early in the calling function (importOneEmail) then step down until you see in which condition it’s going around the ProcessBouncedEmails call.
Thanks @pgr, I’m trying to write a bit of code to:
add no-reply@sendgrid.net to the list of senders to check for bounces (that’s easy).
Add functionality to parse the attachments and look through the attachments for “550” and the campaign ID.
Testing this is going to be tough though if I have to generate a new email every time while I’m developing it.
This would be great functionality to add because most bulk emailers like sendgrid and AWS send back an email that has the original bounce email attached as a file instead of forwarded. This would greatly improve bounce handling for most people that use bulk emailers.
OK, I think I fully understand how bounce handling works! I’ve done a bunch of testing.
So, email attachments are parsed with the existing code. I’ve tested and this works. However, if your email sender does not return your email from: MAILER-DAEMON|POSTMASTER
It will not work, because it ONLY looks at emails from MAILER-DAEMON|POSTMASTER.
So I added this to capture emails from sendgrid:
if (preg_match('/MAILER-DAEMON|POSTMASTER|no-reply/i', (string) $emailFromAddress)) {
You could make it stricter and look for no-reply@sendgrid, but it then subsequently looks for error codes and the campaign id, so it doesn’t really have to be that strict.
This line is modified in the function: campaign_process_bounced_emails
I think you do want to be very specific about your bounce reply. Upon further testing, ANY email that comes from MAILER-DAEMON OR POSTMASTER OR NO-REPLY (in my mod) will be marked as bounced. So you probably want to be very specific like:
if (preg_match('/MAILER-DAEMON|POSTMASTER|no-reply@sendgrid\.net/i', $emailFromAddress)) {
Another process looks for 550 in the email reply and marks the email address as INVALID. Bounced emails and invalid emails are not handled the same (unless they come from one of the email addresses listed, then they are marked as bounced).
Ok first @pstevens big kudus for the initiative on documenting bounce it’s definitely needed and most welcome. I also had to go through a bunch of topics here and searches all over to grasp the surface of what the heck is the bounce email handling here.
Amazing and thank you!
Now, second… Mautic definitely didn’t know this. Since I only glanced at it, do you install it on the same server as sutiecrm and how do you connect both?
About the "only works for mailer-daemon|postmaster … I wonder … how hard would it be to have a library with terms to match that an user could enable/disable or even add more terms for specific use cases?
But documenting where to find that line to add more terms is also kick-ass.
@pgr@maverickws yes, I’m thinking along the lines of how to either:
extend the function to allow more options in /custom/
adding a variable you could include in config_override or something to add more senders.
Adding a field in the inbound bounce handling type email account record that you could enter other bounce senders. (this may be a little more involved).
Re: Mautic… there are lots of posts on here about this. It really is a huge compliment to SuiteCRM. If you have questions about Mautic specifically, open another thread and I’ll be happy to answer there.
definitely didn’t know this. Since I only glanced at it, do you install it on the same server as sutiecrm and how do you connect both?
wouldn’t it be interesting to start a new topic just for this?