Nice work!
It would probably make sense to have that preg_match expression come from a configuration option.
1 Like
Hi guys,
Ok first @pstevens big kudus for the initiative on documenting bounce it’s definitely needed and most welcome. I also had to go through a bunch of topics here and searches all over to grasp the surface of what the heck is the bounce email handling here.
Amazing and thank you!
Now, second… Mautic 
definitely didn’t know this. Since I only glanced at it, do you install it on the same server as sutiecrm and how do you connect both?
About the "only works for mailer-daemon|postmaster … I wonder … how hard would it be to have a library with terms to match that an user could enable/disable or even add more terms for specific use cases?
But documenting where to find that line to add more terms is also kick-ass.
1 Like
That regular expression could be added to the Email Bounce handling account configuration.
It’s just a text field, like “connection string”. It could then be used inside the campaign_process_bounced_emails function.
1 Like
@pgr @maverickws yes, I’m thinking along the lines of how to either:
- extend the function to allow more options in /custom/
- adding a variable you could include in config_override or something to add more senders.
- Adding a field in the inbound bounce handling type email account record that you could enter other bounce senders. (this may be a little more involved).
Re: Mautic… there are lots of posts on here about this. It really is a huge compliment to SuiteCRM. If you have questions about Mautic specifically, open another thread and I’ll be happy to answer there.
2 Likes
I’m not a developer so helping with code will be hard but I’m all ideas 
wouldn’t it be interesting to start a new topic just for this?
EDIT: About Mautic I’ll take a while to search here and read all about it. If then I need anything else I’ll ask. Cheers
Another thing regarding email
I saw the comment on overhauling the whole email system. I’m assuming this does not happen bc people integrate with Outlook and that’s that?
Anyway, on the topic of the email upgrade, I found this topic on the suggestion box forum started in 2014 … till today people asking for caldav/carddav support.
Do these suggestions from here actually get to core devs of SuiteCRM?
It’ seems that this can get quite tricky…
There was a suggestion box / idea graveyard on Trello, lasted for a couple of years, then disappeared, along with the hundreds of suggestions buried therein.
The devs know about most of this, but they are outnumbered, like, a thousand to one. So much to do, nobody to do it.
Email got important improvements recently, the email accounts management is less flaky. For the rest I would bet on incremental improvements. I would be doing quite a lot here if PRs were getting merged, but they’re not.
(my PowerReplacer add on can be used as a sort of low-code language to make inbound email process email intelligently, linking them to records, assigning them, notifying people, testing conditions, etc. You can do amazing stuff when you start getting control of email, and you have a full-blown corporate CRM behind you)
@pstevens ’ work on investigating these complex areas of SuiteCRM has been amazing.
1 Like
LOL I’m just looking at this part now! So basically, ANY email that is returned from MAILERDAEMON or POSTMASTER is marked as “bounced”. There is a secondary procedure which I’m working through now which determines “Why” it bounced.
Basically, the checBouncedEmailInvalid function returns “true” IF … lines in the email description that start with “Status:,” followed by a sequence of three numbers separated by periods. If it finds a line with a permanent error status (5.X.X) and a line that contains “Diagnostic-Code: smtp; 550,” it returns true to indicate that the email is invalid due to a “550” SMTP error.
Once it returns true then another function: createBouncedCampaignLogEntry marks $bounce->activity_type = ‘invalid email’ otherwise, it marks it as ‘send error’. So far pretty much everything I’ve tested gets marked as “send error”. Based on the fact that it’s only looking at the two senders (or an additional one I’ve specified), maybe this could be relaxed a bit and look for the words “error” and “550”, you may get some false positives, but might be way better than getting almost no matches.
1 Like
Ok here it is! My compendium on email bounce handling. If anyone has anything else to add or questions I haven’t answered, let me know.
4 Likes
Bookmarked! Thanks!
After that write-up, what would be your suggestions for future improvement of this in SuiteCRM? We’ve already discussed the ability to configure the regular expression that matches bounces, what else do you think would be helpful?
1 Like
One other thing that would be great, is if there was an API for each of the major outbound mailers like: sendgrid, AWS, Elastic Email, etc. The API bounce handling is much more accurate if you get it right from the sender via API. Mautic gives the user a choice of doing a bounce account like SuiteCRM or selecting one of the popular bulk mailers from a dropdown and just entering the API info.
This is so much easier, then you don’t have worry about the bounce account at all if your primary outbound mailer is a third party like Sendgrid or AWS. They all pretty much have API integration that returns the bounce info.
I had a look at trying to add the different sender as a variable one could enter in the Inbound mail account, then contributing the code to Github. Adding the variable to the function is not a problem. Adding the field to front end editview/detailview I think I could do, but adding a new field to SuiteCRM base code to be included when installed and updated would be a giant stretch for me. Not really sure even where to start with that.
Hi @pstevens amazing work man congrats!
About the email API I will toss my 2 cents:
To send I have found (specially with v8.4, much better than the previous versions) that AWS for example works great.
Configure the SMTP server, username and password which are the credentials given by AWS SES and good to go.
Complaint and Bounce handling by Amazon SES is done using Amazon SNS (simple notification system) it requires some configuration. And it doesn’t work with returned emails, bodies and headers from what I remember on top of my head.
From what I understood also, (and please do correct me if I’m wrong) the bounce email handling account, that does have to be added to SuiteCRM and then selected when building a campaign requires it to be an actual inbound mailbox, but that mailbox is on your server not an AWS inbox.
So I guess it could be interesting yes, but considerations I’d say how the email account we select when building campaigns will be considered when using such plugins, if a message bounces and is delivered to said inbox, when a notification has already been generated, will it create a double bounce or something?
Subscription protocols available for SNS are SQS, Lambda, HTTP, SMS, email and mobile endpoints.
1 Like
@maverickws I’ve never tried AWS specifically, but my understanding from their documentation is that if you set the bounce path and check off “include original headers” it will return the original header containing the campaign id.
So what happens when you select a bounce handling account in the campaign is that it includes a bounce path in the header or the emails which contains your bounce email address which typically is different from the reply to. Most servers honor this and send bounces to the bounce return path specified. However, bulk emailers like AWS, Sendgrid, etc. have their own bounce handling systems and do not honor your bounce path and replace it with their own. So… it’s necessary to configure them to send the bounce notifications to your bounce account. (configuring just in SuiteCRM won’t work because it gets stripped out).
Sendgrid includes the original header by default with the campaign id, but AWS you need to specify that, there is an option for it. (Again, I’m going by reading the documentation but I’d love someone to chime in if that is in fact the case).
1 Like
Ok. I might have to revisit SES and SNS don’t remember some stuff. So trying to understand the final result we get
Configure a bounce SNS topic to deliver (format, email) to the bounce mailbox.
On SuiteCRM
Set Amazon SES SMTP on outbound,
the bounce mail account on incoming
create the campaign, set the bounce mailbox (that will get the bounces handled by amazon sns).
And figuring the Amazon bounce configurations supersede bounce configuration that may be sent by Suite?
Yes, when you send email through a 3rd party like Amazon, Sengrid, Mailgun, etc. they supercede the original mail headers and insert their own bounce path. So you have to configure the 3rd party sender to return bounces to your bounce account (with original headers).
SuiteCRM will process the bounce account via cron job and create the bounce entries in the campaign.
1 Like
Well, sweet! 
ahah. I’ll give this a go not sure today.
The other problem with third party senders is they often don’t return the email from POSTMASTER or MAILER-DAEMON, so SuiteCRM doesn’t parse them. (Sendgrid is an example, which is why you have to mod the line of code in check for bounced emails). However, I can confirm that AWS sends back from MAILER-DAEMON @amazones.com so SuiteCRM will parse it as long as it is delivered to the bounce account with the campaign ID in the header (by selecting the include header option).
This is all theory, since I have not tried AWS specifically, but if you do, please share results. Also if can share screen shots or steps to add the right bounce account to AWS, I’d love to include those in my article too!
1 Like
OK so I have an update with AWS. When you configure SNS in AWS to send bounce notifications you have the option to include original header, which is great because you get the camapaign ID back.
However, there are two problems:
-
The email comes from no-reply@sns.amazonaws.com, not a big issue I can edit SutieCRM to look for this sender.
-
Unfortunately, it comes in JSON format and not plain text. I’ll have to test and see if SuiteCRM still finds what it needs or I have to JSON decode before SuiteCRM can parse it.
Hi there @pstevens thanks for the update. Unfortunately I have not advanced in this matter. Will keep checking your updates and when I’m up to test.