Steps forward in Automated Testing

Good Morning/Afternoon/Evening Everyone!

As part of our Release Notes we would like to demonstrate to the Community that 7.10.5 is more than just a mere bug fix/GDPR feature release. We want to announce that we are marking it as a fundamental step forwards towards a more robust and test driven application.

In the past, we felt our releases just fell short of achieving an ‘excellent’ standard due to the lack of automated testing – there is only so far one can test everything manually before you lose the will to live…
That goes for developers developing as well that without a suitable test suite fixes were applied in faith i.e. only manually tested by peers and dedicated testers…. very tedious and open to untested entry points. So, we felt that with the increasing feedback around Emails and API that we, the product team, needed to take time out to assess what we need in our toolkit to ensure that going forward we feel confident in our releases to release on a random day and all should be working as expected.

Although we stated that updating and improving our test suite was on the Roadmap for 7.11 we felt there was no reason why we couldn’t apply this to the previous branches (7.8.x and 7.10.x) and so that is our goal.

Things we have introduced and why!

[size=4][color=#ff0088]Improved & Passing Travis Build[/color][/size]
If you are aware we have had Travis CI running on each pull request and had been doing so in the past for a number of releases. However it was not quite as accurate as we wanted it to be and was producing a number of false positives due to the tests themselves – that’s not good.
So we made our unit tests stateless (encapsulating the test data per test so they don’t fail due to data from another test) and updated the code where tests were failing accurately (and correctly!)
Read more on the StateChecker here -—automated-testing/#_writing_and_running_tests

[size=4][color=#ff0088]Introduced Mockery[/color][/size]
Mockery is a PHP framework that allows us to mock objects/variables etc to pass into unit & functionality tests. SuiteCRM is a large application and has a lot of areas that require some form of mocking to test accurately. We haven’t yet included anything to mock yet but we anticipate this for when testing the use of external systems i.e. emails
Read more on mockery here -

[size=4][color=#ff0088]Introduced Robo tasks[/color][/size]
Robo (a PHP ) task running framework that provides developers to executed reusable tasks that makes testing, development, setup quicker.
We currently use Robo tasks for test environment set up and SASS rebuilding.
Read more in our documentation of these current tasks -
Read more on Robo here -

[size=4][color=#ff0088]Introduced Code Coverage (CodeCov)[/color][/size]
Code Coverage is the measurable coverage of how thorough your test suite interacts with the methods/functions of the application. The higher the percentage the more test ground is covered.
You can do this already manually but felt that we wanted to advertise our growth and allow community members whom contribute to also be aware of what they provide has an impact on the overall ‘health’ of the test suite and thus the application.
As you can see the coverage isn’t the prettiest and there is still a lot we need to do. But we felt that this visual and public display of what is currently being tested in the system gives us, and potentially community members who provide code contributions, a clear and measurable goal to improve the current status of the coverage with each Pull Requests

Read more on how to view Code Coverage manually here -—automated-testing/#_code_coverage_infos

[size=4][color=#ff0088]Acceptance Tests[/color][/size]
This, we felt, was one of our biggest achievements in the past sprint – the ability to run acceptance tests in our Travis builds. Acceptance tests are tests that are browser driven and re-enacts the user journey to achieve a particular action via the user interface e.g As a user I click here and here to create a field.
We have included only a few of the essential tests like creating modules, and fields in module builder, save records, search and delete. These all currently pass but there is a [color=#ff0088]lot more tests that need to be written to cover the entire application[/color] but this is a great start!
You can read more on how to write, and run acceptance tests here -—automated-testing/#_acceptance_demo_and_install_tests and—automated-testing/#_codeception_functional_and_acceptance_tests

There is still a lot we need to do to get to the goal of successfully achieving continuous deployment (which would be fantastic!). But I hope you and the community all agree this is a tremendous step in the right direction.

We are aware that obviously time has been taken out to get the above in a suitable place for release and doing so we had to compromise on resources to tackle the current issues that the community have been raising – i.e. emails and API. So we would like to thank you for your patience whilst we got those essential ‘unseen’ tasks done.

For the next sprint we will be refocusing our attention on the email module and resolving API functionality/testing so if these areas affect you and there are Github issues open do please interact with the threads and help out with testing/contributing when you can!

Thank you again and we all look forward to the next SuiteCRM release in the coming weeks.


The above post was part of the 7.10.5 Release Announcement, but with Ashley’s approval I have moved it to a separate post, because I think it deserves more attention and I know some people don’t read Release announcements, they just read the title and go directly to the download page :slight_smile:

I’d also like to take the chance to leave my own humble comment on this post.

It’s tricky to balance the attention that needs to be given to everyday problems, GitHub issues, bug fixes, and new features, with the attention to the more structural things in a big project. I guess it’s the typical tension between thinking short-term or thinking long-term.

These improvements to the testing framework have taken up a lot of time of the SuiteCRM Product Team these past few months, perhaps causing some reduced ability to move faster fixing email problems, etc. However, I think they are absolutely The Right Thing To Do. We get waaay too many regressions when we change code in SuiteCRM - somebody from the Community or the Product Team edits the code to improve something, with good will, but even with careful processes and assessments, something else in the product breaks. It’s big, it’s PHP, and it contains some less-than-optimal bits of legacy code.

Automated tests, as we improve coverage, will greatly help us reduce the risk of regressions, and give us more confidence to go ahead and refactor things that really need refactoring.

The Community is also invited to learn how to write tests, either for new features they provide, or for existing code. Do you love a specific module? Use it a lot? Please consider writing tests for it, you will be making a huge contribution to SuiteCRM, right in the place that matters for you.

Thanks to the team for these steps forward - thanks to everybody for their patience with the areas of the project where it’s not as robust as it should be: the hunt is on! Let’s squash bugs one by one so we can enjoy the immense value that this app offers for an unbeatable price :slight_smile:

1 Like

This is the best addition ever to suitecrm.

I definitely agree that we needed some quality improvement and auto testing is a great step .
Many projects have now testing included at the beginning. Sugare crm is an old project that needed this refresh.

I could see suitecrm not moving at the speed it could, because from release to release you had “one bug out and 10 in” :wink:
It had reached a point that I decided not to update , to avoid new bugs to pop in (and I am using LTS). I could upgrade from 7.8.8 to 7.8.10 and lose feature like going back to see my previous emails!

Some bugs where “solved” and would come again and again … There are so many architectures and tests cases that it was impossible to test.

Really looking forward to the upcoming releases, I am convinced they will be much more stable.


PS: “minor release” MAJOR change you mean:)

Hi, this wold be a great addition but could not find (functional tests) in latest LTS or Dev code. Also the Apendix C mentioned above is also missing. Has this changed? Could you please let me know?

There is this

and the code is all here

Some of those things are only in the latest versions… it’s a tricky balance to add tests that delay our builds a lot, if they require us to build for many PHP versions…