Good Morning/Afternoon/Evening Everyone!
As part of our Release Notes we would like to demonstrate to the Community that 7.10.5 is more than just a mere bug fix/GDPR feature release. We want to announce that we are marking it as a fundamental step forwards towards a more robust and test driven application.
In the past, we felt our releases just fell short of achieving an ‘excellent’ standard due to the lack of automated testing – there is only so far one can test everything manually before you lose the will to live…
That goes for developers developing as well that without a suitable test suite fixes were applied in faith i.e. only manually tested by peers and dedicated testers…. very tedious and open to untested entry points. So, we felt that with the increasing feedback around Emails and API that we, the product team, needed to take time out to assess what we need in our toolkit to ensure that going forward we feel confident in our releases to release on a random day and all should be working as expected.
Although we stated that updating and improving our test suite was on the Roadmap for 7.11 we felt there was no reason why we couldn’t apply this to the previous branches (7.8.x and 7.10.x) and so that is our goal. https://suitecrm.com/roadmap/
Things we have introduced and why!
[size=4][color=#ff0088]Improved & Passing Travis Build[/color][/size]
If you are aware we have had Travis CI running on each pull request and had been doing so in the past for a number of releases. However it was not quite as accurate as we wanted it to be and was producing a number of false positives due to the tests themselves – that’s not good.
So we made our unit tests stateless (encapsulating the test data per test so they don’t fail due to data from another test) and updated the code where tests were failing accurately (and correctly!)
Read more on the StateChecker here - https://docs.suitecrm.com/developer/appendix-c—automated-testing/#_writing_and_running_tests
[size=4][color=#ff0088]Introduced Mockery[/color][/size]
Mockery is a PHP framework that allows us to mock objects/variables etc to pass into unit & functionality tests. SuiteCRM is a large application and has a lot of areas that require some form of mocking to test accurately. We haven’t yet included anything to mock yet but we anticipate this for when testing the use of external systems i.e. emails
Read more on mockery here - http://docs.mockery.io/en/latest/
[size=4][color=#ff0088]Introduced Robo tasks[/color][/size]
Robo (a PHP ) task running framework that provides developers to executed reusable tasks that makes testing, development, setup quicker.
We currently use Robo tasks for test environment set up and SASS rebuilding.
Read more in our documentation of these current tasks - https://docs.suitecrm.com/developer/automatedtasks/#_robo
Read more on Robo here - https://robo.li/
[size=4][color=#ff0088]Introduced Code Coverage (CodeCov)[/color][/size]
Code Coverage is the measurable coverage of how thorough your test suite interacts with the methods/functions of the application. The higher the percentage the more test ground is covered.
You can do this already manually but felt that we wanted to advertise our growth and allow community members whom contribute to also be aware of what they provide has an impact on the overall ‘health’ of the test suite and thus the application.
As you can see the coverage isn’t the prettiest and there is still a lot we need to do. But we felt that this visual and public display of what is currently being tested in the system gives us, and potentially community members who provide code contributions, a clear and measurable goal to improve the current status of the coverage with each Pull Requests
Read more on how to view Code Coverage manually here - https://docs.suitecrm.com/developer/appendix-c—automated-testing/#_code_coverage_infos
[size=4][color=#ff0088]Acceptance Tests[/color][/size]
This, we felt, was one of our biggest achievements in the past sprint – the ability to run acceptance tests in our Travis builds. Acceptance tests are tests that are browser driven and re-enacts the user journey to achieve a particular action via the user interface e.g As a user I click here and here to create a field.
We have included only a few of the essential tests like creating modules, and fields in module builder, save records, search and delete. These all currently pass but there is a [color=#ff0088]lot more tests that need to be written to cover the entire application[/color] but this is a great start!
You can read more on how to write, and run acceptance tests here - https://docs.suitecrm.com/developer/appendix-c—automated-testing/#_acceptance_demo_and_install_tests and https://docs.suitecrm.com/developer/appendix-c—automated-testing/#_codeception_functional_and_acceptance_tests
There is still a lot we need to do to get to the goal of successfully achieving continuous deployment (which would be fantastic!). But I hope you and the community all agree this is a tremendous step in the right direction.
We are aware that obviously time has been taken out to get the above in a suitable place for release and doing so we had to compromise on resources to tackle the current issues that the community have been raising – i.e. emails and API. So we would like to thank you for your patience whilst we got those essential ‘unseen’ tasks done.
For the next sprint we will be refocusing our attention on the email module and resolving API functionality/testing so if these areas affect you and there are Github issues open do please interact with the threads and help out with testing/contributing when you can!
Thank you again and we all look forward to the next SuiteCRM release in the coming weeks.
