Test Suite Not Working

A little over a year ago I update my SuiteCRM instance to 7.10.2 from 7.8. Up until recently I haven’t been using tests at all and would like to start implementing them now. I realized that the test framework was not installed on my instance.

I just downloaded the 7.10.2 repo (https://github.com/salesagility/SuiteCRM/tree/v7.10.2/tests) and copied over the tests folder and the codeception.yml to my instance.

When running ‘./vendor/bin/codecept run demo’ I get this error:
Can’t connect to Webdriver at Please make sure that Selenium Server or PhantomJS is running.
So I downloaded selenium server and now I get this error:
[Facebook\WebDriver\Exception\SessionNotCreatedException] Unable to create new service: ChromeDriverService

I guess now it looks like I need to install ChromeDriver? But the readme doesn’t say I need to install anything extra and that all I need to do is run composer install. Am I doing something wrong here? Or do I need to do this additional setup not in the readme?

I am unsure how to answer here, because the tests have evolved so much since that version. It’s a very active part of the codebase, and sometimes with significant changes. We don’t use Selenium anymore, for example…

Also, the Docs have been changing too, to keep up with these changes.

Can you upgrade to the latest 7.10?

You can also try asking on the Developer’s Gitter channel about the tests, Connor Shea will surely be able to help you

1 Like

After messing around for a bit longer I got some tests working and realized I just didn’t really know what I was doing when I posted this. I am planning on upgrading to the latest 7.10 in the next few months but I cannot do it quickly/easily.

The documentation though doesn’t mention how to create custom tests. The automated tests are great but if I make any changes to the base tests those would just be written over during an upgrade, wouldn’t they?

Right now I have just created a separate tests project in the custom directory with codecept to run custom tests. And before deploy I would just run the automated tests provided by suiteCRM and mine separately.

Is this how it should be done?

Yes, I think that’s it. I would only add that if your new tests are for generic functionality, you can change them in core, by making a PR on Github, and that way everybody can benefit.

If they are for your own custom functionality, then probably it’s best to keep them separate, yes.

There’s another dimension to this which is to get the tests to run automatically from GitHub on Travis whenever a PR is made. That can also be configured for specific tests on your own cloned repo. But I am not familiar with that at all, I have no idea how to do it. Of course, the simplest thing is to just ignore this dimension and run the tests locally, and manually.

:wave: I’m glad you were able to figure out how to run the tests locally, we’ve changed the test suite a lot in the last few months since it had a bunch of problems, so I’m honestly kind of surprised the instructions in the docs worked for an older version like that.

I’ve not been able to create custom tests yet for our instance of SuiteCRM since I haven’t had the time yet, but a lot of the work I’ve done on the open source project over the past year has been working toward that goal.

The documentation though doesn’t mention how to create custom tests. The automated tests are great but if I make any changes to the base tests those would just be written over during an upgrade, wouldn’t they?

It should be mostly possible now, but there’s not really any documentation for it since no one else has really done it as far as I’m aware.

Our team has some pretty heavily customized tooling I’ve built that allows us to commit only the CRM files that have been modified to our git repo, which makes it easier to perform upgrades and such. Then a Robo task can be run that pulls in all the other files from the upstream repo to create a working CRM instance.

I’m not sure I’d necessarily recommend doing all that, but that’s how my team gets it to work.

I was finally able to update to 7.10.22 but I am now running into another problem. I have also created a separate clean install of 7.10.22 and am running into the same issues.

  1. When I try to run tests using robo (./vendor/bin/robo tests:unit) I get an error that shows

[Exec] Running ./vendor/bin/phpunit --colors --configuration ./tests/phpunit.xml.dist ./tests/unit/phpunit
‘.’ is not recognized as an internal or external command,
operable program or batch file.

  1. When I try to run the tests using codecept all the test suites except for unit work. When trying to run the unit test suite I get an error telling me that the unit suite does not exist. I found that this is because there is no unit.suite.yml file in the tests directory. When I add it in, the error goes away but not tests run. It looks like this file is not included in the 7.10.22 or in 7.11 so I get the feeling something is wrong with my set up.

I have followed the documentation and this is an issue even on a clean install for 7.10.22 for me.

Oh, are you using Windows? I’m not 100% sure how to run this in Windows, but it might need to be changed to support it. It probably doesn’t treat the ./ as the current directory like bash does.

It might be easier, in Windows, to get this running from inside WSL Ubuntu (an app you install if it’s a modern version of Windows 10

I didn’t think about that. I’m running on windows 10 but through a bash terminal on vscode. So I can run the unit tests using the phpunit command but there is probably something going on when using the robo script because of my set up.