Very slow load of SuiteCRM (server response time > 30 sec)

I’ve got very slow load of SuiteCRM as web site, i.e. server response time is very high.
When start web-site crm.example.com it take more than 30 sec to load web-view (login page). And each requests/links inside CRM take > 30 sec to load.

Here is a test with curl:

curl -s -w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nAppCon time:\t%{time_appconnect}\nRedirect time:\t%{time_redirect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' -o /dev/null https://crm.example.com

Result:


Lookup time:	0.106318
Connect time:	0.307331
AppCon time:	0.327264
Redirect time:	0.000000
PreXfer time:	0.327471
StartXfer time:	32.357595

Total time:	32.357736

Sysinfo
OS: Arch Linux x64, VPS, RAM 8G, SSD, CPU 4 cores
SuiteCRM 7.11.9
Nginx 1.16.1
PHP 7.3.11
PHP-FPM 3.3.11
Zend OPcache 7.3.11
MariaDB 10.4.8

Configs

  • nginx

[code]

server {

listen 443 ssl http2;
listen [::]:443 ssl http2;
root /srv/http/suitecrm;
index index.php index.html index.htm;
server_name crm.example.com;

access_log /var/log/nginx/crm.success.log;
error_log /var/log/nginx/crm.error.log;

client_max_body_size 100M;

ssl_certificate /etc/letsencrypt/live/crm.example.co/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/crm.example.com/privkey.pem; # managed by Certbot

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 9.9.9.9 1.1.1.1 valid=300s;
resolver_timeout 5s;
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
#add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

proxy_buffering off;

 # Enable gzip but do not remove ETag headers                           
 gzip on;                                                               
 gzip_vary on;                                                          
 gzip_comp_level 4;                                                                                                                                
 gzip_min_length 256;            
 gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;                                                                     
 gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

# Block access to stuff in the root
location ~* \.(pl|cgi|py|sh|lua|log|md5)$ {
return 444;
}

# Block access to data folders
location ~ /(soap|cache|upload|xtemplate|data|examples|include|log4php|metadata|modules|diagnostic|blowfish|emailmandelivery)/.*\.(php|pl|py|jsp|asp|sh|cgi|tpl|log|md5)$ {
    return 444;
}

location / { 
    try_files $uri $uri/ =404; 
}	

location ~ \.php$ {
    #include snippets/fastcgi-crm-php.conf;
#if (!-f $document_root$fastcgi_script_name) {
    #    return 404;
    #}
    fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
    set $path_info $fastcgi_path_info;
    try_files $fastcgi_script_name =404;
    include fastcgi_params;
    fastcgi_param PATH_INFO $path_info;
    fastcgi_param HTTPS on;
    # Avoid sending the security headers twice
    fastcgi_param modHeadersAvailable true;
    # Enable pretty urls
    fastcgi_param front_controller_active true;
    fastcgi_intercept_errors on;
    fastcgi_request_buffering off;
    fastcgi_pass    unix:/var/run/php-fpm/php-fpm-crm.sock;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
} 

#location ~ /\.ht {
#    deny all;
#}

}

server {
if ($host = crm.example.com) {
return 301 https://$host$request_uri;
} # managed by Certbot

listen 80 ;
listen [::]:80 ;
server_name crm.example.com;
return 404; # managed by Certbot

}

[/code]- php-fpm

[code]

#/etc/php/php-fpm.d/crm.conf

[crm]
user = http
group = http
listen = /run/php-fpm/php-fpm-crm.sock
listen.owner = http
listen.group = http

pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

[/code]- PHP
Required extensions are set up.
APCu & APC are enabled.
Other options are pretty default.- MariaDB
Default setup.
[/ul]

Logs:
I don’t see any specific error logs.

There is another PHP based service/website on the server and it works great. Different unix sockets are used for php-fpm backend.
I’ve installed suitecrm on Virtual Ubuntu with similar configs and it worked ok.

Also reinstalle the crm service, wiped data, files, databases. But with no luck.

And now I have no clue what to do and what the porblem is.

Any ideas?

A few ideas for you to explore…

  1. If you access the CRM from within the same network (LAN), does the delay disappear?

  2. Are you getting access to both your logs? These should make it clear at least what the CRM is trying to do as it is delayed…

  3. Are your delays always around 30 seconds, or do they vary much?

  4. Check your resources in php.ini (memory_limit, max_execution_time)


Tested fastsgi socket within LAN by cgi-fcgi with the script:


SCRIPT_FILENAME=/srv/http/suitecrm/index.php \
REQUEST_URI=/ \
QUERY_STRING="action=Login&module=Users" \
REQUEST_METHOD=GET \
cgi-fcgi -bind -connect /run/php-fpm/php-fpm-crm.sock

Got the same result ~ 30 sec delay (load of webpage).

I do have full root access. Set up logging level “DEBUG” for SuiteCRM. Nothing fancy from log :frowning:


...
Wed Nov  6 19:56:22 2019 [43881][-none-][WARN] Configuration variable date.timezone is not set, guessed timezone UTC. Please set date.timezo[464/1399]
n php.ini!                                                                                                                                            Wed Nov  6 19:56:44 2019 [46773][-none-][WARN] Configuration variable date.timezone is not set, guessed timezone UTC. Please set date.timezone="UTC" i
n php.ini!                                                                                                                                            
Wed Nov  6 19:56:44 2019 [46773][1][DEPRECATED] Array                                                                                                 
(                                                                                                                                                     )                                                                                                                                                     
                                                                                                                                                      
Wed Nov  6 19:56:44 2019 [46773][1][WARN] CSS File Dawn/yui.css not found                                                                             
Wed Nov  6 19:56:44 2019 [46773][1][WARN] Confirm Opt in is disabled in email settings                                                        
Wed Nov  6 19:56:44 2019 [46773][1][WARN] Confirm Opt in is disabled in email settings                                                        
Wed Nov  6 19:56:44 2019 [46773][1][DEPRECATED] PHP4 Style Constructors are deprecated and will be remove in 7.8, please update your code 
...
Wed Nov  6 19:57:22 2019 [43883][1][WARN] Confirm Opt in is disabled in email settings                                                                
Wed Nov  6 19:57:22 2019 [43883][1][WARN] Confirm Opt in is disabled in email settings                                                                
Wed Nov  6 19:57:23 2019 [43882][-none-][WARN] Configuration variable date.timezone is not set, guessed timezone UTC. Please set date.timezone="UTC" in php.ini!

Re “php_errors.log”. There is no such a file. I’ve changes settings in php.ini:
log_errors = On
error_log = php_errors.log (tried also /var/log/php_errors.log)
#chow http:http php_errors.log
But the file did not appear.

This must be very interesting part. I’ve made checks from web-site, as well as by curl. It gave 32s for web, 30s for curl as delay (± ~20ms).

Resources in php.ini
memory_limit = 128M
max_execution_time = 30
max_input_time = 60

Have tried to change memory to 256M, execution time to 15, input time to 30, other numbers, but nothing have worked.

:frowning:

I think all you need to do is restart your web server so that your php.ini settings can take effect.

Or check that you’re editing the correct php.ini. For this, see Admin / Diagnostics / phpinfo and check the path and the effective values.

I do restart of services (php-fpm, nginx, mariadb) after each change of configs. Sometimes do full server restart. Neither helps.

Editing correct php.ini.
I’ve made test.php with phpinfo() function inside root folder of SuiteCRM. The script is accessible from crm.example.com/test.php, runs without mentioned delay.
Could it be problem of the source code? Or is it just configs failure?

Edit:
And yes, php_errors.log have appeared. Here some logs:

[spoiler]


[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: DynamicField in /var/www/crm/modules/Administration/DiagnosticRun.php on line 655
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: dirname in /var/www/crm/modules/Administration/DiagnosticRun.php on line 656
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening '/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: Connectors in /var/www/crm/modules/Administration/DiagnosticRun.php on line 655
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: dirname in /var/www/crm/modules/Administration/DiagnosticRun.php on line 656
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening '/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(modules/Import/maps/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening 'modules/Import/maps/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: Version in /var/www/crm/modules/Administration/DiagnosticRun.php on line 655
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: dirname in /var/www/crm/modules/Administration/DiagnosticRun.php on line 656
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening '/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: CustomFields in /var/www/crm/modules/Administration/DiagnosticRun.php on line 655
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: dirname in /var/www/crm/modules/Administration/DiagnosticRun.php on line 656
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening '/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: Audit in /var/www/crm/modules/Administration/DiagnosticRun.php on line 655
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: dirname in /var/www/crm/modules/Administration/DiagnosticRun.php on line 656
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(/vardefs.php): failed to open stream: No such file or directory in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Warning:  include_once(): Failed opening '/vardefs.php' for inclusion (include_path='/srv/http/suitecrm:/var/www/crm/include/..:.:') in /var/www/crm/modules/Administration/DiagnosticRun.php on line 660
[08-Nov-2019 16:37:00 UTC] PHP Notice:  Undefined index: comment in /var/www/crm/modules/Administration/DiagnosticRun.php on line 675

[/spoiler]

This must be something specific to your setup. The fact that it is stable taking 30 seconds strongly indicates this is a timeout - something that is hanging your web server, and it is set to timeout after 30 seconds.

This could be a DNS issue, for example, it’s trying to resolve some address, or contact some external site for some reason.

Maybe you can try scanning the connections on your server with netstat during a request… or check nginx error log, or syslog for additional clues. Good luck!

ok. thank you for the support )

I’m complitely lost. Tried to reinstall everything, disabled SSL, stopped all other DB and PHP services, set up NGINX as reverse proxy and managed SuiteCRM through Apache,
BUT nothing works.
And this problem appears only for SuiteCRM.
Really don’t know what to do (

Try Admin / Repairs / Rebuild .htaccess

I’ve just removed it and installed Bitnami docker container (apache at backend, and nginx as reverse proxy). Now it seems to work.
Still don’t know what caused initial problem.

I want to know what Rebuild .htaccess does?

We are running SuiteCRM 7.13, PHP7.4 and for some users loading time for pages is more around 10 sec. We have Nginx running the CRM. Everyone is on same network and has good hardware systems. We don’t have errors in log files too.

I want to know, how to resolve this speed issue? Why is it not running faster for particular users?

Hi @rsp

Some additional data here of your symptoms might help folks come up with suggestions - some of us (like me) are slightly speed obsessed!

Eg:

  • How long ago was this Suite first started ?
  • When did users first start to notice that some of them got slower results.
  • Has it got slowly worse or suddenly worse.
  • If the two users (Fast and Slow) swap computers and login and run the same pages: does the slow page stay with the slow-user’s login not matter which PC?

Understanding the problem pages:

  • what specific pages are slow for the unlucky users - share the URL with us (but strip off your domain at the start if you wish).
  • have other users copy those exact URLs and see what speed they get
  • maybe for those two different users -those pages have different quantities of records in the lists shown? or is it the same number of rows in each.

From any user account - find out how many calls are assigned to each of those 2 users - as that can slow down databases in Calls Edit view - when it creates the panel below ’ Scheduling’.

Database analysis

  • Have you access to the slow-query logs of you database?
  • can you share here the results of the MySQL command: SHOW TABLE STATUS

Thank you so much for guidance. It will help us in future to find out issue related to speed.

It seems that after doing empty cache and hard reload on browser. Users are able to use website much faster than before.

I’m glad that you had a nice fast fix.