Menu won't load when opening via reverse proxy (HAProxy)

I have just successfully installed the newest version 8.6.0 and everything is working fine when opening the site directly, from my local network. But when I configure my HAProxy to open the site from outside my network it does not load correctly – the top menu bar is corrupt. See the pictures below.

The only difference I saw while debugging from my browser is a ‘bad gateway’ error (502) on the graphql request. Any clue how can I troubleshoot that?

I have already spent a day with an assistance of ChatGPT, checking tons of things in HAProxy settings, but they seem to be good, besides this is not the only site I am opening via HAProxy and this is the only one that is experiencing such issues.

Thanks in advance for your advice!

Did you set the variable site_url in your config.php to your external URL e.g. https://crm.mysite.tld ?

No, I did not…
Are you referring to this config.php file

/public/legacy/config.php
?

Yes, that in public/legacy/config.php file.
Please set your site_url to your external URL of your HAProxy for your Suite, like this:
'site_url' => 'https://crm.mysite.tld/'
…and try to browse to your public Suite URL, from outside your network.
Report back here, does it work or no.

It was indeed set to site_url=‘http://192.168.131…205’

So, I changed it to the https://suitecrm.net-extend.pl, repaired & rebuilt, restarted Apache. But still the same:

Try to run these commands to reset file and dir permissions.

find . -type d -not -perm 2755 -exec chmod 2755 {} \;
find . -type f -not -perm 0644 -exec chmod 0644 {} \;
find . ! -user www-data -exec chown www-data:www-data {} \;
chmod +x bin/console

Post errors in the error logs with timestamp from when this menu fail happens?

I reset the permissions - still the same.

The problem is I see no errors in the error.log while opening the page. The regular access.log looks the same in both cases:

Bad (via HAProxy):

192.168.131.1 - - [04/May/2024:13:58:47 +0200] "GET / HTTP/1.1" 200 4397 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "POST /api/graphql HTTP/1.1" 200 25470 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "GET /session-status HTTP/1.1" 200 1870 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "POST /api/graphql HTTP/1.1" 200 281518 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "POST /api/graphql HTTP/1.1" 200 4597 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "GET /legacy/index.php?module=Home HTTP/1.1" 200 6635 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:48 +0200] "GET /legacy/themes/suite8/css/colourSelector.php HTTP/1.1" 200 200 "https://suitecrm.net-extend.pl/legacy/index.php?module=Home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:47 +0200] "POST /api/graphql HTTP/1.1" 200 1549153 "https://suitecrm.net-extend.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:48 +0200] "POST /legacy/index.php?entryPoint=retrieve_dash_page HTTP/1.1" 200 10851 "https://suitecrm.net-extend.pl/legacy/index.php?module=Home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.1 - - [04/May/2024:13:58:48 +0200] "GET /legacy/include/javascript/popup_helper.js?v=rXKFK7H27B7_EkN1W9m1jw HTTP/1.1" 200 3620 "https://suitecrm.net-extend.pl/legacy/index.php?module=Home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"

Good (directly):

192.168.131.198 - - [04/May/2024:13:59:37 +0200] "GET / HTTP/1.1" 200 4469 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:37 +0200] "POST /api/graphql HTTP/1.1" 200 25522 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:37 +0200] "GET /session-status HTTP/1.1" 200 1922 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:37 +0200] "POST /api/graphql HTTP/1.1" 200 3036616 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:37 +0200] "POST /api/graphql HTTP/1.1" 200 2230 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:37 +0200] "POST /api/graphql HTTP/1.1" 200 4660 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:38 +0200] "GET /legacy/index.php?module=Home HTTP/1.1" 200 6721 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:38 +0200] "GET /legacy/themes/suite8/css/colourSelector.php HTTP/1.1" 200 255 "https://suitecrm.c11.net.pl/legacy/index.php?module=Home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:38 +0200] "POST /api/graphql HTTP/1.1" 200 1548668 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:38 +0200] "POST /legacy/index.php?entryPoint=retrieve_dash_page HTTP/1.1" 200 11196 "https://suitecrm.c11.net.pl/legacy/index.php?module=Home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"
192.168.131.198 - - [04/May/2024:13:59:59 +0200] "POST /api/graphql HTTP/1.1" 200 2778 "https://suitecrm.c11.net.pl/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0"

And still - that crazy ‘bad gateway’ error (in the browser console):

That 502 Bad Gateway is from HAProxy.
The logs you shared were from suite’s apache. Instead, you need to look at the php error log, and the suite application error log. More info here:

Since there no special error_log definition in my php.ini-- the PHP errors are being logged in the regular Apache’s error.log.

There is nothing there except for the errors I deliberately generated to confirm where the errors are being logged (last two lines):

marek@suitecrm:/var/www/html/suitecrm/public$ tail -f /var/log/apache2/error.log
[Sun May 05 00:00:01.566951 2024] [ssl:warn] [pid 117855] AH01906: suitecrm.net-extend.pl:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Sun May 05 00:00:01.567041 2024] [ssl:warn] [pid 117855] AH01909: suitecrm.net-extend.pl:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 05 00:00:01.567232 2024] [mpm_prefork:notice] [pid 117855] AH00163: Apache/2.4.52 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun May 05 00:00:01.567240 2024] [core:notice] [pid 117855] AH00094: Command line: '/usr/sbin/apache2'
[Sun May 05 18:42:02.420105 2024] [php:warn] [pid 132446] [client 192.168.131.1:13618] PHP Warning:  Undefined variable $undefinedVariable in /var/www/html/suitecrm/public/test.php on line 3
[Sun May 05 18:42:07.310184 2024] [php:warn] [pid 132446] [client 192.168.131.1:13618] PHP Warning:  Undefined variable $undefinedVariable in /var/www/html/suitecrm/public/test.php on line 3

There is not much in the suitecrm.log either. I set the log level to Error in the configuration.
There is no difference between the bad (via HAProxy) and good (direct) environments:

Bad:

Sun May  5 20:12:43 2024 [133476][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:12:43 2024 [133476][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:12:43 2024 [133476][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:12:43 2024 [133476][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:12:43 2024 [133486][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:12:43 2024 [133486][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:12:43 2024 [133486][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:12:43 2024 [133486][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources

Good:

Sun May  5 20:13:07 2024 [133474][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:13:07 2024 [133474][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:13:07 2024 [133474][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:13:07 2024 [133474][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:13:07 2024 [133477][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:13:07 2024 [133477][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources
Sun May  5 20:13:07 2024 [133477][1][ERROR] Unable to find relationship projects_contacts_resources
Sun May  5 20:13:07 2024 [133477][1][ERROR] Unable to find relationship am_projecttemplates_contacts_resources

So, it must be something about HAProxy.
I have plenty of web applications what are being opened through HAproxy (Discourse forum like this one included), and I do not experience such issues :frowning:

hi @mwojtaszek ,
did you solve the problem? I have the same error setting up Ngnix as reverse proxy.
If you solved maybe could help me. Anyway I think I will open a new topic as well here.

Thanks!

Actually, I gave up for now. What worked for me was either using a Cloudflare Zero Trust (aka Cloudflare tunnel) or a port forwarding on a router. I was not able to make the reverse proxy working, unfortunately.

I am having exactly the same problem. Running SuiteCRM on docker using HAProxy with the following backend configuration:

##
backend suitecrm
  mode http
  option forwardfor
  option tcp-check

  balance roundrobin
  server crm-01 docker-server-01:10080 check

The menu does not appear and if we inspect what is going on, a 502 Bad Gateway error is returned when requesting a POST to https://xxxx.xxxx.xxx.xx/api/graphql

We can see that there are some requests to the same graphql uri that do work.
The one that doesn’t work is the one that is close to 3 MB. Could it be that HAProxy is closing this connection too soon?

I think so, you need to find out how to increase size or time for such heavy requests.

  1. What version HAproxy you running?

  2. HAproxy can return this 502 code:

502 when the server returns an empty, invalid or incomplete response, or
when an “http-response deny” rule blocks the response.

  1. Similar issue:
    Haproxy randomly returns 502 Bad Gateway · Issue #2368 · haproxy/haproxy · GitHub

  2. If you believe this 502 is an issue in HAproxy, post full details in a New Issue.

  3. Here is an example question about 502 error on HAproxy, post on the forum with your full details.

I had the same problem. Haproxy sets the maximum number of headers in a request, default is 101. In haproxy.cfg is necessary modify or add tune.http.maxhdr higher than default. (HAProxy Enterprise Documentation version 2.9r1 (1.0.0-328.464) - Configuration Manual)

image

3 Likes

This solved my problem.
Thanks Tomas.

The number of headers received by the POST to https://xxxx.xxxx.xxx.xx/api/graphql was larger than the default maximum in HAProxy. Changed this configuration and voilá… fixed.

2 Likes

@mwojtaszek please click “Soluition” under @tomas_calde_r post if it solves your issue so other users would find this solution faster.