Scheduler jobs "stuck" on running - queued

👨‍💻 Development Help
1

It’s quite likely I’ve screwed something up, but need some help troubleshooting this one.

I had an issue that required me to restore the installation from a backup a couple of weeks ago, and I’m now realizing that all the ‘schedulers’ have not been running since.

Environment:
Ubuntu Linux 20.04.6/Apache
SuiteCrm V 7.14.2

image
image1101×433 43.6 KB

I’ve used the Rebuild Schedulers tool to go back to ‘factory’ schedulers.
Also, deleted the hung jobs from the job_queue table.
Even still, all the jobs seem to hang at ‘running-queued’.

I’ve reset all File and Folder ownership, and permissions;
The crontab file looks ok to me;
Not seeing any related errors in the suite error log;

Anyone have any thoughts on how to track this down? (@pgr ??)

2

Check if you have any errors in the log files or web server log.

Also, it could be your customize code. You need to check syntax properly.

Also, you may need to restart services like PHP, web server and database.

3

I’m seeing a ton of these in the Apache log…does this seem relevant?
(Smells like the library wasn’t fully restored or something…?? Yikes??)

AH01071: Got error ‘PHP message: PHP Warning: Class “Google_Service” not found in /var/www/html/vendor/google/apiclient-services/autoload.php on line 21; PHP message: PHP Warning: Class “Google_Service_Resource” not found in /var/www/html/vendor/google/apiclient-services/autoload.php on line 21; PHP message: PHP Warning: Class “Google_Model” not found in /var/www/html/vendor/google/apiclient-services/autoload.php on line 21; PHP message: PHP Warning: Class “Google_Collection” not found in /var/www/html/vendor/google/apiclient-services/autoload.php on line 21’,

I also forgot to mention, that the Executing the “Diagnostic tools” feature will also not ‘work’…does that help point to anything?

Is there a method to make sure all the (non-custom) files are in place? (for example, will ‘upgrading’ to the new version re-install all the libraries etc? or perhaps some other method?)

4

There are commands to install or update the composer from your SuiteCRM root directory. You will find it in this forum.

Also, maybe you could install the below and check:

composer require google/apiclient-services

5

Ok, i have the google services problem either solved, or disabled at this point, but still getting all the schedulers to ‘stuck’.

Can you confirm, IF you run the ‘rebuild’ schedulers feature, ‘should’ that remove all your custom schedulers?? My system is behaving that way, which if NOT expected, might point me more certainly toward my custom schedulers.

Thoughts?

Any other tests you can think of that might point me at the issue?

Do any of these apache errors log entries help??

[Sat Mar 01 13:40:09.075260 2025] [proxy_fcgi:error] [pid 722] [client 97.103.252.251:59609] AH01071: Got error ‘PHP message: PHP Warning: Undefined array key “advanced_search” in /var/www/html/include/SearchForm/SearchForm2.php on line 626’, referer: https://zzzzzzzzzzz/index.php?module=Administration&action=index

[Sat Mar 01 13:42:27.266586 2025] [proxy_fcgi:error] [pid 721] [client 97.103.252.251:59789] AH01071: Got error ‘PHP message: PHP Warning: Undefined array key “advanced_search” in /var/www/html/include/SearchForm/SearchForm2.php on line 626’, referer: https://zzzzzzzzzzzzzzzzzzzz/index.php?module=Schedulers&action=index

[Sat Mar 01 13:43:46.349058 2025] [proxy_fcgi:error] [pid 381] [client 97.103.252.251:59841] AH01071: Got error ‘PHP message: PHP Warning: Undefined array key “advanced_search” in /var/www/html/include/SearchForm/SearchForm2.php on line 626’, referer: https://zzzzzzzzzzz/index.php?module=Administration&action=index

[Sat Mar 01 13:44:11.837087 2025] [proxy_fcgi:error] [pid 722] [client 97.103.252.251:59875] AH01071: Got error ‘PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524; PHP message: PHP Warning: Undefined property: Scheduler::$job_url in /var/www/html/data/SugarBean.php on line 2524’, referer: https://zzzzzzzzzzzzzzzzzzzzzz.com/index.php?module=Administration&action=RebuildSchedulers

6

If a workflow is stuck running go to database in “job queue” and delete the “queued” job that’s “running”

This will allow it to start again on the next cron cycle.

7

Yeah…I’ve done that a few times.
Seems something is making them ‘all’ hang.

I’ve rebuilt the schedulers (which seems to delete all the custom scheduled jobs from the Dbase), and disabled all the schedulers except: “Run Report Generation Scheduled Tasks” (to try to isolate any misbehaving ones)…

Each time I reset them, and wait for the Report Gen job to run, it first goes to “queued - queued”, then after a few seconds “running - queued”…but it hangs there forever.

Then I’ll go into Dbase, and delete that running job,…but same keeps happening.

I’m not seeing any errors in the SuiteCrm error log that seem to correlate, and I’ve solved a bunch of ‘other’ errors in the Apache log, and now left with a bunch of ‘Smarty’ deprecation errors…but don’t seem relevant.

I’m fearful that it’s related to some custom scheduler or other custom code I’ve injected, but when I used the ‘rebuild’ schedulers feature, all the custom schedulers I’d bult over time seem to have been deleted (though the code is still there).

Any thoughts on how to track this down?

8

Run cron from ssh and see if you get an error output.

Likely culprits are php memory limit, timeout limit or php modules missing.

9

Hey, thanks for that!!
This is interesting.
I got this in the error log when running cron manually.

Sun Mar 2 18:31:01 2025 [2094][1][FATAL] Job runs too frequently, throttled to protect the system.

I’ve restarted apache, rebooted the server, even rebooted the entire VPS…still seeing the same.

Any thoughts?

10

Check with your hosting provider first.

11
12

Ok, I had to go a little more ‘nuclear’ than I’m comfortable with.

I did what you’re describing, set the ‘min_cron_interval’ to 0.
But I was still getting fatal error from aow_utils.php on line 645.
I commented that line out (for now), and the schedulers are now working again.
(Clearly this is NOT a good permanent solution…)
It said I was declaring $sfh more than once?
(Does that help identify where I might look?)

I’m sure something else is still in bad shape…but I’m struggling to track it down.

Is there someone you trust that I might be able to contract with to help me?

13

I see that in my logs too in the last month but differently to you, all my schedulers all seem to run OK.

That error text is thrown in file: include/SugarQueue/SugarCronJobs.php

And seems to be caused by a lock file date - something on my set up is creating that file wrongly - possibly due to a 3rd-party plugin we added recently?