7.11.10 to 7.11.12 upgrade wizard immediate 500 Error

[dsoden]

very new to suitecrm and this is my first upgrade gone wrong. I’m hoping to learn this system then support customers I find, but am now questioning how stable this app is. I had hell with the initial install too also tossing 500 errors which i tracked down and solved realizing it was me not properly configuring or ignoring the MAIL SMTP setup on installation - sadly there is no error handling in the app to prevent ignorant admin user mistakes, on installs just allows a 500 crash… BACK TO THE CURRENT UPGRADE problem…

Upgrading tosses out yet another non helpful 500 error - when I configure PHP to show errors live on the site… I am getting this…

Parse error: syntax error, unexpected end of file, expecting function (T_FUNCTION) or const (T_CONST) in /home/crmdf/public_html/include/database/DBManager.php on line 4075

Fatal error: Class 'DBManager' not found in /home/crmdf/public_html/include/database/MysqlManager.php on line 100

upgradeWizard.log is clean - it never finished, or it actually did but I could not tell - all I know is it was running the wizard looking good, then BAM! 500 error and sniffing the code is not helpful as I am not sure how the app works in the core code, and as a user admin I really do not care to. the DBManager file and the ABSTRACT CLASS is present in that file… so the class is there in code and file name, but I can not find any include statement to link the inheriting subclass, which could be the problem, but I fail to see how this passed QA to be released so I’m not believing it and accept my ignorance. something else must be causing this all.

Other than the what i can see at face value, and being a noob to this system, I have no clue what to do, or how to fix this. Any seasoned SuiteCRM SME out there? I am willing to learn and am hoping this app is more stable than my experience, otherwise supporting customers on it is going to be a nightmare till I can get my head around this and skill up.

Operating system** CentOS Linux 7.7.1908
Webmin version**|1.941|
Virtualmin version**|6.08|
Kernel and CPU**|Linux 3.10.0-1062.12.1.el7.x86_64 on x86_64|
Processor information**|Intel® Xeon® CPU E5-2673 v4 @ 2.30GHz, 2 cores|
PHP** 7.2.24
Apache version** 2.4.6

[pgr]

Check the Release notes, there are a few things there about Upgrade problems:

Debugging steps for use in upgrade version prior to 7.11.11 and 7.10.23

Known issues with the upgrade process fixed as part of this release will unfortunately not take effect until the next upgrade cycle. Therefore it is vital that if you encounter any issue while installing that you review and follow the recommended process as noted below: 1: The screen becomes blank and stops loading. Symptoms Only visible are the background colours for the header and left-sidebar. The page has stopped loading, and the refresh icon is stationary. The loading centre box has d…

Also, between every upgrade attempt make sure you completely remove the cache/upgrades/temp directory.

Tell me if this works, if not, there are other settings we can adjust in your php.ini

[dsoden]

in the release notes 7.11.12 I saw no mention of this, however I do see issues in 7.11.11. I guess the lesson here is to read ALL release notes in between versions one is upgrading. NOTED…

That said the x.x.11 release notes mention or upgrade issue reference herein, has nothing to do with my specific 500 error and specifically the php coding errors (missing class) being thrown.

I deleted the cashed upgrade temp as suggested and refreshed - still the same 500 error. I also restarted Apache. Is the site fully shot? Is my only recourse an entire system restore from backup? If hope not. If the developers know of upgrade issues - im seeing a repeating pattern - no error handling code to prevent known issues. This appears to now be the 2nd consecutive patch release with MAJOR upgrade issues for a 3rd dot (x.x.X) version number number fix .10 to .11 - supposedly reserved for patches and fixes. This in it self is ironic because I broke and downed my entire system over a proactive application of a patch. Disheartening to say the least, and is a admin user confidence killer in the quality of the releases and application as a whole. I really want to LIKE SuiteCRM but I’m getting anxiety using this is the real world.

How can a php.ini tweak help solve this coding issue? I am eager to know. Thank you very much for your assistance.

[pgr]

Sorry about your troubles. Part of our own sorrow is that some of the upgrade issues can only be fixed in two installations (one puts a fixed upgrade code in place, the next one runs it)

Anyway - the basic problem is an ineffective directory iteration routine which got more problematic as our packages grew. The php.ini settings relevant here are simply the resources:

max_execution_time
memory_limit

Now, it is possible that the first botched upgrade you had really broke the app - maybe it simply crashed in the middle of copying a file. So perhaps I would try replacing that file with its integral contents and see if you can get your server back up.

If your server is still on 7.11.10 then this would be it:

–> https://raw.githubusercontent.com/salesagility/SuiteCRM/v7.11.10/include/database/DBManager.php

Back up your file first, then replace it in its entirety with that one, and see if that fixes the FATAL. Let me know how it goes.

[pgr]

Always keep this advice in mind, please:

@pgr

Also, between every upgrade attempt make sure you completely remove the cache/upgrades/temp directory.

And another clean up that might help is removing upload/upgrade/patch

[dsoden]

Ok the take away here is my site is trashed and needs to be restored from backup. This was a SCRATCH site that I used to lean and play with - I had 2 contacts and was playing with the API capabilities. If this were a client - this would be a major problem for me. Glad I let the system sit for 30 days so I can test all aspects.

My takeaway from this is to use a dedicated VM instance and snap the disk just before upgrade. This will make DR restores a breeze. I can also restore a snap to a new VM instance and test the upgrade more easily. Sorry to say I now am feeling burned by the upgrade process - and have to mitigate potential whole disasters like this. Half the battle is knowing.

MAY I BEG for you guys to ADD a maintenance mode option to prevent non admins from changing data while I down a system to snap it? Prevents my phone from ringing. For now, I’ll redirect all requests to a maintenance site. Works but a PITA.

Thanks for all your help and super fast responses. =)

[pgr]

@dsoden

use a dedicated VM instance and snap the disk just before upgrade

highly recommended!

Why you should be running SuiteCRM in a VM

Maintenance mode was requested before, it’s not in the Product Team cards for the near future, but I am hoping somebody from the Community might take up the task

[dsoden]

so this was eating me… and I wanted to know so I restored my entire server from a snap. Really did not want to as it put may other things at risk of something else went wrong.

anyway its back and I set all the settings in the php.ini as per the documentation
https://docs.suitecrm.com/admin/installation-guide/upgrading/

I did experience several issues on multiple attempts and snap restores.
1st attempt I forgot to clear the upgrade cache and 500 error again from the webserver side not php. had to restore a snap again.

2nd attempt I had the issues from the previous post you provided above about upgrade issues from x.x.11 where the middle screen went blank. now the problem is it kicked me to the login screen and now it seams the only account that was in the system my admin account is GONE it refuses to let me login. I’m getting seriously super frustrated… this IMO/IME is just ridiculous and I feel like tossing my wireless mouse at the wall like a pitcher tosses a ball in the world series.

(Edit: after the failed login attempt I missed the text that showed up after the failed attempt:

"You have been logged out because your session has expired.

I checked the DB and the user appears to be there and is not GONE so bad use of words, apologies - but the symptoms acts as if the account is missing or invalid. Clearly the upgrade failed doing something and does not allow the sign in method to properly work anymore

the upgrade wizard log shows it clearly choked backing up

Thu, 05 Mar 2020 11:12:09 -0500 [UpgradeWizard] - Backing up file: /home/crmdf/public_html/vendor/google/apiclient-services/src/Google/Service/QPXExpress/PricingInfo.php

Thu, 05 Mar 2020 11:12:09 -0500 [UpgradeWizard] - Backing up file: /home/crmdf/public_html/vendor/google/apiclient-services/src/Goo

)
What are people suppose to do with real production systems? I am grateful this is just a lab experiment but seriously though…

[pgr]

Again , sorry for your troubles… I don’t see other people having this magnitude of a mess that you’re getting - there might be something else lurking in the background that we haven’t nailed yet.

If you’re starting from scratch, an Install instead of an Upgrade would solve a lot of stuff.

If you’re technical enough for this, it might help to read the issue and the proposed fix:

Interrupted / Hanging Upgrades

Fix #8261, Fix #7242 - Handling of temp files during Upgrades

If you want to try the fix yourself, you would need to replace the files inside the Zip upgrade package with the fixed versions of them. Sorry, but there’s no simpler way to do this…

[dsoden]

Ok disregard the majority of my previous post - I found a server side problem related to disk quotas being exceeded which caused my login issue. the server side 500 error which I encountered first is REAL

Server logs reads:

[Thu Mar 05 12:00:07.595905 2020] [core:error] [pid 57727] [client 40.138.171.140:61796] End of script output before headers: index.php, referer: http://crm.decisionforward.com/index.php?module=Administration&action=index

upgrade wizard log reads:

Thu, 05 Mar 2020 11:12:09 -0500 [UpgradeWizard] - Backing up file: /home/crmdf/public_html/vendor/google/apiclient-services/src/GooThu, 05 Mar 2020 11:59:31 -0500 [UpgradeWizard] - setting session variables...

Thu, 05 Mar 2020 11:59:31 -0500 [UpgradeWizard] - [At commit.php]

Thu, 05 Mar 2020 11:59:31 -0500 [UpgradeWizard] - Setting error_reporting() to E_ERROR while running upgrade

Thu, 05 Mar 2020 11:59:31 -0500 [UpgradeWizard] - backing up files to be overwritten...

[pgr]

Do you have the PHP logs enabled in php.ini (check key error_log)?

If so, please check to see if there are any resource-exhaustion FATALs there. These would be consistent with the random interrupted executions (even if it’s the bug’s fault that it is taking too long). At least, seeing the messages there would be a diagnosis.

Other things that might be interfering: SELinux restrictions; ownership/permissions issues, .htaccess issues.

(I am just spitting out ideas so that, in case one doesn’t help, you can try the other one)

[dsoden]

was showing php errors in real time - let me reset the server due to too many attempts on this server one last time to clean up the mess and enable the logging. I’ll get back to you.

[dsoden]

[05-Mar-2020 12:28:32 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:28:36 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:29:32 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:29:41 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:30:50 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:30:50 America/New_York] PHP Notice:  Undefined index: sqlSkippedQueries in /home/crmdf/public_html/modules/UpgradeWizard/commit.php on line 85
[05-Mar-2020 12:31:22 America/New_York] PHP Notice:  Undefined variable: suitecrm_version in /home/crmdf/public_html/cache/upgrades/temp/Kwa312/scripts/pre_install.php on line 88

resulted in a internal server error page being displayed. closing browser and returning to the admin upgrade wizard to run again - this additional info was logged.

[05-Mar-2020 12:32:55 America/New_York] PHP Notice:  Undefined index: additional_step in /home/crmdf/public_html/modules/UpgradeWizard/index.php on line 291
[05-Mar-2020 12:32:55 America/New_York] PHP Notice:  Undefined index: schema_change in /home/crmdf/public_html/modules/UpgradeWizard/commit.php on line 78
[05-Mar-2020 12:32:55 America/New_York] PHP Notice:  Undefined index: sqlSkippedQueries in /home/crmdf/public_html/modules/UpgradeWizard/commit.php on line 85
[05-Mar-2020 12:33:21 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:21 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:21 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:21 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:31 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:31 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:31 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:31 America/New_York] PHP Warning:  copy(): The second argument to copy() function cannot be a directory in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 232
[05-Mar-2020 12:33:36 America/New_York] PHP Notice:  Undefined index: post_install in /home/crmdf/public_html/modules/UpgradeWizard/uw_utils.php on line 3113
[05-Mar-2020 12:33:36 America/New_York] PHP Notice:  Undefined variable: path in /home/crmdf/public_html/cache/upgrades/temp/Kwa312/scripts/post_install.php on line 210
[05-Mar-2020 12:33:36 America/New_York] PHP Notice:  Undefined variable: path in /home/crmdf/public_html/cache/upgrades/temp/Kwa312/scripts/post_install.php on line 215
[05-Mar-2020 12:33:40 America/New_York] PHP Warning:  dir(/home/crmdf/public_html/cache/upgrades/temp/Kwa312/scripts/files_to_add_post/themes/): failed to open dir: No such file or directory in /home/crmdf/public_html/include/dir_inc.php on line 57

[dsoden]

I just happened to check the ABOUT in the admin area after revisiting the upgrade wizard as mentioned above and it shows:

Version 7.11.12
Sugar Version 6.5.25 (Build 344)

curious I went to the upgrade wizard again and it has an error now showing…

Database failure. Please refer to suitecrm.log for details.

I checked the indicated log and the tail, it shows:

#13 /home/crmdf/public_html/index.php(52): SugarApplication->execute()
#14 {main}
Thu Mar  5 14:57:42 2020 [125440][1][FATAL] Mysqli_query failed.
Thu Mar  5 14:57:42 2020 [125440][1][FATAL] Error inserting into table: upgrade_history: Query Failed: INSERT INTO upgrade_history (id,filename,md5sum,type,status,version,manifest,date_entered,enabled)
					VALUES ('5552d706-adff-e7d5-f0e0-5e6159ebc601','upload://upgrades/patch/SuiteCRM-Upgrade-7.11.x-to-7.11.12.zip','617e2a1f01863ace3c73440e78cd2520','patch','installed','7.11.12','','2020-03-05 19:57:42',1): MySQL error 1062: Duplicate entry '617e2a1f01863ace3c73440e78cd2520' for key 'upgrade_history_md5_uk'
Thu Mar  5 14:57:42 2020 [125440][1][FATAL] Exception handling in /home/crmdf/public_html/include/MVC/Controller/SugarController.php:400
Thu Mar  5 14:57:42 2020 [125440][1][FATAL] Exception in Controller: Database failure. Please refer to suitecrm.log for details.
Thu Mar  5 14:57:42 2020 [125440][1][FATAL] backtrace:
#0 /home/crmdf/public_html/include/database/DBManager.php(353): sugar_die('Database failur...')
#1 /home/crmdf/public_html/include/database/DBManager.php(328): DBManager->registerError('Error inserting...', 'Error inserting...', true)
#2 /home/crmdf/public_html/include/database/MysqliManager.php(179): DBManager->checkError('Error inserting...', true)
#3 /home/crmdf/public_html/include/database/DBManager.php(519): MysqliManager->query('INSERT INTO upg...', true, 'Error inserting...')
#4 /home/crmdf/public_html/data/SugarBean.php(2411): DBManager->insert(Object(UpgradeHistory))
#5 /home/crmdf/public_html/modules/UpgradeWizard/commit.php(425): SugarBean->save()
#6 /home/crmdf/public_html/modules/UpgradeWizard/index.php(292): require('/home/crmdf/pub...')
#7 /home/crmdf/public_html/include/MVC/View/SugarView.php(834): include_once('/home/crmdf/pub...')
#8 /home/crmdf/public_html/include/MVC/View/views/view.classic.php(72): SugarView->includeClassicFile('cache/jsLanguag...')
#9 /home/crmdf/public_html/include/MVC/View/SugarView.php(226): ViewClassic->display()
#10 /home/crmdf/public_html/include/MVC/Controller/SugarController.php(435): SugarView->process()
#11 /home/crmdf/public_html/include/MVC/Controller/SugarController.php(375): SugarController->processView()
#12 /home/crmdf/public_html/include/MVC/SugarApplication.php(113): SugarController->execute()
#13 /home/crmdf/public_html/index.php(52): SugarApplication->execute()
#14 {main}

I ran Select * from upgrade_table and I have this 1 row:

"3623c7f7-528b-3bc4-c56c-5e6137314f06","upload://upgrades/patch/SuiteCRM-Upgrade-7.11.x-to-7.11.12.zip","617e2a1f01863ace3c73440e78cd2520","patch","installed","7.11.10","","","","","2020-03-05 17:32:55","1"

so I deleted that row and revisited the upgrade wizard yet again and not it wants to pick up on the step commit upgrade so I click next.

and yet another server internal error is returned.

no real error in the php.log, or upgradewizard.log or suitecrm.log

based on what I am seeing upgrade is impossible from 7.11.10 to 7.11.12.
I’ve given up. Will fresh install 7.11.12 and wait for the next upgrade experience before recommending this to clients. Another upgrade experience like this and the system is a no go for my practice.

[pgr]

My interpretation of this information is that your upgrade has successfully completed. However the upgrade wizard is still convinced you have to do one more step. Only it’s the last step, which is just a briefing of what happened, doesn’t affect anything.

If you clear cache/upgrades/temp directory, and upload/upgrade/patch, you will clear your Upgrade Wizard and have a working 7.11.12.

But I am not satisfied with the health of your system, it seems to breakdown in the middle of every operation, which is really weird.

May I ask what you have in memory_limit and max_execution_time under Admin / Diagnostics / phpinfo? Make sure you get the values exactly from there so we can be certain of the effective values. Thanks.

[dsoden]

512M
6000

I’ve given up and deleted this. 17.11.12 also fails to install fresh and new s well, and I documented that all here:

Post