AntonK
17 January 2024 03:57
1
Hello
Iāve noticed that Elastic search reindex does not work after upgrade to 7.14.2 and PHP 8.2.
I know that there is an open bug report about re index not working but Iāve found that even full index in CLI does not work any longer.
Before Iāve wiped all indexed records from Elasticsearch I could search for records successfully, but only for older records, newer records could not be found.
Iām running on Debian latest, Apache 2, PHP 8.2.7 in php-fpm mode
Elasticsearch 7.17.16
When I use robo to run a full index job I get the following
vendor/bin/robo elastic:index 0
[=][02:59:12] Starting indexing procedures
[@][02:59:12] Indexing is performed using JsonSerializerDocumentifier
[@][02:59:12] A full indexing will be performed
[@][02:59:12] Removed index āaccountsā
[@][02:59:12] Created new index āaccountsā
[@][02:59:12] Removed index ācontactsā
[@][02:59:12] Created new index ācontactsā
[@][02:59:12] Removed index āleadsā
[@][02:59:13] Created new index āleadsā
[@][02:59:13] Removed index ānotesā
[@][02:59:13] Created new index ānotesā
[@][02:59:13] Removed index āaos_contractsā
[@][02:59:13] Created new index āaos_contractsā
[@][02:59:13] Removed index āmeetingsā
[@][02:59:13] Created new index āmeetingsā
[@][02:59:13] Removed index āopportunitiesā
[@][02:59:13] Created new index āopportunitiesā
[@][02:59:13] Indexing module Accountsā¦
[@][02:59:14] Finished Accounts. Processed 975 Records
[@][02:59:14] Indexing module Contactsā¦
[@][02:59:14] Finished Contacts. Processed 728 Records
[@][02:59:14] Indexing module Leadsā¦
[@][02:59:14] Finished Leads. Processed 4 Records
[@][02:59:14] Indexing module Notesā¦
[@][02:59:15] Finished Notes. Processed 1000 Records
[@][02:59:15] Indexing module AOS_Contractsā¦
[@][02:59:15] Finished AOS_Contracts. Processed 228 Records
[@][02:59:15] Indexing module Meetingsā¦
[@][02:59:15] Finished Meetings. Processed 37 Records
[@][02:59:15] Indexing module Opportunitiesā¦
[@][02:59:15] Finished Opportunities. Processed 362 Records
[@][02:59:15] No record has been indexed
[=][02:59:15] Indexing complete
And while indexes are created , they are empty
curl ālocalhost:9200/_cat/indices?vā
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases gXCDGK_gR9GG4IgMUtzX3g 1 0 42 4 39.2mb 39.2mb
yellow open aos_contracts ZJkpOEEYSuqHQgRnRVKcyQ 1 1 0 0 227b 227b
yellow open notes aQIzK2IiTL66UTinYuBGqA 1 1 0 0 227b 227b
yellow open leads H_EQgxDcT52lPlgt0omYcg 1 1 0 0 227b 227b
yellow open meetings VrDtnRR0QoKHW-rfj90hKg 1 1 0 0 227b 227b
yellow open accounts T0gdKV8ySpWiAovpEqa3hQ 1 1 0 0 227b 227b
yellow open opportunities d74huQyxSUmZYlaCxIMYUQ 1 1 0 0 227b 227b
yellow open contacts jDcqig1KRWOWweDj_a-YMQ 1 1 0 0 227b 22
I wonder if anyone has any ideas how to deal with this.
Anton
The fix is here, it just hasnāt been added in yet. Thanks go to @jansiero
salesagility:hotfix
ā JanSiero:bugfix_10207
opened 07:57PM - 03 Dec 23 UTC
This bugfix solves multiple issues concerning elasticsearch indexing, specificalā¦ ly https://github.com/salesagility/SuiteCRM/issues/10207 and https://github.com/salesagility/SuiteCRM/issues/10209
## Description
Elastic search indexing by cron has been broken for some time, especially due to some code reworking. This PR clears a series of bugs and issues which should make Elastic Search run more stable and user friendly
## Motivation and Context
### `modules/Schedulers/_AddJobsHere.php`
- The method "runElasticSearchIndexerScheduler" is called from
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/modules/SchedulersJobs/SchedulersJob.php#L525-L529
`call_user_func_array` is called with "$data" which is an array that consists out of two items:
1. $this - the SchedulersJob object
2. $this->data - the parameters with which elastic should run
Consequently the method runElasticSearchIndexerScheduler should contain _two_ arguments, of which only the second ($data) is used with as default value an empty JSON object: "{}"
- As the parameters are passed as a JSON String and a friggin' purifier htmlencodes them on saving, these need to be html-decoded.
- JSON_decode should output as an associative array, hence the "true" argument
### `lib/Search/ElasticSearch/ElasticSearchIndexer.php`
Coalesce with an empty array in case the index was not created to avoid PHP warnings
### `data/SugarBean.php`
With commit https://github.com/salesagility/SuiteCRM/commit/26c92909d0bd58da76cce33799c3887494f5335a the variable "$rows_found" was initialized at the start of the method process_list_query,
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/data/SugarBean.php#L4248-L4250
essentially turning the following:
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/data/SugarBean.php#L4350-L4352 into dead code and pullNextBatch in
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/lib/Search/ElasticSearch/ElasticSearchModuleDataPuller.php#L166-L174
returning null instead of the records that require indexing. Therefore the check has been changed to test `$rows_found == 0`
### `lib/Search/UI/SearchThrowableHandler.php` and `modules/Home/language/en_us.lang.php`
When for some reason for one or more modules no index has been created, then when using global search an unhelpful message "an unknown error has occurred" is displayed. The changes in these files make sure an appropiate error message and solution (perform full indexing) is provided.
### `modules/Administration/Search/ElasticSearch/Controller.php`
On the Admin / ElasticSearch page the last run of an elastic search job is displayed. If you are not located in the UK or PT, but in another time zone, the displayed time and the human readable time difference is off by one or more hours. "check_date_relationships_load" makes sure the times and time formats are according to the time zone location of the user.
### `lib/Search/Index/IndexingSchedulerTrait.php`
On the configuration page of elasticsearch there is checkbox to enable it. Until this is saved, elastisearch is still disabled,which inhibits the Full and Partial indexing (cron will skip it silently). It is counterintuitive that this setting has to be saved first, before being allowed to manually index elasticsearch.
This change will allow for manual (full or partial) indexing even when elastic search is still disabled: When running a manual index the $option array will contain {partial: true} or {partial: false}, else it will be empty.
This also enables an administrator to first check if indexing works correctly before enabling Elasticsearch systemwide.
### `modules/Administration/Search/ElasticSearch/Controller.php` and `modules/Administration/Search/ElasticSearch/scripts.js` and `modules/Administration/language/en_us.lang.php`
It is counterintuitive that a manual indexing will not use the entered host parameters. Pass the elastic host credentials when manually starting a full or partial indexing, save them so they will be used on the next scheduled indexing.
Check the return value for an indexing request and give a useful response
### `modules/Administration/Search/ElasticSearch/view.tpl`
The JavaScript file ajaxSubmit.js was removed in 7.14 and should not be referenced.
### `modules/Schedulers/Scheduler.php` and `modules/Schedulers/language/en_us.lang.php`
Create a Scheduler entry for Elasticsearch. As long as elasticsearch hasn't been enabled this scheduler will not attempt to index anything.
It should be considered to also create this entry when upgrading.
## How To Test This
See #10207 and #10209
## Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
### Final checklist
- [x] My code follows the code style of this project found [here](https://docs.suitecrm.com/community/contributing-code/coding-standards/).
- [ ] My change requires a change to the documentation.
- [x] I have read the [**How to Contribute**](https://docs.suitecrm.com/community/contributing-code/) guidelines.
Please also consider approving the documentation changes:
https://github.com/salesagility/SuiteDocs/pull/630
AntonK
17 January 2024 15:43
4
@rsp
Iāve tried this fix, it stops cron from generating errors, but does not resolve my problem.
I still get No record has been indexed in differential or full indexing.
Is there a way to release a hotfix? I can test it on my system, if you need a tester
Anton
rsp
17 January 2024 15:47
5
Did you try chris001ās solution?
AntonK
17 January 2024 16:11
6
@rsp
No I didnāt . Iāve never used a github so Iām not sure how to pull the diff from it.
The solution youāve linked just had 2 small code changes , so Iāve tried to use it by changing files manually.
I really donāt want to modify by hand all the files listed in solution linked by @chris001 .
If you know of a guide on how to pull changed files from github, let me know so I can test it.
Anton
rsp
17 January 2024 17:47
7
I think you need to update your code from here:
salesagility:hotfix
ā JanSiero:bugfix_10207
opened 07:57PM - 03 Dec 23 UTC
This bugfix solves multiple issues concerning elasticsearch indexing, specificalā¦ ly https://github.com/salesagility/SuiteCRM/issues/10207 and https://github.com/salesagility/SuiteCRM/issues/10209
## Description
Elastic search indexing by cron has been broken for some time, especially due to some code reworking. This PR clears a series of bugs and issues which should make Elastic Search run more stable and user friendly
## Motivation and Context
### `modules/Schedulers/_AddJobsHere.php`
- The method "runElasticSearchIndexerScheduler" is called from
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/modules/SchedulersJobs/SchedulersJob.php#L525-L529
`call_user_func_array` is called with "$data" which is an array that consists out of two items:
1. $this - the SchedulersJob object
2. $this->data - the parameters with which elastic should run
Consequently the method runElasticSearchIndexerScheduler should contain _two_ arguments, of which only the second ($data) is used with as default value an empty JSON object: "{}"
- As the parameters are passed as a JSON String and a friggin' purifier htmlencodes them on saving, these need to be html-decoded.
- JSON_decode should output as an associative array, hence the "true" argument
_Rolled back changes of #10280_
### `lib/Search/ElasticSearch/ElasticSearchIndexer.php`
Coalesce with an empty array in case the index was not created to avoid PHP warnings
### `data/SugarBean.php`
With commit https://github.com/salesagility/SuiteCRM/commit/26c92909d0bd58da76cce33799c3887494f5335a the variable "$rows_found" was initialized at the start of the method process_list_query,
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/data/SugarBean.php#L4248-L4250
essentially turning the following:
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/data/SugarBean.php#L4350-L4352 into dead code and pullNextBatch in
https://github.com/salesagility/SuiteCRM/blob/54bc56c3bd9f1db75408db1c1d7d652c3f5f71e9/lib/Search/ElasticSearch/ElasticSearchModuleDataPuller.php#L166-L174
returning null instead of the records that require indexing. Therefore the check has been changed to test `$rows_found == 0`
### `lib/Search/UI/SearchThrowableHandler.php` and `modules/Home/language/en_us.lang.php`
When for some reason for one or more modules no index has been created, then when using global search an unhelpful message "an unknown error has occurred" is displayed. The changes in these files make sure an appropiate error message and solution (perform full indexing) is provided.
### `modules/Administration/Search/ElasticSearch/Controller.php`
On the Admin / ElasticSearch page the last run of an elastic search job is displayed. If you are not located in the UK or PT, but in another time zone, the displayed time and the human readable time difference is off by one or more hours. "check_date_relationships_load" makes sure the times and time formats are according to the time zone location of the user.
### `lib/Search/Index/IndexingSchedulerTrait.php`
~~On the configuration page of elasticsearch there is checkbox to enable it. Until this is saved, elastisearch is still disabled,which inhibits the Full and Partial indexing (cron will skip it silently). It is counterintuitive that this setting has to be saved first, before being allowed to manually index elasticsearch.~~
~~This change will allow for manual (full or partial) indexing even when elastic search is still disabled: When running a manual index the $option array will contain {partial: true} or {partial: false}, else it will be empty.~~
~~This also enables an administrator to first check if indexing works correctly before enabling Elasticsearch systemwide.~~
_Rolled back changes of #10280_
### `modules/Administration/Search/ElasticSearch/Controller.php` and `modules/Administration/Search/ElasticSearch/scripts.js` and `modules/Administration/language/en_us.lang.php`
~~It is counterintuitive that a manual indexing will not use the entered host parameters. Pass the elastic host credentials when manually starting a full or partial indexing, save them so they will be used on the next scheduled indexing.~~
Check the return value for an indexing request and give a useful response
### `modules/Administration/Search/ElasticSearch/view.tpl`
The JavaScript file ajaxSubmit.js was removed in 7.14 and should not be referenced.
### `modules/Schedulers/Scheduler.php` and `modules/Schedulers/language/en_us.lang.php`
Create a Scheduler entry for Elasticsearch. As long as elasticsearch hasn't been enabled this scheduler will not attempt to index anything.
It should be considered to also create this entry when upgrading.
## How To Test This
See #10207 and #10209
## Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
### Final checklist
- [x] My code follows the code style of this project found [here](https://docs.suitecrm.com/community/contributing-code/coding-standards/).
- [ ] My change requires a change to the documentation.
- [x] I have read the [**How to Contribute**](https://docs.suitecrm.com/community/contributing-code/) guidelines.
Please also consider approving the documentation changes:
https://github.com/salesagility/SuiteDocs/pull/630
If you have dev instance try on it.
To apply and test the Elasticsearch fix from @jansiero :
Go to your Debian linux terminal,
cd
(change directory) to the base directory of your SuiteCRM 7.14.2, for example: cd /home/suite7/public_html
Make sure you have these commands installed, run: apt -y install wget git
Get the fix: wget https://github.com/salesagility/SuiteCRM/pull/10275.diff
Apply the fix: git apply 10275.diff
Go to Admin, Repair, Quick Repair Rebuild.
Go to Admin, Elasticsearch, try everything, see how it works!
Report back your results!
AntonK
19 January 2024 16:16
9
@chris001
Thank you, Iāll try on a weekend
Anton
Hi, because the current code base consists out of different commits, applying the diff wonāt work.
To apply the patch you can go to Admin / Module Loader and install the module that I attach here.
Please note:
It only installs on Suite 7.14.2
Before upgrading please first uninstall this module
@chris001 thank you for having kept this issue on the radar in the forum and on Github!
patch_10275.zip (124.4 KB)
1 Like
rsp
12 September 2024 17:01
11
Did you able to set up the Elasticsearch on your SuiteCRM?