Global Search stopped working

Hi,
I am running Version 7.9.7 in a VirtualBox on Ubuntu 16.04 server / Ubuntu 16.04 server.
Daily backups are a breeze. ( Thank you ‘pgr’!)
Until today. :frowning:
As usual, I stopped the server in VB, backed up the directory, re-started, and boom. The Global Search, which our sales people use to scan the contents of Notes, does not produce anything at all. I have no idea what of what happened.
I am getting some "FATAL’ errors in the log file.
Could someone take a look, please? Thanks!
Here is what I see in the log file, repeating every once in a while:

Wed Nov 28 10:16:00 2018 [830][da418dbc-fcbc-b3fd-207b-5baa761b1004][FATAL] Exception handling in /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/Controller/SugarController.php:402
Wed Nov 28 10:16:00 2018 [830][da418dbc-fcbc-b3fd-207b-5baa761b1004][FATAL] Exception in Controller: File ‘modules/AOD_Index/Index/Index/segments_1842’ is not readable.
Wed Nov 28 10:16:00 2018 [830][da418dbc-fcbc-b3fd-207b-5baa761b1004][FATAL] backtrace:
#0 /var/www/html/InuitGalleryOfVancouverCRM/modules/AOD_Index/Lib/Zend/Search/Lucene/Storage/Directory/Filesystem.php(359): Zend_Search_Lucene_Storage_File_Filesystem->__construct(‘modules/AOD_Ind…’)
#1 /var/www/html/InuitGalleryOfVancouverCRM/modules/AOD_Index/Lib/Zend/Search/Lucene.php(412): Zend_Search_Lucene_Storage_Directory_Filesystem->getFileObject(‘segments_1842’)
#2 /var/www/html/InuitGalleryOfVancouverCRM/modules/AOD_Index/Lib/Zend/Search/Lucene.php(570): Zend_Search_Lucene->_readSegmentsFile()
#3 /var/www/html/InuitGalleryOfVancouverCRM/modules/AOD_Index/AOD_Index.php(342): Zend_Search_Lucene->__construct(‘modules/AOD_Ind…’)
#4 /var/www/html/InuitGalleryOfVancouverCRM/modules/AOD_Index/AOD_Index.php(74): AOD_Index->getLuceneIndex()
#5 /var/www/html/InuitGalleryOfVancouverCRM/modules/Home/UnifiedSearch.php(225): AOD_Index->find(‘abby quartin’)
#6 /var/www/html/InuitGalleryOfVancouverCRM/modules/Home/UnifiedSearch.php(83): doSearch(Object(AOD_Index), ‘abby quartin’, 0, 20)
#7 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/View/SugarView.php(791): include_once(’/var/www/html/I…’)
#8 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/View/views/view.classic.php(74): SugarView->includeClassicFile(‘modules/Home/Un…’)
#9 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/View/SugarView.php(199): ViewClassic->display()
#10 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/Controller/SugarController.php(432): SugarView->process()
#11 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/Controller/SugarController.php(375): SugarController->processView()
#12 /var/www/html/InuitGalleryOfVancouverCRM/include/MVC/SugarApplication.php(105): SugarController->execute()
#13 /var/www/html/InuitGalleryOfVancouverCRM/index.php(52): SugarApplication->execute()
#14 {main}
Wed Nov 28 10:16:04 2018 [3250][1][FATAL] Job ea71e46e-d9d0-7269-4491-5bfedbcb6686 (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:17:02 2018 [3265][1][FATAL] Job 2e0d303e-7205-f109-6dbd-5bfedb2e6659 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:18:05 2018 [3282][1][FATAL] Job 21636fba-924d-a5f6-6b18-5bfedb98497d (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:19:01 2018 [3292][1][FATAL] Job 4f33e81c-a744-407e-0320-5bfedb0763e3 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:20:05 2018 [3302][1][FATAL] Job 74aa172d-bd8a-658a-d65b-5bfedc04d254 (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:21:01 2018 [3312][1][FATAL] Job a9fafa8c-7683-cf2a-a81a-5bfedc45fcc5 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:22:05 2018 [3321][1][FATAL] Job 7fd939b5-c612-5767-b03d-5bfedc922ddb (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:23:01 2018 [3333][1][FATAL] Job b47d4ec5-0ab2-c893-7a16-5bfedccbb1cd (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:24:05 2018 [3344][1][FATAL] Job 86650920-fb3c-fa7a-192b-5bfedd7df382 (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:25:01 2018 [3357][1][FATAL] Job b5efb097-fff4-b5f2-ade4-5bfedd38c9fd (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:26:05 2018 [3367][1][FATAL] Job 8a47f96d-ca83-e3de-fd78-5bfedd54f65f (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:27:01 2018 [3377][1][FATAL] Job bb167a65-5e36-c166-a93e-5bfeddf2d3b2 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:28:05 2018 [3388][1][FATAL] Job 9e4a8740-c8a0-4a4e-6ffc-5bfedda16365 (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:29:01 2018 [3398][1][FATAL] Job d8473c0d-34bf-2376-3124-5bfedeebbfc3 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:30:05 2018 [3408][1][FATAL] Job c888abe7-7f6b-5944-1d07-5bfede4cf8bc (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:31:01 2018 [3421][1][FATAL] Job a50f0e55-032d-9ac5-2041-5bfedea9c158 (Optimise AOD Index) failed in CRON run
Wed Nov 28 10:32:05 2018 [3432][1][FATAL] Job d549082a-cbad-51f2-cf59-5bfede09a7ff (Perform Lucene Index) failed in CRON run
Wed Nov 28 10:33:01 2018 [3443][1][FATAL] Job 1c1dc103-c2a5-0e83-d51c-5bfedf298871 (Optimise AOD Index) failed in CRON run

Your backup probably caught a search Reindexing operation in the middle of running.

A safer backup process would involve shutting down both the web server and the cron jobs during the backup.

To fix this you can probably just reindex, lI’m hoping you don’t have huge amounts of data, or it could be a while to complete.

Everything you need to know:

https://pgorod.github.io/Reindex-AOD/

1 Like

Hi pgr,
Thank you!
Will try it in a day or two ( just running off my feet with other crap at the moment ). Sales folks are OK to search Notes directly until then.
Just wanted to make sure you know that I really appreciate your advice and suggestions.
Thanks again,
Greg

1 Like