AOD - First time lucene index runs for 18 hours and counting

English Language - Forum Advanced OpenSales, Workflow, Portal support
1

I have a SuiteCRM instance with 20000 records in two modules. Yesterday I started cron.php on the server. Lucene indexing hasn’t stopped since.

It is constantly shuffling data in crm/modules/AOD_Index/Index/Index/, which probably means it’s not stalled or anything. I have reniced the process to 19, so it wouldn’t slow down rest of the hosted sites.

But seriously, what is it actually doing and is it supposed to run this long?

2

40 hours.

3

It basically just recreates the index file over and over again, at first I thought that it was doing some btree shuffling or whatever, but every time a new .cfs file is created … it runs the index again. I am now at _gxc.cfs. Which from base36 is 21936 repetitions of the same index.

Basically, it’s a bug

4

Now, one solution would be, to monitor the cron job and kill it after a few minutes, but that might mess up other scheduled services. And then a sql call would have to be made to modify the job log, so as to allow the index to run next time.

5

I love how I actually have to browse the source to find this

index.php?module=AOD_Index

very interesting site. No other way of getting there I think…

Total records 12404
Indexed records 12410
Unindexed records -6
Failed records 0
Index file count 7

Unindexed records -6… yep… seems reasonable… I wonder how that number got there…

{$revisionCount-$indexedCount}

fine… what are those vars

       
$moduleCounts = array();
        foreach($beanList as $beanModule => $beanName){
            $bean = BeanFactory::getBean($beanModule);
            if(!$bean || !method_exists($bean,"getTableName") || !$bean->getTableName()){
                continue;
            }
            $query = "SELECT COUNT(DISTINCT b.id) FROM ".$bean->getTableName()." b WHERE b.deleted = 0";
            $moduleCounts[$beanModule] = $db->getOne($query);
        }
        $revisionCount = array_sum($moduleCounts);
        $indexedCount = $db->getOne("SELECT COUNT(*) FROM aod_indexevent WHERE deleted = 0 AND success = 1");

Apart from the fact that $moduleCounts sounds dirty… It looks fine…

Join me for the next instalment of hunt for the lucene index bug, or some other awful name. Anyway I am getting quite vexed with AOD…

6 
crmdev/custom/modules/Schedulers/Ext/ScheduledTasks/scheduledtasks.ext.php:$job_strings[] = 'aodIndexUnindexed';
crmdev/custom/modules/Schedulers/Ext/ScheduledTasks/scheduledtasks.ext.php:function aodIndexUnindexed()
crmdev/custom/Extension/modules/Schedulers/Ext/ScheduledTasks/AODScheduler.php:$job_strings[] = 'aodIndexUnindexed';
crmdev/custom/Extension/modules/Schedulers/Ext/ScheduledTasks/AODScheduler.php:function aodIndexUnindexed()

Why can’t this be somewhere logical…

7

Hello beautiful.


function aodIndexUnindexed(){
    $total = 1;
    while($total > 0){
        $total = performLuceneIndexing();
    }
    return true;
}

It’s the while that gets you every time…

8 
$query = "SELECT b.id FROM ".$bean->getTableName()." b LEFT JOIN aod_indexevent ie ON (ie.record_id = b.id AND ie.record_module = '".$beanModule."') WHERE b.deleted = 0 AND (ie.id IS NULL OR ie.date_modified < b.date_modified)";

Oh joy, this is going to be some bs somewhere isn’t it…

9 
SELECT DISTINCT(`record_module`) FROM `aod_indexevent`;
SELECT b.id,b.* FROM leads b LEFT JOIN aod_indexevent ie ON (ie.record_id = b.id AND ie.record_module = 'Leads') WHERE b.deleted = 0 AND (ie.id IS NULL OR ie.date_modified < b.date_modified);

date_modified: 2017-05-06 13:06:00

DING DING DING, we have a candidate! Will he get a cigar or will it be latrine duty. Find out in our next installment…

10

Maybe, certainly contributed to the problem.

♫ AND THE SEARCH GOES ON AND ON AND ON ♫

11

Fine, let’s reset the indexing process and do a fresh index. Oh boy, this is taking longer than expected. Well renice it is and I am gonna get some grub [hehe].