AOD - Advanced Open Discovery search and permission issue

erikvanberkum · 11 April 2014 05:18

When I search in via AOD, I get 18 times the same search record as a result, same time displayed I have the entry only one time in the crm database but the AOD index seems to duplicate them.

I went to the /modules/AOD_Index/Index/Index directory and see that the user permissions are root in stead of www-data. I tried to change the the permission but after a little while they show up as root again.

I think i made an error with running the sugar cron job as root in stead of www-data user, might this be the cause of the problem, i am now running the cron as www-data. Some insight is appreciated.

Should i rebuilt? as mentioned in post #4094?

To rebuild the Index, delete the Index directory in /modules/AOD_Index and clear the records in aod_index and aod_indexevent. Then your scheduled jobs will rebuild the Indexes (Indices?)

Have a great day,

Erik

Jim · 14 April 2014 09:34

Hi Erik,

I’d reccommend rebuilding. Those instructions are correct.

18 duplicate results wouldn’t be caused by running cron as root however so this may be a bug in AOD. Is it possible some of the results are deleted records or do all links go through to the same record?

Thanks,
Jim

klou · 14 April 2014 20:03

I’ve definitely found that the permissions of the Index matter.

Originally, I set up my cron jobs via crontab -e (as root), but the created index files then had root:root permissions. Editing /etc/crontab and specifying the www-data user created the correct permissions (with the rebuilt indexes).

erikvanberkum · 15 April 2014 07:52

Jim,

this is a new data set which has been put in manually by hand and is still small in size, around 600 leads or so and contacts and accounts are still under 100. So never deleted the any records. All records refer to the same when clicked upon.

At present my search is not working I am getting blank screens often,

i have deleted the Index directory in /modules/AOD_Index

but didn’t clear the records in the database yet, which i will soon do.

clear the records in aod_index and aod_indexeven is just a matter of going to phpmyadmin, select the records and press empty I assume?

Jim · 15 April 2014 07:58

Hi Erik,

Yeah emptying using phpmyadmin is file. Regarding the blank screens, was there any PHP errors or warnings when this was happening?

Thanks,
Jim

erikvanberkum · 15 April 2014 08:51

Jim,

Didn’t see any php errors but do I have to turn this on?

I emptied the aod_index and aod_indexevent in the database and deleted /modules/AOD_Index/index/index directory.

Having the cron running now with www-data permission we should be good now will keep you posted

erikvanberkum · 17 April 2014 03:13

Jim

I am all good now after resetting everything and permission correction search works normal and gives good results.

So now I run into the next issue which is that we have a database with Korean, Japanese and Chinese characters.
but searching in Chinese character doesn’t give any results back. While the standard search returns some results.

Can you give me some guidance how to trouble shoot or investigate this matter.

Thanks

Erik

Jim · 17 April 2014 08:20

Hi Erik,

This is the first release of AOD and, as such, hasn’t been tested with exotic characters. This is something we want to support however and will be a goal for us for future versions. That being said there are some changes you could make to try and add support.

In

modules/AOD_Index/AOD_Index.php

at the start of the getDocumentForBean method adding

Zend_Search_Lucene_Analysis_Analyzer::setDefault(new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num_CaseInsensitive());

may improve the results. Unfortunately this will mean rebuilding the index

Thanks,
Jim

klou · 8 May 2014 23:56

Haven’t checked into the Index as much as my users are just NOW getting into it, but now I’m getting the same duplication (times 6000) of entries.

I haven’t done the math to figure out how many times the Indexing event has run, but still something to keep an eye on.

erikvanberkum · 29 May 2014 04:50

Jim I am not complete clear were i should add this can you be more specific?

see below my code:

Zend_Search_Lucene_Analysis_Analyzer::setDefault(new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num_CaseInsensitive());
public function getDocumentForBean(SugarBean $bean){
if($bean->module_name == ‘DocumentRevisions’){
$document = $this->getDocumentForRevision($bean);
}else{
$document = array(“error”=>false,“document”=>new Zend_Search_Lucene_Document());
}
if($document[“error”]){
return $document;
}

erikvanberkum · 29 May 2014 06:26

I ignored this issue a while but today i have investigated the matter again. I have many duplicates in my search results so i decided to rebuilt the index (delete directory/empty tables of database) and check again. After one hour i ended up with one lead that had been duplicated already 369 times. This particular lead had a typo in the name which i corrected and editing a field seems to cause the duplication. All other leads don’t have duplication errors.

Checking the MySQL database and do a search got me only back one result so the problem is not in the MySQL database.

Next thing I did edit a record, don’t change anything, save it and 10 minutes later, 440 duplicates of that specific lead while unedited leads stay cool at one search result.

The problem is probably my gut feeling in the /modules/AOD_Index/Index/Index files. Permission of the folder is 755 and the files itself have the 666 permissions. Any idea how to further troubleshoot this issue?

Jim · 29 May 2014 13:00

Hi Erik.

The line should be placed inside the method:

public function getDocumentForBean(SugarBean $bean){
    Zend_Search_Lucene_Analysis_Analyzer::setDefault(new Zend_Search_Lucene_Analysis_Analyzer_Common_Utf8Num_CaseInsensitive());

Regarding the duplicates this is a bug.

Indexed items are supposed to be removed from the index when they are updated, this isn’t happening correctly in the current version which causes the duplicates.

This will be fixed in a future version but in the meantime you can replace the remove method in AOD_Index.php with the below:

    public function remove($module, $beanId){
        $term  = new Zend_Search_Lucene_Index_Term($module.' '.$beanId, 'aod_id');
        $query = new Zend_Search_Lucene_Search_Query_Term($term);
        $hits = $this->getLuceneIndex()->find($query);
        foreach ($hits as $hit) {
            $this->getLuceneIndex()->delete($hit->id);
        }
    }

And replace the line:

$document["document"]->addField(Zend_Search_Lucene_Field::UnIndexed("aod_id", $bean->module_name." ".$bean->id));

in the getDocumentForBean method with:

$document["document"]->addField(Zend_Search_Lucene_Field::Keyword("aod_id", $bean->module_name." ".$bean->id));

Hope this helps,
Jim

erikvanberkum · 4 June 2014 09:19

Jim

I am happy to report that my search results a good now with the updates given. I have still some Apache errors coming up but not double entries anymore. Thanks for the good work. Search with Chinese/Korean/Japanese characters i was unable to get to work, Apache is complaining about illegal characters.

any updates i am happy to test more info required let me know,

Erik