Elastic search questions

💬 General Discussion
1

I have played with elasticsearch today and I am confused (and documentation is inconclusive at best).

Does elastisearch index every module (and all fields within the module) which is selected/approved in search settings? Does this apply to custom modules as well?

Or we need to select the module and manually add fields into the code just like with baked in global search?

p.s. my confusion comes from the fact that I have some custom fields which I manually added to global search, but for some reason of the other I could not find my records in those custom fields (I also guess it does not index documents within suitecrm)
p.p.s. I am also confused on how much time elasticsearch need to build initial index, my install appeared to be finished in seconds which I find a bit too optimistic
p.p.p.s. it would be nice to consider adding fields to search from GUI via studio instead of manually tinkering (possibly feature request for 8.10?)

1 Like
2

Any ideas on my elastic search issue? Or a pointer to somewhere where full functionality of elastic search integration is explained in detail (and which, if any, options are possible)?

Also, it appears that elastic search is potential dead-end, so is there any future in it or should I move back to classic search (and modifying files manually to fix the search issues)

3

Hello,

I think the very first question that we need to answer is:
How many records do you have in your SQL DB?
If it’s anything below 1 million, I wouldn’t even think about adding ElasticSearch into the tech stack.

As you’ve already figured out, there are many open questions and little documentation.
You can check out the sugar side of the docs:
https://support.sugarcrm.com/knowledge_base/installation_upgrade/installing_and_administering_elasticsearch/#Key_Metrics
https://support.sugarcrm.com/documentation/sugar_developer/sugar_developer_guide_14.2/architecture/elastic_search/
It might not apply 100% anymore, but the roots are similar.

If your initial index takes only a few seconds, most likely it’s an indicator that you don’t need ElasticSearch, due to the amount of data you have :wink:
In a recent project, we had around to 700k contacts and plenty of other records in related modules.
It took close to two hours to re-index, if I recall correctly.

Have you checked in Kibana, whether everything looks ok?

As for the documents: I believe you’re right - haven’t seen that as well.

As for the UI:
As you can already see from all the replies to your topic and the many topics around ElasticSearch in the forums here :wink: it seems to be a low priority.
I assume, that most of the SuiteCRM installations on earth contain a few thousand / seldom tens of thousands records, with its (current) B2B and little marketing automation focus.
So therefore, I believe there are many more pressing topics for the UI than the ElasticSearch integration (Campaigns, KanBan, Admin, Themes are just a few ideas).
Which leads to the requirement of a developer to get things integrated.

A dead end? Well - ElasticSearch is a great search engine / index and it keeps on developing like adding Vector capabilities.
Those help little with the standard SuiteCRM search, since this looks into the traditional index.

In several newer projects, AI, semantic search etc. are on the table.
Going forward, there might be two sorts of searches - your traditional SQL query and then a semantic search feature that can build reports, filters etc.

Elastic Search, I’m not sure - maybe before going down that route, I’d look into the business requirements and maybe take a CDP / data lake into consideration as well. Not the same benefits, but maybe more beneficial to a business with that many data records to search through.

1 Like
4

The reason I moved to elasticsearch is because I (naively) belived that elasticsearch will also index my documents as well as custom fields I have defined in my CRM (I really do not like to mess up with files to add fields to the search).

My contact base is just a few thousand contacts so seconds for indexing appears to be ok, if it started indexing document then it would take much more time (it does index filenames).

Interestingly, with elastic I do get more results but I also think that relevance of this results is off (what I am looking for is there, but maybe not on my first result screen).

5

You’re actually hinting at an interesting requirement.

I’ve got a few projects where I’ve integrated NextCloud and other software for the DMS part and the CRM just holds a link to the document.

If one only stored the CRM generated documents inside the CRM, then you’d find the data via the standard search.

But if you want to upload documents to the CRM and search index them, then it would make sense to find them.
I’ve got the impression, that DMS in the CRM (not only Suite - other CRM I’ve worked with as well) just isn’t great UI / UX and therefore people would still to what they have (Sharepoint / G Drive / Dropbox) or move to something more dedicated (Nextcloud, Pimcore, …).

May I ask: What’s the nature of the documents / what (business) type of documents are you uploading?

6

I store a lot of legal documents (contracts, opinions, different kind of memos or settlements). Not all of them are text PDFs, some are scanned.

My current idea is to create N8N workflow which will download each file, do OCR and write summary somewhere in suitecrm (there are other, more immediate plans too).

7

Yes, good idea - I’ve set this up in a project as well.
OCR + AI summaries work very well through n8n.