Elasticsearch throw "type is missing" and no index is populated

isaacmarco · 5 January 2022 18:02

My suitecrm is 7.12.2 . This is the first time I try to enable elasticsearch and I get this error when I try a full index. Where should I look to solve the “type is missing” reported?

Please help…

[=][18:53:41] Starting indexing procedures
[@][18:53:41] Indexing is performed using JsonSerializerDocumentifier
[@][18:53:41] A full indexing will be performed
[@][18:53:41] Removed index 'leads'
[@][18:53:42] Created new index 'leads'
[@][18:53:42] Indexing module Leads...
[!][18:53:42] Failed to index module Leads! Exception details follow
[!][18:53:42] Elasticsearch\Common\Exceptions\BadRequest400Exception: {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: type is missing;"},"status":400} in /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:693
  ~  ~  ~  ~  Stack trace:
  ~  ~  ~  ~  #0 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(333): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)
  ~  ~  ~  ~  #1 /var/www/sugarcrm/vendor/react/promise/src/FulfilledPromise.php(28): Elasticsearch\Connections\Connection->Elasticsearch\Connections\{closure}(Array)
  ~  ~  ~  ~  #2 /var/www/sugarcrm/vendor/ezimuel/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
  ~  ~  ~  ~  #3 /var/www/sugarcrm/vendor/ezimuel/ringphp/src/Core.php(341): GuzzleHttp\Ring\Future\CompletedFutureValue->then(Object(Closure), NULL, NULL)
  ~  ~  ~  ~  #4 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(345): GuzzleHttp\Ring\Core::proxy(Object(GuzzleHttp\Ring\Future\CompletedFutureArray), Object(Closure))
  ~  ~  ~  ~  #5 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(241): Elasticsearch\Connections\Connection->Elasticsearch\Connections\{closure}(Array, Object(Elasticsearch\Connections\Connection), Object(Elasticsearch\Transport), Array)
  ~  ~  ~  ~  #6 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(110): Elasticsearch\Connections\Connection->performRequest('POST', '/_bulk', Array, '{"index":{"_ind...', Array, Object(Elasticsearch\Transport))
  ~  ~  ~  ~  #7 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php(1946): Elasticsearch\Transport->performRequest('POST', '/_bulk', Array, '{"index":{"_ind...', Array)
  ~  ~  ~  ~  #8 /var/www/sugarcrm/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php(347): Elasticsearch\Client->performRequest(Object(Elasticsearch\Endpoints\Bulk))
  ~  ~  ~  ~  #9 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(497): Elasticsearch\Client->bulk(Array)
  ~  ~  ~  ~  #10 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(452): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer->sendBatch(Array)
  ~  ~  ~  ~  #11 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(266): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer->indexBatch('Leads', Array)
  ~  ~  ~  ~  #12 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(213): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer->indexBeans('Leads', Array)
  ~  ~  ~  ~  #13 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(146): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer->indexModule('Leads')
  ~  ~  ~  ~  #14 /var/www/sugarcrm/lib/Search/ElasticSearch/ElasticSearchIndexer.php(586): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer->index()
  ~  ~  ~  ~  #15 /var/www/sugarcrm/lib/Robo/Plugin/Commands/ElasticSearchCommands.php(120): SuiteCRM\Search\ElasticSearch\ElasticSearchIndexer::repairElasticsearchIndex(false, 0)
  ~  ~  ~  ~  #16 [internal function]: SuiteCRM\Robo\Plugin\Commands\ElasticSearchCommands->elasticIndex('0', 0, Array)
  ~  ~  ~  ~  #17 /var/www/sugarcrm/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
  ~  ~  ~  ~  #18 /var/www/sugarcrm/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
  ~  ~  ~  ~  #19 /var/www/sugarcrm/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
  ~  ~  ~  ~  #20 /var/www/sugarcrm/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(313): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
  ~  ~  ~  ~  #21 /var/www/sugarcrm/vendor/symfony/console/Command/Command.php(259): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
  ~  ~  ~  ~  #22 /var/www/sugarcrm/vendor/symfony/console/Application.php(938): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
  ~  ~  ~  ~  #23 /var/www/sugarcrm/vendor/symfony/console/Application.php(266): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
  ~  ~  ~  ~  #24 /var/www/sugarcrm/vendor/symfony/console/Application.php(142): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
  ~  ~  ~  ~  #25 /var/www/sugarcrm/vendor/consolidation/robo/src/Runner.php(282): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
  ~  ~  ~  ~  #26 /var/www/sugarcrm/vendor/consolidation/robo/src/Runner.php(158): Robo\Runner->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput), Object(Robo\Application), Array, Object(Composer\Autoload\ClassLoader))
  ~  ~  ~  ~  #27 /var/www/sugarcrm/vendor/bin/robo(49): Robo\Runner->execute(Array)
  ~  ~  ~  ~  #28 {main}
[@][18:53:42] 0 modules, 1 records and 18 fields indexed in 1.275 s
[=][18:53:42] Indexing complete

samus-aran · 5 January 2022 19:57

mmm initial searching I don’t see anything obvious. The assumption that you have followed the documentation to set everything up.

If so, I would look at where this gets kicked off:

github.com

salesagility/SuiteCRM/blob/a4afbded5514f64e11e9d4eaf995bfe96fda86c9/lib/Search/ElasticSearch/ElasticSearchIndexer.php#L111

    
      
              try {
                  return $sugar_config['search']['ElasticSearch']['enabled'];
              } catch (Exception $exception) {
                  LoggerManager::getLogger()->fatal("Failed to retrieve ElasticSearch options");
          
          
        return false;
              }
          }
          
          
/** @inheritdoc */
          public function index()
          {
              $this->logger->info('Starting indexing procedures');
          
          
    $this->resetCounts();
          
          
    $this->logger->debug('Indexing is performed using ' . $this->getDocumentifierName());
          
          
    if ($this->differentialIndexing) {
                  $this->lastRunTimestamp = $this->readLockFile();
              }

Which then specifically here. Perhaps see what this spits out in the logs and see if the mapping is incorrect.

github.com

salesagility/SuiteCRM/blob/a4afbded5514f64e11e9d4eaf995bfe96fda86c9/lib/Search/ElasticSearch/ElasticSearchIndexer.php#L194

    
      
           * @param array|null $body options of the index
           */
          public function createIndex(string $index, array $body = null): void
          {
              $params = ['index' => $index];
          
          
    if (!empty($body) && is_array($body)) {
                  $params['body'] = $body;
              }
          
          
    $this->client->indices()->create($params);
          
          
    $this->logger->debug("Created new index '$index'");
          }
          
          
/** @inheritdoc */
          public function indexModule($module)
          {
              $isDifferential = $this->differentialIndexing();
              $dataPuller = new ElasticSearchModuleDataPuller($module, $isDifferential, $this->logger);

isaacmarco · 6 January 2022 19:23

Elasticsearch is running in the same server. The ping test works OK and the “Installation test” responses with no errors…

# curl -X GET "localhost:9200/"
{
  "name" : "_XBnDUB",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "hpQwwU3hQECWuJsUACa2yA",
  "version" : {
    "number" : "5.6.16",
    "build_hash" : "3a740d1",
    "build_date" : "2019-03-13T15:33:36.565Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

I’ve enabled and disabled different modules in the suitecrm “Search settings” and same results were got.

The problem occurs here:

github.com

salesagility/SuiteCRM/blob/a4afbded5514f64e11e9d4eaf995bfe96fda86c9/lib/Search/ElasticSearch/ElasticSearchIndexer.php#L497

    
      
          /**
           * Sends a batch request with the given params.
           *
           * Sends the requests and attempts to parse errors and fix the number of indexed records in case of errors.
           *
           * @param array $params
           */
          private function sendBatch(array &$params): void
          {
              // sends the batch over to the server
              $responses = $this->client->bulk($params);
          
          
    if (isset($responses['errors']) && $responses['errors'] === true) {
                  // logs the errors
                  foreach ($responses['items'] as $item) {
                      $action = array_keys($item)[0];
                      $type = $item[$action]['error']['type'];
                      $reason = $item[$action]['error']['reason'];
                      $this->logger->error("[$action] [$type] $reason");
          
          
            if ($action === 'index') {

and this is my var_dump($params) used in the line, but I don’t know where is the problem:

array(1) {
  ["body"]=>
  array(2) {
    [0]=>
    array(1) {
      ["index"]=>
      array(2) {
        ["_index"]=>
        string(8) "contacts"
        ["_id"]=>
        string(36) "16494016-fa03-55b5-9e51-4f3b89d52f57"
      }
    }
    [1]=>
    array(3) {
      ["name"]=>
      array(3) {
        ["name"]=>
        string(23) "Rafael Monroy Caballero"
        ["first"]=>
        string(6) "Rafael"
        ["last"]=>
        string(16) "Monroy Caballero"
      }
      ["meta"]=>
      array(3) {
        ["created"]=>
        array(3) {
          ["date"]=>
          string(16) "02/15/2012 10:32"
          ["user_id"]=>
          string(36) "c2e00472-84ea-5c12-4970-4c8e38071b64"
          ["user_name"]=>
          string(13) "sonia.jimenez"
        }
        ["modified"]=>
        array(3) {
          ["date"]=>
          string(16) "02/15/2012 10:32"
          ["user_id"]=>
          string(36) "c2e00472-84ea-5c12-4970-4c8e38071b64"
          ["user_name"]=>
          string(13) "sonia.jimenez"
        }
        ["assigned"]=>
        array(2) {
          ["user_id"]=>
          string(36) "c2e00472-84ea-5c12-4970-4c8e38071b64"
          ["user_name"]=>
          string(13) "sonia.jimenez"
        }
      }
      ["email_and_name1"]=>
      string(26) "Rafael Monroy Caballero <>"
    }
  }
}

isaacmarco · 10 January 2022 11:02

The problem was solved. SuiteCRM 7.12.2 requires ElasticSearch 7 and not 5.6. Documentation about set up Elastic Search is not upgraded Set up Elasticsearch :: SuiteCRM Documentation

I think it was deployed here https://github.com/salesagility/SuiteCRM/pull/9171

isaacmarco · 10 January 2022 11:07

I have sent a patch for documentation here… https://github.com/salesagility/SuiteDocs/compare/master...marcoblancas:patch-1

3boptic · 22 March 2022 20:29

Is there any guide on upgrading Elastic Search?