Importing 4000 contacts takes a very long time and maxes out RAM/CPU

💬 General Discussion
1

Hello! I have recently upgraded SugarCRM to SuiteCRM 7.1.2, and I love it! Works great, I like it so much more than Sugar, so good job! =D

However, I’m having problems importing contacts that were not present before upgrading. I have about 50 CSV lists of contacts to import, each ranging from 4000 to 8000 contacts, with only about 6 fields each. When I import a CSV with 4000 contacts, it remains on the modal dialog
“Importing 1 to 4000 of 4000” for around an hour. Until I increases the server’s memory from 2 GB to 4 GB, this would often consume all of the available memory and kill apache, leaving the file half-imported. During the import process now, the 3 instances of apache that are running are consuming over 2 GB of memory and around 100% of 2 of the 3 available CPU cores!

Finally, when the process is complete, I get this error in the modal window:

NOTICE: [8] Undefined index: Import on line 250 in file 
NOTICE: [8] Undefined index: Import on line 250 in file 
NOTICE: [8] Undefined index: Import on line 250 in file 
NOTICE: [8] Undefined index: Import on line 250 in file 
NOTICE: [8] Undefined index: Import on line 250 in file 
NOTICE: [8] Undefined index: Import on line 250 in file

…etc., scrolling down hundreds of lines

Something is definitely wrong! Something I know is different since upgrading to SuiteCRM is that it used to import in blocks of either 100 or 1000 (cant remember) so if I uploaded a CSV with 4000 records, the modal window would say

“Importing 1 to 100 of 4000”
then
“Importing 101 to 200 of 4000”
… etc, rather than doing it all at once?

Here is the import section of my config.php

  'import_max_execution_time' => 36000,
  'import_max_records_per_file' => 10000000,
  'import_max_records_total_limit' => '',

After upgrading to SuiteCRM, import_max_records_per_file was set at 100, but this would halt the import after 100 records, and spit out that same error as above, at least now I can import thousands at once even if it does nearly kill the server :wink:

I also get some sql syntax errors relating to the import, which I’ll be happy to post if someone thinks it’s relevant, but I have a feeling it has more to do with the fact that it’s importing them all as one bulk operation rather than splitting the import into chunks like it used to.

I tried to research this on my own, but all the SugarCRM stuff I could find was old and didn’t seem really relate

Thanks in advance for your help!

2

Turn of Notices in your php.ini settings and I would also try to break up your large csv files into smaller ones so that you can find errors in the csv files themselves more easily. We have not made any changes from Sugar CE in SuiteCRM with regards to the import functionality so nothing will have changed.

3

Alright, thanks for clarifying that. I realized the high CPU usage was due to an addon module, and disabling it makes the import work with a normal load. Still get all of those errors, but the imports are working! I just dis 24000 at once and it took awhile but worked fine =D

Is there a way I should mark this thread as closed or solved?