Import issues : stat and touch

obekatli · 17 February 2020 12:34

Hi all,

There are several threads in the forum with import, stat & touch, most of them are older.

Just a similar experience on this case(7.11.11 yet to be upgraded to 7.11.12) on CentOS with php 7.3, memory_limit=512M and timeouts adjusted for long durations.

I have started import of 14K Accounts to a fresh installation.

I have experienced dropdown error for Account type for an account. At the end Import showed on screen for the record which has incorrect drop down value for account type. So I received Step 5: View Import Results screen with 1 error.
I have corrected error and tried to run Import again for only new records and expected to import only one account but no it took like first import and Step 5: View Import Results Screen shown. When I check accounts I see that it has imported all records against the setting and I had 29K accounts all with duplicates.

So no View Import Results and duplicates created.

I have deleted all the accounts(and duplicates) and re-tried with the file which I have corrected account type for a record and it took 16mins to import all 14K accounts in a single run. Step 5 View Import Results shown as usual. I have checked the Accounts, all are OK, Security Groups and users are correctly assigned for accounts.

But the suitecrm.log has following error lines from the beginning of process, not the end:
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://IMPORT_Account_1-0 cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://import/dupes_1.csv cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://import/dupesdisplay_1.csv cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://import/errorrecords_1.csv cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://import/errorrecordsonly_1.csv cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://IMPORT_Account_1-1 cannot be touched
Mon Feb 17 12:51:41 2020 [3864][1][ERROR] File upload://IMPORT_Account_1-2 cannot be touched
…
…
Mon Feb 17 12:51:43 2020 [3864][1][ERROR] File upload://IMPORT_Account_1-86 cannot be touched
Mon Feb 17 12:51:43 2020 [3864][1][ERROR] File upload://IMPORT_Account_1-87 cannot be touched
Mon Feb 17 12:52:00 2020 [3862][1][FATAL] Caught error: stat(): stat failed for upload/import/status_1.csv
Mon Feb 17 12:52:00 2020 [3862][1][FATAL] Caught error: touch(): UploadStream::stream_metadata is not implemented!
Mon Feb 17 12:52:00 2020 [3862][1][ERROR] File upload://import/status_1.csv cannot be touched

So, if import file contains incorrect dropdown value, it tells about it but the option to import only the new ones, not the duplicates not affected and it imported all again and if this is the case, the duplicates created, Import summary is not shown. I cannot reproduce this with smaller import file.

After that I have tried to import Contracts of these accounts. It was OK but the suitecrm .log has following additionaly:(similar stat and touch errors)

Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://IMPORT_AOS_Contracts_1-0 cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://import/dupes_1.csv cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://import/dupesdisplay_1.csv cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://import/errorrecords_1.csv cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://import/errorrecordsonly_1.csv cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://IMPORT_AOS_Contracts_1-1 cannot be touched
Mon Feb 17 13:12:59 2020 [17082][1][ERROR] File upload://IMPORT_AOS_Contracts_1-2 cannot be touched
Mon Feb 17 13:13:09 2020 [3893][1][FATAL] Caught error: stat(): stat failed for upload/import/status_1.csv
Mon Feb 17 13:13:09 2020 [3893][1][FATAL] Caught error: touch(): UploadStream::stream_metadata is not implemented!
Mon Feb 17 13:13:09 2020 [3893][1][ERROR] File upload://import/status_1.csv cannot be touched

But the Contracts successfully created and related to the corresponding accounts, no issues.

So, any ideas about this stat and touch errors ?

obekatli · 18 February 2020 13:27

Hi all,

Today I have tried with 7.11.12 and found that first part of

Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-0 cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://import/dupes_1.csv cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://import/dupesdisplay_1.csv cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://import/errorrecords_1.csv cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://import/errorrecordsonly_1.csv cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-1 cannot be touched
Tue Feb 18 14:15:05 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-2 cannot be touched
…
Tue Feb 18 14:15:08 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-145 cannot be touched
Tue Feb 18 14:15:08 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-146 cannot be touched
Tue Feb 18 14:15:08 2020 [2352][1][ERROR] File upload://IMPORT_Account_1-147 cannot be touched

messages come before actual import process but after selected the file and mapping properties.

And immediately after clicking IMPORT NOW

Tue Feb 18 14:20:59 2020 [1900][1][FATAL] Caught error: stat(): stat failed for upload/import/status_1.csv
Tue Feb 18 14:20:59 2020 [1900][1][FATAL] Caught error: touch(): UploadStream::stream_metadata is not implemented!
Tue Feb 18 14:20:59 2020 [1900][1][ERROR] File upload://import/status_1.csv cannot be touched

messages appear in the suitecrm.log. But import process commenced,

And after 4minutes Importing counter popup disappeared and no Step 5: View Import Result screen displayed. As this is the first run after initial installation, it is strange that import breaks in the middle.

There is no information in any log.

I have checked Accounts and 3000 accounts imported but the others not processed.

Any ideas ?

obekatli · 18 February 2020 13:51

Update: I have deleted imported 3000 records and started Import again. Nothing changed in config files and now it has imported 14K Accounts from the same CSV file in 15:24 and no errors.

This is simple validated(No Dropdown errors, no calculated fields etc) CSV.

So, these error Messages even called as Fatal have NO visible/yet detectable affects on Import process or imported records. I have checked some of my Accounts with sampling and all seems fine.

Although no timeouts in httpd or php logs for the first attempt, I have checked my settings and they are:
config.php of suitecrm : import_max_execution_time’ => 7200

suitecrm.conf of httpd : TimeOut 3600 ProxyTimeout 3600

Luckily I have imported all my accounts but have not understood my it did not complete at first attempt.

Just wanted to share the experience, commencing to Contracts, Contacts, Tasks etc import

obekatli · 18 February 2020 15:22

Update: Till now I have imported Contracts and Contacts.

Same set of Error/Fatal messages displayed for their import also. And for contacts (which has more than 1K) I have experience again a sudden disappear of Import progress popup and, import breaks.

I have applied same method, deleted the imported contacts so far, re-try the import without changing anything and it has completed processing all the imports and show Step 5 View Import Results at second time.

This behaviour is strange.

pgr · 18 February 2020 17:18

These errors are in suitecrm.log, right?

Have you also been keeping an eye on php_errors.log?

obekatli · 19 February 2020 06:29

Yes they are from suitecrm.log and nothing in php log. The last php error in the log was from Module Loader case(subsequent language pack installation).

obekatli · 19 February 2020 09:50

UPDATE: Today I have prepared Notes to be imported. 106K notes.

a. Same touch messages(more than before, because file is bigger)
b. Same stat messages.
c. Same “popup disappearing” case occurred after importing 59.900 notes in 46mins, screen remained on Step4. No running process on linux top, no error messages on php log or suitecrm log, no timeouts.
d. I have re-tried same file with adding ParentID and Subject to duplicate control and clicked import, after 6 minutes. But I have a new case screen below, no text after numbers, 1 errors , 2468 imported? 103910?

pgr · 19 February 2020 12:28

Como está en tu php.ini el error_reporting y el log_errors?

Y error_log?

obekatli · 19 February 2020 12:39

Hi pgr,

According to phpinfo of Diagnostic tool

error_reporting : 25219 = E_ALL & ~E_NOTICE & ~E_STRICT & ~E_DEPRECATED
log_errors : On
error_log : /var/log/php-fpm/www-error.log and no errors in it.

obekatli · 20 February 2020 10:59

Hi Pedro,

Today I have tried with smaller groups but result is the same.

Centos 8.1-1911-php 7.3.14-SuiteCRM 7.11.12(fresh install), php parameters above.
4 cores, 3GB RAM-, HDD -virtual machine

a. I have deleted all the records, pruned the tables, optimized the tables.
b. Tried to import in groups of ~10.000 records- average 10mins-max 16mins.

No issues for import. For some groups Importing Records popup disappear in the middle and process stopped, no error in logs. It cannot be timed out with above timeout parameters(6000).

Sometimes, Import File Properties changes without my intention, I always needed to check Date and Time format before proceeding to next step and some time I found Time format was changed, sometimes Date format was changed. I even did not logout from Admin between import groups.

Screen as below at the end:

pgr · 20 February 2020 13:56

I frequently see CentOS users complaining about strange things, maybe it’s the SELinux stuff? Are you using it?

Also, see if this helps you:

obekatli · 20 February 2020 14:18

Good point, yes SELinux is enabled and enforcing as it should be. I always keep eyes on audit.log. I do not see any “denied” in audit.log, so I do not think it would be the problem.

Thanks for the article, I expect that method would be better and faster but requires more effort to import records.

On the other hand, same screen works fine, displayed fine when I have imported 800 records for example, but after 1500 records button texts get lost, but import is correct. This cannot be related with data insert method I guess. I suspect from some memory issues during the code or php methods but cannot identify them.

Anyway, I have successfully imported whole my data, Accounts, Contracts, Contacts, Notes, Tasks. I have also applied Country->Dropdown (not working for Filter but not so important) and your custom theme solutions.

As you have always said, I always need to check logs(php-fpm logs, httpd logs, suitecrm log, audit log), carefully set php parameters and permissions, php/httpd timeout parameters. I have checked so many posts in the forum and suprisingly see that you have recommended patiently the same topics/commands.

Thanks for all to you and the team. We are ready to use SuiteCRM for the real life.

I would say I have no problems for import after all efforts and saw the records are successfully in SuiteCRM and related to each other. I’m also intending to prepare a simple guide for import to be a reference for others.

It is nice to see all previous Contacts, Notes, Tasks of an Account all together in an organised way.

I have also tested the mobile interface and ta-daaa, it is great even for smaller phone display sizes.

Thanks.