Exporting large reports ~600k lines

SuiteCRM Forum - English Language Developer Collaboration
1

Hey !

Me again :stuck_out_tongue:

I’m facing a problem when i try to make export report. I need to generate a large file but i got 502 error. May be i’m missing a conf value i’ll hope you can help me.

thanks !

2

Check out apache and php error log . It could be an issue of max_execution_time of script.
Based on error you can set your php.ini

3

Hey :slight_smile:

I have a connection timed out (using nginx) i raised executions time to one hour for, nginx, php fpm (ini & www.conf)

4

Please confirm if it works for you. So thread can be marked as solved

5

This is not the solution.

It’s still not exporting, the max that I can reach is ~45k rows but it’s not enough =/

Now I have a 504 o/

6

Instead of guessing what the problem might be, you really should be looking at your PHP log, as instructed above by @sagarjaydeep, to see which exact error you’re getting. This will point you to the correct php.ini parameter to adjust.

If your export turns out to really be too big, you might be better off trying it directly from the database with phpMyAdmin or similar.

7

[error] 26428#26428: *5935 upstream timed out (110: Connection timed out) while reading response header from upstream,

8

It could be an issue of proxy_read_timeout or fastcgi_read_timeout on your nginx.

9

in the file nginx.conf :

    keepalive_timeout 900;
    proxy_read_timeout 900;

in vhost :
fastcgi_read_timeout 900;

I could raise to 50k but it doesn’t take 15min to execute… Still missing something

10

Hey !

are we missing something else ?

11

Sorry, I don’t have any further assumption as i don’t have it on front of me. Only log will help you here for further investigation.

12

I have a 504 with

nginx error log :

empty

php7.2-fpm log :

empty too

Chrome page :

504 Gateway Time-out

The server didn’t respond in time.

13

If i got logs, i think i would be able to debug… Someone already made big exports ? More than 100k ?

14

Don’t recall which entries to increase but I had the same issue and had to increase various php config entries; eventually I can export 700k lines of records in the Accounts module to export without hanging/crashing

I googled on how to incrase timeouts and filesizes in the PHP config file

15

It could be a better idea to get data directly from the database, for those volumes.

This is about Imports but it might also help:

16

Thanks for the post.

I have created a new function in reports that i called fast export, i can export now ~600k lines in few seconds without changing any php configuration.

Thanks for your help !

@David_iTeks How long take your export ?

17

Hi Seijitsu - I’d be interested in implementing your function. If you’re willing to share your function and steps to implement it, I’d greatly appreciate it.
Having to reconfigure PHP is a real pain but that’s the only way I could do it.

Thanks.

18

Depends of what you have,

If you have a custom/modules/AOR_Reports/* working i can give you the code, if not I have to make a complete tutorial =)

19

Hi Seijitsu,
i also want to export more than 50k rows of Reports module,but how it is possible,you write a script for this.Can you please help me where i write the script and what the code is ?

Thanks,
Vishal Raj