Splitting uploads folder into subdirectories

Hi,

I’m just investigating SuiteCRM, and one of the ‘issues’ I’ve come across is that the uploads folder is not broken up into any sort of structure.
Ideally we would like it broken up into /uploads/year/month/{uuid}

I’ve come across http://cheleguanaco.blogspot.com/2015/05/sugarcrm-customization-custom-upload.html but it seems that the

$sugar_config['upload_wrapper_class']

is not used anywhere in SuiteCRM.

Is it possible to customize the upload behavior in an upgrade safe way?

1 Like

It’s not possible with that technique, because it is for code after version 6.5 which is when we forked.

It might be complicated to do a generic solution all in one place, but might be easier to achieve for just one module. Probably you are most concerned about one kind of thing - email attachments, or documents, or what?

Oh right, that’s a shame - I was hoping there was a single entry point for file uploads that I could extend.
My concern is that storing all the files in a single directory is going to cause issues down the line for even simple operations like ‘ls’.

There will be 10k’s of files if we migrate everything, so I don’t want to be dealing with that later!

I guess I will have a play and see what I can come up with - even if it is not upgrade safe - we can apply any changes needed back again on the test environment first.

1 Like

Remember this is an open-source, community project. If you can make a good, generic solution, you don’t have to worry about it being upgrade-safe because we can put it into the core code. :slight_smile:

This has been requested by other people, and I agree it makes perfect sense to have some separation.

So I had a little play and it seems a relatively small change to work with what I’ve tested so far.
I need to do some more testing and try some other modules, but I’ll post something back after.

Interestingly, when you delete a file it already moves it to a split structure:


/**
 * Returns path for files of bean or false on error
 *
 * @return bool|string
 */
public function deleteFileDirectory()
{
    if (empty($this->id)) {
        return false;
    }
    return preg_replace('/^(..)(..)(..)/', '$1/$2/$3/', $this->id);
}

In my Blog I have a few articles that can give you a more general idea

https://pgorod.github.io/How-Attachments-Stored/
https://pgorod.github.io/How-Documents-Stored/
https://pgorod.github.io/How-Photos-Stored/

If there isn’t a single place to change all those kinds of files, then I would say a solution just for one of them would be a step forward.

One thing that concerns me, if we make this change in the core code, is how to handle existing files when people upgrade. Perhaps leave them where they are and have some sort of mixed system (new files go in the new place, old files in the old).

Thanks pgr,

it does seem that there is a single place to modify for file uploads.
I haven’t extensively tested it, but it seems to be working so far, both uploads and deletes work, as does fetching the file again.

https://github.com/benperiton/SuiteCRM/commit/979711cc70810c28c694d74d8c36dac171a946d2#diff-6ca394ee6942027dd242d85ef14d3a7e

I’m not sure what the best way to handle that would be. Maybe an option on install that chooses how it should be organised, so it is for fresh installs only.
Then perhaps come up with a migration plan - but with lots of files, it could be painful!

Either way, I will continue to test and see if anything breaks.

1 Like

It’s nice to know all the different kinds of files are flowing through the same bit of code.

I guess the way to split the directories will generate some controversy. One directory per day is clearly too much for most people. Maybe a split by module would be nice to have. We’ll have to come up with some configurability or a middle ground…

Thanks for your contribution, it’s looking good! :slight_smile:

I love the general initiative. Having an upload folder with >10GB data is a pain to work with in many settings. And this happens fast when dealing with automatic Email integration (attachments).

Backing up files for example.
Or archiving and deleting only old ones…

To me once a day would be overkill.
But once a year absolute minimum.
Once a month would work great for me.

A good design imho would be to get the path and filename from a special string that can be changed in “Admin/Settings”.

This string could reference variables from the bean and the date.


/var/www/html/upload/%MODULE/%MONTH/%FILENAME.%EXTENSION

Avoiding using GUIDs for file names would also help, it makes looking for files in the file system much more straight-forward.

Maybe simple Filenames can be used and the GUID could just be appended to the name in the case of a file conflict. Note that there will be less conflicts once we break up the directories.

Better yet if this string definition could be made independent, by module. Maybe not from the UI, it’s too much work, but something that we could put into a configuration file to override the generic string. This would open the way for Invoices saved with the exact filenames you want for accounting, for example, like MyClient-001-May2019.pdf, etc

I am just thinking out-loud here, I am not asking @benperiton to implement all this :slight_smile: anything simple, but working, is better than a big plan that never gets done. Thanks

I think I’d simply split the upload dir based on the first character of the file GUID, so that when you get a download request you don’t have to use getBean() (and hit the database) to figure out where you stored it :slight_smile: Thanks @benperiton for the contribution!

@fanton.ff

the advantages of your proposal seems to be it’s simple to implement, solves the basic problem of splitting by directories.

The disadvantage would be that it’s not human readable, when seen from the file system. This means sysadmins don’t get an appreciation of where the disk space is going, which modules are guilty, etc.

The need to hit the database would not be necessarily a problem, it depends on what information is in the URL produced for that download. It could contain enough information (module, date, file name) to reach the file.

Thanks for commenting!

Following this thread !

Would be super usefull.

Was there any progress on this topic in the SuiteCRM v8?

I believe this was not changed by SuiteCRM v8.

Is there any solution (even paid) for this?

If there is enough interest, I wouldn’t mind putting a version of this inside my PowerFields add-on, which is available to people who sponsor me on Github (Pro Tier).

I think it makes sense to be included there in the sense of making the path construction a “calculated field” (which is what that add-on is about). Click my user name to see my signature for a link to my add-ons.

2 Likes

@vamos_tamos I guess what I wrote above implies that the price for this is $59 USD per month, which I admit is too expensive if all you want is the “split uploads” feature. Of course, if you use the rest of the add-on features (advanced templates and calculated fields) then perhaps the price seems more fair.

But suppose you only need the split uploads? How much would you be willing to pay?

And if others in this thread can also give their evaluation, I would appreciate it.

1 Like