HTML-entities are auto-encoded for custom-fields (_cstm)

karada · 9 July 2020 09:11

Hi,
I made the observation that custom fields automatically html-encode symbols such as & etc if editing fields via a module front-end in SuiteCRM. This is only done for custom fields. Field values that are stored in the main table (no _cstm suffix), this auto-encoding is not applied.
Can I somehow turn it off for the custom fields?

pgr · 9 July 2020 09:19

I think this happens for all fields, not just custom fields.

Can you please provide an example of a non-custom field that doesn’t get html-encoded when saving?

karada · 9 July 2020 09:29

We are not using any of the default modules, in case this is a difference but if I edit the “name” or “description” field I receive the html entities unecoded as I wrote them in the fields.
Just any custom field (_c) is stored with &_amp; etc.

I can replicate that for any module. This seems to be a general behavior?

I am using SuiteCRM 7.11 on a Maria DB in case this of relevance?

pgr · 9 July 2020 09:38

Let’s pick a specific module/field and do all our tests on that one, ok? For example, Accounts module, description field. How does that work for you?

Also, please keep an eye on the database, and confirm how it looks there, also for the custom fields. We need to understand if the problem is happening when saving, or when displaying - two different moments.

karada · 9 July 2020 09:48

Ok I added a custom field to the Accounts module for a test case:

test 1) Write & (entered in Account-editview)
Hello & world
result in DB:
(name) --> Hello & world
(custom-field) --> Hello &_amp; world

test2) edit-view read from DB data of test1
both fields show: Hello & world (no html-entities - its correctly decoded)

test3) edit-view read from DB
I manually edited both fields to have html-entities in the DB (so the name field the custom field has that already)
Observation 1) Both fields are correctly decoded and appear as & when editing the record and seeing in the editview
Observation 2) When clicking on save, the name field is stored without decoding! again

Summary: Storing of field values to default fields always stores html entities decoded when saving from an edit view, custom fields are always encoded. No problem for reading or showing DB data, both are displayed correctly regardless of their DB representation.

pgr · 9 July 2020 09:57

In your Observation 2), please clarify:

a. how does it look in the DB?
b. how does it look on screen (wrong, I guess)?
c. if you reload the screen (detail view) does it still appear wrong?

karada · 9 July 2020 10:02

a)
name: Hello & world
custom-field: Hello &_amp; world

b)
its presented correctly, both fields show ‘Hello & world’

in Editview
on the Detailview

c)
nope, bot fields are correctly shown on editview and detailview but their DB representation is still as in a)

Do yo have a chance to replicate it? by storing an & in any custom field?

pgr · 9 July 2020 10:35

But if they always show correctly, what is your problem? In your initial post, I thought you were complaining about how it displayed. Maybe you were complaining about how it is stored in the DB instead? Why is this a problem for you?

I am just trying to understand better what the problem is, sorry for so many questions.

karada · 9 July 2020 10:40

No, my problem is that they are stored differently in the DB. I would want all fields without html-entities. This is an inconsistency how data is stored, some fields come out in way A) while some come out way B).

I would want a consistent way otherwise I have to remember which fields require decoding and which don’t ( I am exporting the data to another application where this different way of storing is a problem, i.e. some columns are used in a group by fashion and & vs &_amp; are two different groups)

pgr · 9 July 2020 10:56

Ok.

This is a big problem in SuiteCRM - it uses excessive (or misplaced) security encodings. HTML encodings should be applied before sending HTML to the client, instead, they are applied when storing in DB (only DB-related escaping should be applied at that point).

You can try seeing what happens as the request enters SuiteCRM, here:

github.com

salesagility/SuiteCRM/blob/master/include/utils.php#L2492


{
    return substr($str, 0, strlen($begin)) == $begin;
}


// Returns TRUE if $str ends with $end
function str_end($str, $end)
{
    return substr($str, strlen($str) - strlen($end)) == $end;
}


function securexss($value)
{
    if (is_array($value)) {
        $new = array();
        foreach ($value as $key => $val) {
            $new[$key] = securexss($val);
        }


        return $new;
    }
    static $xss_cleanup = ['&quot;' => '&#38;', '"' => '&quot;', "'" => '&#039;', '<' => '&lt;', '>' => '&gt;', '`' => '&#96;'];

I don’t understand the reason for the different treatment of custom vs normal fields. But maybe if you set some breakpoints there you can compare how each is handled, and confirm that in fact this is where the change is occurring.

karada · 9 July 2020 11:09

Thanks for the hint. I will try to trace it.
I just noticed this happens only for &, the other symbols are not affected. I was too focused on the &-issue and assumes that it applies to other html-entities, too.

Did you manage to replicate the problem?

pgr · 9 July 2020 11:26

Sorry, I am too busy to go into replicate-and-try-to-fix mode…

But I know for a fact that this is a problem, for the past 2-3 weeks I’ve been facing this because of the difficulties sending un-broken email HTML content to the server, while working on this.

I realized our securexss function is misguided and it would require a massive fix, with implications in many points of SuiteCRM.

Funny, it is too safe - breaks legitimate uses of HTML in data…

However in your case, if it changes only one entity, and if it behaves differently for custom fields, I fear that problem might be elsewhere, which would mean you would still need to go hunt for it…

pgr · 9 July 2020 14:20

Another idea for you: what exactly is the type of the two fields you are using in your tests?

Make sure they are the exact same type. There is different handling according to SugarField type, and maybe that is what causes the different behaviour, not the fact that it is custom.

karada · 9 July 2020 15:36

Hm, so both fields end-up as varchar in the database like the standard name and description field. I think in SuiteCRM they were defined as TextFields, is there an alternative to that which would circumvent the problem?

pgr · 9 July 2020 18:14

The type they have in the vardefs is what determines the different SugarField handling. It is not the same thing as the database type. For example, there are different types for these kinds of SugarFields…

name
text
description
link

… and they all get converted to the same type in the database.

If you can find which types they have in the vardefs you can look into the classes handling the save, I think you might find some code there that runs differently for each case.