custom field/table disable in different groups

Hi together,

Ive created a custom table for contacts to seperate contacts to special needs. This field gets populated in every group ive created as well.
I dont want group user to see this additional field - does anybody know how to do this.