Never really understood that Secruity group.
So as a workaround I have made a logic hook on before save that assign the note to the authors. And now He can only edit/delete the note he is assigned to (So his own note)
function AssignationUserNoteFonction($bean, $event, $args)
{
//Created by contien l'id de l'auteur
$IDAuteur = $bean->created_by;
//Prend l'ID est le rentre dans le champ Assigné à
$bean->assigned_user_id = $IDAuteur;
}