The Security Groups Subpanel is not created by default for custom modules.
You have to create it manually in custom/Extension/YOUR_MODULE/Ext/Layoutdefs.
Create securitygroups_Layoutdefs.php in that folder (the file name is not significant) with the following contents:
<?php
$layout_defs['YOUR_MODULE']['subpanel_setup']['securitygroups'] = array(
'top_buttons' => array(array('widget_class' => 'SubPanelTopSelectButton', 'popup_module' => 'SecurityGroups', 'mode' => 'MultiSelect'),),
'order' => 100,
'sort_by' => 'name',
'sort_order' => 'asc',
'module' => 'SecurityGroups',
'refresh_page'=>1,
'subpanel_name' => 'default',
'get_subpanel_data' => 'SecurityGroups',
'add_subpanel_data' => 'securitygroup_id',
'title_key' => 'LBL_SECURITYGROUPS_SUBPANEL_TITLE',
);
You would also need to define the label for LBL_SECURITYGROUPS_SUBPANEL_TITLE in your custom module language file.
Then run Repair / Quick Repair