It can be done 
If you look at the layout defs of your subpanel you will see the definitions for the edit and remove buttons. I am looking at custom/modules/Contacts/metadata/subpanels/Account_subpanel_contacts.php
In there I can see :
'edit_button' =>
array (
'vname' => 'LBL_EDIT_BUTTON',
'widget_class' => 'SubPanelEditButton',
'module' => 'Contacts',
'width' => '5%',
'default' => true,
),
'remove_button' =>
array (
'vname' => 'LBL_REMOVE',
'widget_class' => 'SubPanelRemoveButton',
'module' => 'Contacts',
'width' => '5%',
'default' => true,
),
notice that they both point to a widget_class
'widget_class' => 'SubPanelEditButton']
And
'widget_class' => 'SubPanelRemoveButton',
Widget classes are contained in : include/generic/SugarWidgets. But what you want to do is create your own widget class in custom/include/generic/SugarWidgets
If you inspect the actual edit button you will see a link similar to the following :
index.php?module=Contacts&action=EditView&record=6aa797c0-3b56-05b2-b2db-57ac8d113a4c&parent_module=Accounts&parent_id=cef2707e-3fd0-11e5-39d7-57557b10ceaf&return_module=Accounts&return_id=cef2707e-3fd0-11e5-39d7-57557b10ceaf&return_action=DetailView&return_relationship=contacts
What you need to do is create your definition in the subpanel layout defs that points to your own custom widget class that adds a link to your custom action in a similar manner to the above link.
Simple!!