You can make your own SQL query. It’s for a subpanel, I am not sure if it will work on List views, probably yes.
This should make all the hard processing be handled by MySQL, and everything will be done with one JOIN query for all rows, instead of a separate query per row. this should probably bring the performance into acceptable levels.