You can delete the entire folder cache/upgrades/temp.
There was a bug that caused files to pile up in that folder. Check how much things improve after deleting that.
After that, use some utility software to know where your files are. In Linux, you can use du for this; or ncdu which is the same with a nicer UI. In Windows, I suggest TreeSize Free.