De-duplicating libraries will help a little bit, but not much; however, you don't need to manually go through all your addons and pull out the libraries, and you definitely shouldn't just delete the duplicates. If you want to see how much difference it makes, I'd suggest using the Curse Client, and enabling the "Install libraries separately" option, then click the "Reset All" button next to that option, then "updating" all of your addons to reinstall the latest version of each one with the libraries installed as separate addons. You may find that some addons still have libraries inside (even ones from Curse) -- those addons may not be configured properly to use standalone libraries, so you should just leave them alone. I don't anticipate that this will make much difference though. The last time I experimented, I was using about 200 addons, and switching to standalone libraries only reduced loading time by about 10-15%.
However, based on your list, I'd guess that the majority of the time you spend staring at loading screens is taken up by just a few of your addons. I can personally attest to Grail/Wholly and TomTom being
very slow loaders (Grail/Wholly due to the massive amount of data it loads, and TomTom because of the Astrolabe library), and I'd guess several of your other addons with large databases (Altoholic, Engravings, Gatherer, LightHeaded, maybe some others) are probably also fairly slow.
You can use
Warmup to see how long each addon is taking to load, but keep in mind that (a) running Warmup makes everything load 2-4x
slower, and (b) load times for addons loading libraries will be somewhat inaccurate, especially for the first addon loading a given library, but you should be able to get a general idea of which addons are taking the most time to load. Don't forget to disable it when you're done, as it drastically increases loading times across the board.