There is no common method for additional vendor buttons, is there?
 

Similiar to the worldmap, the vendor frame becomes a problem when multiple addons try to extend the functionality of the vendor frame with additional buttons.
I have multiple buttons at the same place that are trying to sell grey items, transmog stuff, buy tradeskill reagents and probably even more things.

I am not sure how to solve this mess because not every addon author can and should check which other addons are there so he can move the button out of the way.
This becomes even worse with addons like GoblinVendorShrinker or Vendorer that modify the size of the default frame.

So i was wondering if there are efforts to have some kind of a library where all interested addons can register their buttons and we have some kind of dropdown or menu bar where everything is neatly ordered.

It doesn't directly answer the question you're asking, but you might want to make it configurable where your buttons are placed. If the user can drag them where they want then they can make sure it works with their configuration.