Use modifier key to enable mouse mode?

I know that I can use shift+mouse to use my terminal based mouse mode (selection, etc). Is there any way to flip it so that unmodified mouse is like set mouse=, and modified mouse (let’s say ctrl+mouse, or maybe ctrl+alt+mouse) is like set mouse=a.

Or could I do keymaps for up and down events so that I could run the mouse mode toggle code?

Or is there an existing plugin to do something like this?

Thanks!