Error in FocusLost Autocommand

I use an autocmd to ensure the x-windows primary selection is updated with the neovim selection:

autocmd FocusLost * norm "*ygv

This has worked well, apart from when I’m in terminal mode. If I switch (desktop) windows while in Terminal mode, i get an annoying:

Error detected while processing FocusLost Autocommands for "*":           
Can't re-enter normal mode from terminal mode

Does anyone know of a solution?