Stop LSP & clear its automands under specific conditions

Language Server Protocol (LSP)
1

I want to stop the LSP when I open a minified file or a file under node_modules for example, so I have these autocmds

augroup stop_lsp
  autocmd!
  autocmd BufEnter,WinEnter */node_modules/* :LspStop
  autocmd BufLeave */node_modules/* :LspStart
  autocmd BufEnter,WinEnter *.min.* :LspStop
  autocmd BufLeave *.min.* :LspStart
augroup END

This kind of works, because the LSP will stop after it was attached, so after it is stopped the autocmds from the custom on_attach are still running & they will throw errors for example these highlight autocmds will fail.

local function on_attach(client)
  if client.resolved_capabilities.document_highlight then
    vim.api.nvim_exec(
      [[
      hi! link LspReferenceRead SpecialKey
      hi! link LspReferenceText SpecialKey
      hi! link LspReferenceWrite SpecialKey
      ]],
      false
    )

    utils.augroup(
      "__LSP_HIGHLIGHTS__",
      function()
        vim.api.nvim_command(
          "autocmd CursorHold <buffer> lua vim.lsp.buf.document_highlight()"
        )
        vim.api.nvim_command(
          "autocmd CursorHoldI <buffer> lua vim.lsp.buf.document_highlight()"
        )
        vim.api.nvim_command(
          "autocmd CursorMoved <buffer> lua vim.lsp.buf.clear_references()"
        )
      end
    )
  end
end

So I’m looking for a way to stop the LSP if possible before it’s being attached or clear the autocmds when it’s stopped?