Can't get autocmd function to ignore certain buffer types. Help?

Hey all!

I just took the neovim pill a few weeks ago, and now I can’t seem to get enough. Longtime vim user, and I am blown away at all the new functionality of neovim and the new plugins. Thanks so much to everyone!

I’m trying to convert a Trailing White Space vimscript I had in my old .vimrc. I think I got the conversion done, but now when I bring up certain plugins (togglerterm in floating mode, alpha, Telescope), the trailing whitespace is run, and it makes it look ugly. I’d like to ignore certain buffer types, and I can’t seem to make it work. Looking for any help.

Herere is the original vimscript:

augroup WhitespaceMatch    
  " Remove ALL autocommands for the WhitespaceMatch group.    
  autocmd!    
  autocmd BufWinEnter * let w:whitespace_match_number =    
        \ matchadd('ExtraWhitespace', '\s\+$')    
  autocmd InsertEnter * call s:ToggleWhitespaceMatch('i')    
  autocmd InsertLeave * call s:ToggleWhitespaceMatch('n')    
augroup END 
   
function! s:ToggleWhitespaceMatch(mode)    
  let pattern = (a:mode == 'i') ? '\s\+\%#\@<!$' : '\s\+$'    
  if exists('w:whitespace_match_number')    
    call matchdelete(w:whitespace_match_number)    
    call matchadd('ExtraWhitespace', pattern, 10, w:whitespace_match_number)    
  else    
    " Something went wrong, try to be graceful.    
    let w:whitespace_match_number =  matchadd('ExtraWhitespace', pattern)    
  endif    
endfunction

highlight ExtraWhitespace ctermbg=red guibg=red   

And here is what I converted it to. I am commenting on the lines that don’t seem to work, instead of adding them in later. I do believe I have reached parity with the old script, but I can’t figure out how to ignore certain buffer types.

vim.cmd([[highlight ExtraWhitespace ctermbg=red guibg=red]])

local autocmd = vim.api.nvim_create_autocmd
local augroup = vim.api.nvim_create_augroup

local whtSpcGrp = augroup("ExtraWhiteSpace", { clear = true})

local excluded_buffertypes = {"toggleterm", "alpha", 'TelescopePrompt', 'Trouble', 'help', 'TelescopeResults'}

autocmd("BufWinEnter", {
  callback = function()
 
    -- my attempt to return early and not act on buffer type.
    if vim.tbl_contains(excluded_buffertypes, vim.bo.filetype) then
      return
    end

    vim.w.whitespace_match_number = vim.fn.matchadd('ExtraWhitespace', [[\s\+$]])
  end,
  group = whtSpcGrp,
})


autocmd("InsertEnter", {
  callback = function()
    ToggleWhitespaceMatch('i')
  end,
  group = whtSpcGrp,
})

autocmd("InsertLeave", {
  callback = function()
    ToggleWhitespaceMatch('n')
  end,
  group = whtSpcGrp,
})


function _G.ToggleWhitespaceMatch(mode)

  -- my attempt to return early and not act on buffer type.
  if vim.tbl_contains(excluded_buffertypes, vim.bo.filetype) then
    return
  end

  local pattern = ""
  if mode == "i" then
    pattern = [[\s\+\%#\@<!$]]
  else
    pattern = [[\s\+$]]
  end

  if vim.fn.exists(vim.w.whitespace_match_number) then
    vim.fn.matchdelete(vim.w.whitespace_match_number)
    vim.fn.matchadd('ExtraWhiteSpace', pattern, 10, vim.w.whitespace_match_number)
  else
    vim.w.whitespace_match_number = vim.fn.matchadd('ExtraWhiteSpace', pattern)
  end
end

When I run this, it works as I would expect in my normal files. However, when I run toggleterm, or alpha, it finds the whitespaces and highlights them. I’d really prefer it doesn’t.

I know there are other Trailing Whitespace plugins I could use, and I’m open to them, but converting this vimscript to lua was incredible practice in learning lua, neovim lua scripting, and neovim. I just got stuck on this.

Thanks for any advice.

Instead of using autocommand(s), why not try to apply filetype specific highlightings? For example, if you work with Python often, you could add this line to the ftplugin/python.vim file in the runtimepath:

highlight ExtraWhitespace ctermbg=red guibg=red

If you prefer Lua syntax instead, add the following line to the ftplugin/python.lua instead:

vim.api.nvim_set_hl.highlight(
    0, "ExtraWhitespace", { guibg = red, ctermbg = red }
)