Why did nvim_set_keymap() changed and vim.keymap.set() didn’t?
Not sure what you mean. vim.keymap.set() was introduced after support for callback was added to nvim_set_keymap. So vim.keymap.set() didn’t change, it’s totally new.
The distinction between rhs and callback in nvim_set_keymap was necessary because it can be called over rpc. So the type of rhs was kept the same (string) and a callback key was added to the options table, which is only usable inside lua script (not over rpc).
which-key just needs a small update (a couple of PRs are already open) to fix the issue.
Let me reiterate that: the use of callback for native Lua function mappings is very much necessary complexity; it couldn’t have been done any other (better) way – hence the more convenient vim.keymap API on top of it.
A reasoned PR (explaining the rationale) introducing vim.keymap.get would be well-received, though.