Both jedi-language-server and pyls are very slow, unfortunately (they’re both powered by jedi, which is the culprit here). There’s actually an option to have jedi skip analysis of some modules, and you may want to enable that for slow modules (it’s called auto_import_modules: Settings — Jedi 0.18.0 documentation)
The jedi-language-server readme has instructions on how to set jedi settings: GitHub - pappasam/jedi-language-server: A Python language server exclusively for Jedi. If Jedi supports it well, this language server should too.