Calling Neovim internal functions with LuaJIT FFI (and Rust)

In theory, using LuaJIT’s FFI it’s possible to access functions internal to Neovim, and after some discussion on Gitter I was able to get a mostly-working example. I’m posting it here for discussion purposes and maybe to inspire someone to write something cool.

Note: This code is by no means an example of the best way to do things, it’s just what I came up with after something like 30 minutes to an hour of tinkering around. I’m sure it could be made safer, better, etc.

Also, thank you to @mjlbach for help with getting these examples to compile and @bfredl for the C example and saying how this could be done/was possible.

The C Version

I’ll start out with an example using C and then move on to the Rust version. For this one, we first create a main.c file with the following contents:

// main.c

extern int name_to_color(const unsigned char *name);

int get_magenta(void)
{
  return name_to_color((unsigned char *)"Magenta");
}

I think the code is pretty self-explanatory, but just in case here’s what it’s doing: we define an external function name_to_color which will be referencing this function from src/nvim/syntax.c in Neovim. Then we create a function get_magenta which calls this function with a value of "Magenta". This should be roughly equivalent to running :lua vim.api.nvim_get_color_by_name("Magenta") from within Neovim once we call it using LuaJIT FFI.

Okay, so now that we have that C file, let’s compile it into a library we can load using LuaJIT’s ffi:

# This should work on Linux, but not macOS
$ gcc -o main.so main.c -shared -fPIC

# If you're using macOS, run this instead
$ gcc -o main.so main.c -shared -fPIC -Wl,-undefined -Wl,dynamic_lookup

Now that we have a main.so file we can use, let’s create a new file main.lua with this code inside:

-- main.lua

local ffi = require'ffi'

-- I don't think the `void` here is actually necessary; seems to work
-- without it.
ffi.cdef [[
int get_magenta(void);
]]
local lib = ffi.load('./main.so')
assert(lib.get_magenta() == 16711935)

The code here is pretty straightforward. First, we require the LuaJIT module ffi, and then we define the C function we’ll be using from main.c using ffi.cdef (if you’re unfamiliar with Lua, the [[ ... ]] syntax just denotes a multi-line string; in this case, it’s not actually necessary, but if we had multiple declarations it’d make things nicer to look at). Then, we load our main.so that we just compiled with gcc, and then from there we can use our C function get_magenta and assert that the number returned for the color is correct.

To run this, open up main.lua with Neovim and then run :luafile %. Assuming everything is set up correctly (LuaJIT installed and Neovim compiled with support for it, which unfortunately I don’t know how to check/do off the top of my head; feel free to leave a comment though if you do), there should be no errors, meaning the assert didn’t fail and everything worked. And just like that, you’ve accessed an internal Neovim function via LuaJIT FFI and C!

The Rust Version

The Rust version of this isn’t that much more complicated, it just takes a bit more setup. First, you’ll want to create a new Rust library project by doing cargo new --lib <project-name>. You can call the project whatever you like; for the purpose of this example, I’ll be using rust-example as the project name. Then, you’ll want to add these lines to your Cargo.toml:

# Cargo.toml

[package]
# ... I'm leaving out lines put here by cargo new --lib
build = "build.rs"

[lib]
crate-type = ["cdylib"]

[dependencies]
# You could probably use a slightly newer/older version, this is just
# what I used.
libc = "0.2"

Next, you’ll want to add the build.rs file in the root of the project you just created (so in the same directory as your Cargo.toml) with these contents:

// build.rs

fn main() {
    // NOTE: Both of these might not be necessary, and there may be a better
    // way of doing this. This is just what I found to work.
    println!("cargo:rustc-cdylib-link-arg=-Wl,-undefined");
    println!("cargo:rustc-cdylib-link-arg=-Wl,dynamic_lookup");
}

The purpose of this is just to tell the Rust linker to ignore the fact that the extern "C" function we’ll define doesn’t exist at compile-time (since we aren’t linking to a library that defines them, they’ll just be available at runtime within the LuaJIT FFI environment in Neovim). From there, put these lines of code in src/lib.rs:

// src/lib.rs

use std::ffi::*;

extern crate libc;

extern "C" {
    fn name_to_color(x: *const libc::c_char) -> i32;
}

#[no_mangle]
pub extern fn get_magenta() -> i32 {
    unsafe {
        name_to_color(CString::new("Magenta").unwrap().into_raw())
    }
}

I think the code here is pretty self-explanatory, so I won’t explain it in detail; we’re basically just doing the same thing as the C code from earlier, but in Rust instead of C.

Now, we can create our new main.lua file in the root of the project (so same directory as Cargo.toml and build.rs):

-- main.lua

local ffi = require'ffi'

-- I don't think the `void` here is actually necessary; seems to work
-- without it.
ffi.cdef [[
int get_magenta(void);
]]

-- The exact string passed to `ffi.load` here will vary depending on 
-- your operating system and what you named your Rust project.
local lib = ffi.load('./target/debug/librust_example.dylib')
assert(lib.get_magenta() == 16711935)

The only real difference here versus the previous main.lua is the path we give to ffi.load.

After that, you can run the lua file using the same command as from before (:luafile % after opening the Lua file in Neovim) and if all goes well, there should be no errors!

Conclusions/Discussion

The idea of accessing Neovim functions directly like this via FFI is quite interesting to me; perhaps it could be used as an alternative to msgpack-rpc or remote plugins, at least for non-garbage-collected languages like Rust and C? It would take some work, but I wouldn’t be surprised if it were possible.

I tried for a bit to do the above with some other internal Neovim functions and the Rust version, and I ran into issues with the C declarations not being defined; it would’ve taken a fair bit of time to define the proper declarations using ffi.cdef, so I stopped. @bfredl had a PR that generated a lot of the C declarations though, and I think with some extra work in the Neovim core like that more interesting things could be created like this. Having an explicit definition of what’s safe to use and what isn’t (or similar) would also be helpful, I would think.

14 Likes

I love how simple this is, considering how it seemed like a ridiculous idea at first (at least to me).

I wonder if there are really significant performance gains to be had by doing this instead of round-tripping data back and forth over a socket. Or if there are additional “hooks” into Neovim internals that would make sense to expose through a C API that wouldn’t make sense to expose over the RPC API.

Obviously would have to do some benchmarks to really see what the difference is (which I might try in the future if I can figure out how to call, say, nvim_eval; although I guess I could do that with what I have here), but my assumption is that it would be faster. With a msgpack-rpc plugin, there are quite a few steps before things are done: first you pass the values you want to send to Neovim into a higher-level function, which then encodes that into msgpack, and then after that sends it to Neovim, which I’m assuming decodes it, and then does what you want. Compare that to this, which calls the functions through FFI, which is pretty direct by comparison: no sending over a socket/process, no encoding/decoding. I can’t imagine that would be slower, at least not in any significant way. But again, I’ll have to see if I can benchmark the two and have actual numbers to compare.

The biggest gain with doing this in my opinion is that, with the right libraries to abstract away the lower-level setup, it would be easier to use (and probably also understand/reason about) from a plugin developer’s perspective. For example, neovim-scorched-earth is a Rust plugin built with neovim-lib, and if you look through the code there’s quite a bit of extra setup that’s necessary to spawn the binary, use rpcnotify to run commands, etc. (see autoload/scorchedEarth.vim specifically). With this, a lot of that could ideally be removed, and the commands could simply be aliases for FFI function calls.

1 Like

nvim --version shows info about lua version too . So that can be used to know if luajit is avilable . At runtime jit lua table can probably be used . Though it probably would have benn better if neovim could only be compiled with luajit . I just find it weird neovim allows to be compiled with any lua version (jit, 5.1, 5.2, 5.3, 5.4) when the versions aren’t even fully compatable with themselves😣

You only get LuaJIT with Lua 5.1 as far as I know. See also “Why Lua 5.1 instead of Lua 5.3+?”

We actually do this in Telescope / Plenary:

We call the neovim C code from Lua :slight_smile:

7 Likes

You only get LuaJIT with Lua 5.1 as far as I know. See also “Why Lua 5.1 instead of Lua 5.3+?”

I know that neovim recomands compileing with luajit . luajit itself supports lua5.1 standered + some 5.2 features . Neovim doesn’t enforce it . It successfuly compiles and runs without luajit . cmake flags -DPREFER_LUA=ON
-DLUA_INCLUDE_DIR= allow neovim to be compiled with any version of lua . Even if it’s compiled with lua5.1 it wonn’t be able to use this FFI as thats luajit only feature . Also being compiled with other version breaks other stuff too . There’s even some small stuff added for 5.2 compatability compat.lua in neovim source code :smiley: . I was talking about that . I think neovim should enforce being compiled with luajit that way plugins wonn’t have to worry about what version of lua user has .

1 Like

We cannot enforce it (as luajit doesn’t build everywhere) but we are actively pushing/collaborating with distro packagers so that luajit enabled versions reach the vast majority of users. A plugin is free to assume luajit.

AFAIK the 5.2/5.3 compat stuff was for the use of a “lua” executable as part of the build process itself, not for in-process code (though it could look that way as the lua stdlib code is shared).

4 Likes

In case anyone is interested gitsigns now uses ffi for running diffs by using neovim’s internal xdiff library. This has enabled a huge uplift in performance for larger files.
Checkout the code here: gitsigns.nvim/diff.tl at main · lewis6991/gitsigns.nvim · GitHub

3 Likes

@lewis6991
Hi I am looking for a function compare 2 line of text and give a different column of 2 line then I can highlight it.
Is there any c or lua function can do that .
I already try your run_diff function but it don’t have column position .