Calling a gRPC Service from NGINX
using OpenResty and Lua FFI
If you are using NGINX, you may be familiar with the ability to run Lua programs for various parts of NGINX ( init_by_lua , log_by_lua, content_by_lua). This is an extremely powerful feature and can be very useful for features like authentication. For this article, lets say that you want to call a gRPC service from NGINX. We are going to figure out if this is possible and, provided things go well, show you how to do it!
Understanding the Use Case
NGINX does not provide support for Lua by default. You will have to install the ngx_http_lua_module for NGINX which will give you access to the functions listed above (and many more). This module is provided by a project named “OpenResty.” While we could manually put NGINX and the Lua module together ourselves, OpenResty provides some tooling to start using their plugin right away. For this article, I will be using the openresty/openresty Docker image.
Now that we have our NGINX server, we will need to craft an NGINX config to call Lua code. They have an easily accessible example located in their “Getting Started” guide. Below is a slightly modified version of the Getting Started configuration:
# omitting header
http {
server {
listen 8989;
location = /api {
content_by_lua_block {
ngx.say("<p>hello, world</p>")
}
}
}
}In the above example, we are able to run a Lua script every time someone send a request to /. Great! Now we just need to find out how to call a gRPC service. (Note: if you are trying to call an HTTP service, that can be done with ngx.location.capture).
A quick google for “Lua gRPC” turns up one promising result: jinq0123/grpc-lua. I noticed that there were several dependencies that would need to be installed and wondered if there might be another way to accomplish the task using only default libraries. Let’s keep this option in the back of our mind if the follow doesn’t work out.
If you had an existing command line client that could talk to a gRPC service, that would be one way to accomplish our task (see os.execute). That is a hack but might get the job done for those that have other concerns. If we want a slightly more native solution, what are our options? Foreign Function Interfaces! While still a bit hacky, LuaFFI provides a way to call C functions from Lua programs.
Modifying the NGINX configuration above, we can write something like the following:
content_by_lua_block {local ffi = require("ffi")
ffi.cdef[[
int printf(const char *fmt, ...);
]]
ngx.say(ffi.C.printf("Hello %s!", "world"))}
Here is the result from cURL:
$ curl http://localhost:8989/api
12Because the return value of printf is the number of bytes printed, this result shouldn’t seem too surprising.
Great, so we are able to craft a working FFI using the standard C Library. Perhaps there is a way to call our gRPC service if we can compile the .proto interface definition to C++? Let’s see!
Writing the gRPC Service
We know that we need to create some sort of shared client library for Lua to use. Searching for “grpc c++,” we can find a few places to start. I will be using the “gRPC Basics” tutorial for C++.
The first thing to do when writing a gRPC service is to define the interface (.proto). Here is the interface I will be using:
syntax = "proto3";package routeprinter;message PrintableRoute {
string route = 1;
int32 status = 2;
}message PrintRouteResponse {
int32 success = 1;
}service RoutePrinter {
rpc PrintRoute(PrintableRoute) returns (PrintRouteResponse) {}
}
Now is when you would normally compile client and server stubs and start filling in your business logic. Inside of the guide, they link to a Makefile that can be used to accomplish this. The Makefile will also compile the client and server into executables. This Makefile defined two tools that are required for generating C++ clients and servers: protoc and grpc_cpp_plugin. The grpc_cpp_plugin doesn’t look like it can be installed via a package manager so we will either have to compile it ourselves or find another distribution strategy. After some searching, I have found grpc/cxx which contains both of the required binaries and will provide a clean and repeatable build environment for us.
After copying the Makefile, I updated most of the names of the files to routeprinter. To make sure I understood all of the dependencies in the build system. After, I implemented the C++ server ( routerprinter_server.cc) and created a short script to run the Docker images and build the sample C++ server ( make clean routerprinter_server).
$ ./generate_cpp_server.sh
rm -f *.o *.pb.cc *.pb.h routeprinter_client routeprinter_server
protoc -I . --cpp_out=. routeprinter.proto
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter.pb.o routeprinter.pb.cc
protoc -I . --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` routeprinter.proto
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter.grpc.pb.o routeprinter.grpc.pb.cc
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter_server.o routeprinter_server.cc
g++ routeprinter.pb.o routeprinter.grpc.pb.o routeprinter_server.o -L/usr/local/lib `pkg-config --libs protobuf grpc++` -Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed -ldl -o routeprinter_serverAfter compilation, you should be able to run the server inside of the container:
Server listening on 0.0.0.0:50051Here are the shared objects our server executable depends on (when run outside of the container). You can see that the linker is not able to resolve libgrpc++ and libprotobuf shared object files.
# formatted, a bit
$ ldd routeprinter_server
linux-vdso.so.1 (0x00007ffdf1fc9000)
libprotobuf.so.15 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00...)
libgrpc++.so.1 => not found
libgrpc++_reflection.so.1 => not found
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00.)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00...)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000...)
/lib64/ld-linux-x86-64.so.2 (0x00007f2c...)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2...)After writing the server, you should see that you are only overriding functions that are defined in the other object files generated by protoc and the grpc_cpp_plugin. Perhaps we can use one of the intermediate object files to have access to functions that call our gRPC service? Looking at the Lua FFI API, we can see that ffi.load(...) is how we can load arbitrary shared object files.
We have several options at where to place this shared object file. Looking at the above ldd output, /lib/x86_64-linux-gnu/... seems like a decent option. Running ldd inside of the container shows that libprotobuf and libgrpc++ end up in /usr/local/lib/.... Looking at the created object files, I tried using routeprinter.grpc.pb.o. After trying both of these directories with a bunch of permutations of the name given to ffi.load("..."), I took some time to reflect.
I had seen a previous example of OpenResty using cjson. Maybe this could help out? Looking for cjson inside of the OpenResty docker container, I found out that it is a shared object library located in /usr/local/openresty/lualib/cjson.so. This is great! I can use this to compare and contrast against our intermediate object file. After continuing to run permutations of names, I tried loading cjson with ffi.load("cjson.so") which failed. Hmm. At this point, I found that using the full path to cjson.so was able to succeed. Switching the object file to routeprinter.grpc.pb.o resulted in a new error! (note that I am bind mounting in this file as routeprinter.so)
$ ./start_nginx.sh
2019/05/19 16:40:39 [error] 1#1: init_by_lua error: init_by_lua:7: libprotobuf.so.15: cannot open shared object file: No such file or directory
stack traceback:
[C]: in function 'load'
init_by_lua:7: in main chunkAha! We need our shared obejcts from grpc/cxx inside of openresty/openresty. This can be solved with Docker multistage build and copying all of the grpc libraries over to the directory with our system libraries:
FROM grpc/cxx:1.12.0 as grpclibs
FROM openresty/openresty:1.13.6.2-xenial
COPY --from=grpclibs /usr/local/lib/* /lib/x86_64-linux-gnu/After switching our NGINX image to this customer one, we get the following error:
2019/05/19 16:23:52 [error] 1#1: init_by_lua error: init_by_lua:7: /usr/local/openresty/lualib/routeprinter.so: only ET_DYN and ET_EXEC can be loaded
stack traceback:
[C]: in function 'load'
init_by_lua:7: in main chunkGoogling further, this is because our Makefile has not compiled these files as either a shared object file or an executable. I am already compiling my server so I figured I would give that a try. And… it crashes. 😱 OK so maybe we do need to change the way we compile our application. This SO post lists using the -shared option. Adding this option to LDFLAGS, we are now left we a new error when compiling:
/usr/bin/ld: routeprinter.pb.o: relocation R_X86_64_PC32 against symbol `_ZN12routeprinter33_PrintableRoute_default_instance_E' can not be used when making a shared object; recompile with -fPICso we add CXXFLAGS += -fPIC to our Makefile and NGINX starts fine and returns the same results as the above cURL! Hooray!
Understanding Lua’s FFI
So far, we were able to compile our server with the -fPIC option, enable -shared when linking, and load our server executable in our NGINX configuration file without crashing anything. Time to call some functions!
First, let’s trying a call a function that hopefully doesn’t exist to see what sort of error we get. Adding
grpc_rp = ffi.load("...")
grpc_rp.Hello()results in:
2019/05/19 16:49:12 [error] 5#5: *1 lua entry thread aborted: runtime error: content_by_lua(default.conf:50):19: missing declaration for symbol 'Hello'Referencing the FFI Tutorial Page (the zlib example is helpful), we can see that we need to define the C functions we would like to call in the ffi.cdef(...) function call. So which functions would we like to call in our executable? Is there a way to list all of the available functions? Possibly via reflection?
I explored the reflection method using this SO post but that didn’t end up helping. Lua is only going to add functions to our object if we tell it to! I have written the void RunServer() function in routeprinter_server.cc so perhaps we can use that (at this point, I just want to get FFI working, not that I actually want to run my server here). After adding this function to ffi.cdef(...), I received the following:
/usr/local/openresty/lualib/routeprinter.so: undefined symbol: RunServerYikes! So we think a bit harder and the memories of our College class comes back and we remember our professor’s talk about linking. Unfortunately, we only remember the one tacky phrase out of it all: name mangling. Because we are compiling C++ and not C, the symbols inside of our shared object file aren’t necessarily the symbols we use in C++ code. objdump -S routerprinter_server is a great way to inspect these symbols. Moreover, we can examine the cjson.so binary to reach our conclusion.
root@5f2c1638e7a9:/usr/local/openresty/lualib# objdump -S cjson.so | grep -n json_decode
3412:0000000000004fd0 <json_decode>:
3424: 4ff3: 74 14 je 5009 <json_decode+0x39>
3434: 501c: 0f 84 39 03 00 00 je 535b <json_decode+0x38b>
3446: 5051: 76 13 jbe 5066 <json_decode+0x96>
...We can see multiple places in cjson.so where parts of the code call json_decode. Line 3412 (and what would have been the following lines) contains the declaration of the function. If we were to use json_decode in ffj.cdef(...), this is the function that would be called. We can look at our own binary for RunServer and see what options are available.
# objdump -S routeprinter.so | grep -n RunServer
3093:0000000000024890 <_Z9RunServerv@plt>:
20058:00000000000315fe <_Z9RunServerv>:
20136: 31748: eb 7e jmp 317c8 <_Z9RunServerv+0x1ca>
20148: 31773: eb 17 jmp 3178c <_Z9RunServerv+0x18e>
20153: 31787: eb 03 jmp 3178c <_Z9RunServerv+0x18e>
20158: 3179b: eb 03 jmp 317a0 <_Z9RunServerv+0x1a2>
20163: 317ac: eb 03 jmp 317b1 <_Z9RunServerv+0x1b3>
20182: 317e1: e8 aa 30 ff ff callq 24890 <_Z9RunServerv@plt>And we have found the “mangled” name _Z9RunerServerv! Summing it up, we are left with the following:
ffi.cdef[[
int printf(const char *fmt, ...);
void _Z9RunServerv();
]]
print("before run server")
grpc_rp["_Z9RunServerv"]()
ngx.say(ffi.C.printf("Hello %s!", "world"))And voila!
... [lua] content_by_lua(default.conf:51):24: before run server, client: 172.18.0.1, server: localhost, request: "GET /api HTTP/1.1", host: "localhost:8989"
Server listening on 0.0.0.0:50051We’ve managed to call C++ from Lua! 😹
The Other Way
So we would never write something like this in production but we have found that it is theoretically possible to run generated C++ code from Lua (our server’s RunServer used the code generated by gRPC C++). This definitely begs the question, how did grpc_lua end up implementing something like this?
Following the API, the first call is grpc.import_proto_file(...). This calls into the jinq0123/luapbintf package’s import_proto_file function. Searching for ImportProtoFile in that repository, we find that this function call to an Importer class that appears to be imported from the C++ protobuf library. I am not sure where the Importer is used later but it is likely given to some gRPC library for stub generation.
Eventually, you end up making a call by creating a service_stub and calling sync_request. These are defined in jinq0123/grpc_lua. This package contains two important directories, src/lua and src/cpp. The lua directory contains some high level Lua APIs for clients to use. The cpp directory also contains Lua API definitions but declared in C++, not in Lua. They also use the extern "C" keyword to avoid the mangling part that we hit in the above section. Inside of the C++ functions, there are several calls to another library that actually makes the network calls: jinq0123/rpc_cb_core. Take a look at ServiceStub::SyncRequest and Channel::MakeSharedCall (which calls to grpc_channel_create_call).
Wrapping It Up
It is pretty clear that the grpc_lua library will work with any .proto file and the code in between just seems “easy” enough to navigate if you are worried about performance concerns. It its probably the best solution for something that needs to be written today and for those that might not want to mess with C++ compilation for their NGINX setup.
I may investigate using extern C in the future though I would imagine most of the surrounding code would end up duplicating a lot of the grpc_lua library anyway. Thank you for reading!
Code Located on GitHub:
