Calling a gRPC Service from NGINX

using OpenResty and Lua FFI

https://pixabay.com/photos/water-pipe-plumbing-pipeline-2852047/

Understanding the Use Case

NGINX does not provide support for Lua by default. You will have to install the ngx_http_lua_module for NGINX which will give you access to the functions listed above (and many more). This module is provided by a project named “OpenResty.” While we could manually put NGINX and the Lua module together ourselves, OpenResty provides some tooling to start using their plugin right away. For this article, I will be using the openresty/openresty Docker image.

# omitting header
http {
server {
listen 8989;
location = /api {
content_by_lua_block {
ngx.say("<p>hello, world</p>")
}
}
}
}
content_by_lua_block {local ffi = require("ffi")
ffi.cdef[[
int printf(const char *fmt, ...);
]]
ngx.say(ffi.C.printf("Hello %s!", "world"))
}
$ curl http://localhost:8989/api
12

Writing the gRPC Service

We know that we need to create some sort of shared client library for Lua to use. Searching for “grpc c++,” we can find a few places to start. I will be using the “gRPC Basics” tutorial for C++.

syntax = "proto3";package routeprinter;message PrintableRoute {
string route = 1;
int32 status = 2;
}
message PrintRouteResponse {
int32 success = 1;
}
service RoutePrinter {
rpc PrintRoute(PrintableRoute) returns (PrintRouteResponse) {}
}
$ ./generate_cpp_server.sh
rm -f *.o *.pb.cc *.pb.h routeprinter_client routeprinter_server
protoc -I . --cpp_out=. routeprinter.proto
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter.pb.o routeprinter.pb.cc
protoc -I . --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` routeprinter.proto
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter.grpc.pb.o routeprinter.grpc.pb.cc
g++ -std=c++11 `pkg-config --cflags protobuf grpc` -c -o routeprinter_server.o routeprinter_server.cc
g++ routeprinter.pb.o routeprinter.grpc.pb.o routeprinter_server.o -L/usr/local/lib `pkg-config --libs protobuf grpc++` -Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed -ldl -o routeprinter_server
Server listening on 0.0.0.0:50051
# formatted, a bit
$ ldd routeprinter_server
linux-vdso.so.1 (0x00007ffdf1fc9000)
libprotobuf.so.15 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00...)
libgrpc++.so.1 => not found
libgrpc++_reflection.so.1 => not found
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00.)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00...)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000...)
/lib64/ld-linux-x86-64.so.2 (0x00007f2c...)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2...)
$ ./start_nginx.sh
2019/05/19 16:40:39 [error] 1#1: init_by_lua error: init_by_lua:7: libprotobuf.so.15: cannot open shared object file: No such file or directory
stack traceback:
[C]: in function 'load'
init_by_lua:7: in main chunk
FROM grpc/cxx:1.12.0 as grpclibs
FROM openresty/openresty:1.13.6.2-xenial
COPY --from=grpclibs /usr/local/lib/* /lib/x86_64-linux-gnu/
2019/05/19 16:23:52 [error] 1#1: init_by_lua error: init_by_lua:7: /usr/local/openresty/lualib/routeprinter.so: only ET_DYN and ET_EXEC can be loaded
stack traceback:
[C]: in function 'load'
init_by_lua:7: in main chunk
/usr/bin/ld: routeprinter.pb.o: relocation R_X86_64_PC32 against symbol `_ZN12routeprinter33_PrintableRoute_default_instance_E' can not be used when making a shared object; recompile with -fPIC

Understanding Lua’s FFI

So far, we were able to compile our server with the -fPIC option, enable -shared when linking, and load our server executable in our NGINX configuration file without crashing anything. Time to call some functions!

grpc_rp = ffi.load("...")
grpc_rp.Hello()
2019/05/19 16:49:12 [error] 5#5: *1 lua entry thread aborted: runtime error: content_by_lua(default.conf:50):19: missing declaration for symbol 'Hello'
/usr/local/openresty/lualib/routeprinter.so: undefined symbol: RunServer
root@5f2c1638e7a9:/usr/local/openresty/lualib# objdump -S cjson.so | grep -n json_decode
3412:0000000000004fd0 <json_decode>:
3424: 4ff3: 74 14 je 5009 <json_decode+0x39>
3434: 501c: 0f 84 39 03 00 00 je 535b <json_decode+0x38b>
3446: 5051: 76 13 jbe 5066 <json_decode+0x96>
...
# objdump -S routeprinter.so | grep -n RunServer
3093:0000000000024890 <_Z9RunServerv@plt>:
20058:00000000000315fe <_Z9RunServerv>:
20136: 31748: eb 7e jmp 317c8 <_Z9RunServerv+0x1ca>
20148: 31773: eb 17 jmp 3178c <_Z9RunServerv+0x18e>
20153: 31787: eb 03 jmp 3178c <_Z9RunServerv+0x18e>
20158: 3179b: eb 03 jmp 317a0 <_Z9RunServerv+0x1a2>
20163: 317ac: eb 03 jmp 317b1 <_Z9RunServerv+0x1b3>
20182: 317e1: e8 aa 30 ff ff callq 24890 <_Z9RunServerv@plt>
ffi.cdef[[
int printf(const char *fmt, ...);
void _Z9RunServerv();
]]
print("before run server")
grpc_rp["_Z9RunServerv"]()
ngx.say(ffi.C.printf("Hello %s!", "world"))
... [lua] content_by_lua(default.conf:51):24: before run server, client: 172.18.0.1, server: localhost, request: "GET /api HTTP/1.1", host: "localhost:8989"
Server listening on 0.0.0.0:50051

The Other Way

So we would never write something like this in production but we have found that it is theoretically possible to run generated C++ code from Lua (our server’s RunServer used the code generated by gRPC C++). This definitely begs the question, how did grpc_lua end up implementing something like this?

Wrapping It Up

It is pretty clear that the grpc_lua library will work with any .proto file and the code in between just seems “easy” enough to navigate if you are worried about performance concerns. It its probably the best solution for something that needs to be written today and for those that might not want to mess with C++ compilation for their NGINX setup.