OpenResty XRay™ YLua User Manual

ylua - Compiler for the ylua language for writing Lua apps’ tracing tools in a Lua-like language.

# Table of Contents

# Synopsis

# Language syntax examples

probe foo(a, b)
    print("a = ", a)
    print("b = ", b)
end

probe bar()
    print("k = ", k)  -- k is an upvalue in the target for function bar.
end

Back to TOC

# Language features

# Lua syntax supported

The following Lua operators and syntax are supported:

• unary #
• unary not
• unary -
• binary +, -, *, /, %, ..
• binary >, >=, <, <=, ==, !=
• binary and, or.
• postfix [KEY]
• postfix .KEY
• parentheses (())
• vararg (...)
• booleans (true, false)
• if/elseif/else statements
• return statement
• do statement
• for v = fr, to, step do loop statement
• for i, e in ipairs(tb) loop statement
• for k, v in pairs(tb) loop statement
• while statement
• repeat ... until statement
• break statement
• local variable declarations (with multiple variables and initializer expressions)
• assignment statement (with multiple assignment support)
• nil value
• function call syntax foo(...), foo"...", foo{...}, and etc.

Back to TOC

# Built-in functions supported

# Lua built-in functions

The following Lua built-in functions are supported:

• require
• print
• type
• select
• error
• tostring
• tonumber
• ipairs (only as the exp-list in the for ... in loop)
• pairs (only as the exp-list in the for ... in loop)
• table.maxn

Back to TOC

# Extra built-in functions

The following extra standard functions are supported:

• dump

  Dump details of the Lua value argument.

• out

  Similar to print(), but do not output a trailing newline character automatically

• identity

  Directly return the argument value.

• exit

  Exit the probing session (i.e., the whole tracing tool).

• warn

  Emit a warning message to stderr.

• contains

  Returns true if the 1st string argument contains the 2nd string argument, or false otherwise.

• upval

  syntax: upval(name)

  syntax: upval(func, name)

  Returns the upvalue of the name specified by the name argument. The associated function can be either the current function being probed or the one explicitly specified by the func argument. Only Lua functions take named upvalues. This upval() function can also be used in Lua function probe specifier, as in

  probe (upval(require "foo".blah, "uvname")) (a, b)
      ...
  end
  

Back to TOC

# Probes

# Lua function probes

# Lua function entry probes

Dynamic probes on Lua function entry points are supported. Certain Lua built-in functions implemented directly in hand-crafted assembly code might miss the probes, however, as some of the API functions under the math namespace.

Below is an example:

probe foo(a)
    print("arg a = ", a)
end

It prints out the value of the actual argument a of the global Lua function foo in the target every time that function is entered (due to a function call).

Vararg syntax is also supported when the number of actual Lua function arguments in the target process may change. For example:

probe foo(...)
    for i = 1, select('#', ...) do
        print(i, ": ", select(i, ...))
    end
end

This probe handler will output all the actual arguments in every invocation of the global Lua function foo in the target. One typical output might look like this:

1: 3
2: hello
3: 3.140000

In case the target Lua function might return composite Lua values like Lua tables, we should use the dump() builtin function instead in this example, as in

probe foo(...)
    for i = 1, select('#', ...) do
        print(i, ": ", dump(select(i, ...)))
    end
end

One sample output is like below:

1: 171
2: "hello"
3: true
4: table (GCtab*)0x7f96f754d830 (narr=0, nrec=1):
 key:
  "dogs"
 value:
  -3.140000

Note how the output of dump() differ from printing out the Lua target values directly.

For Lua function entry probes, the parameter variables, upvalue variables, and global variables can all be referenced. Local variables which are not parameters cannot be referenced in such probe handlers since they are not initialized at that point, naturally. Below is an example:

probe foo(a)
    print(a + b)
end

Compiling this ylua code example will yield the following warning:

WARNING: bar: symbol 'b' is assumed to be an upvalue or a global variable in the target Lua process at test.ylua line 2

Obviously, the referenced variable b is not declared and will be assumed to be either an upvalue for the function foo in the target, or a global Lua variable of the current Lua thread when the probe is fired. An upvalue of the current Lua function take precedence over a global variable lookup, for obvious reasons.

Any Lua primary expressions can be used as probe specifier, as in

probe (package.loaded["io"].open)(file_name, mode)
    print("opening file ", file_name, " with mode ", mode)
end

Or even with tracer-space built-in function calls, as in

probe (require "io".open)(file_name, mode)
    print("opening file ", file_name, " with mode ", mode)
end

Note that the parentheses are required to enclose the expression otherwise there is grammatical ambiguities.

Back to TOC

# Lua function return probes

Dynamic probes on Lua function return points are supported. Tailcalls might miss our probes, however, due to their inherent “goto” nature.

Consider the following example:

probe foo -> ()
    print("function foo returning!")
end

This probe handler will prints out the line function foo returning! every time the global Lua function foo() is returned in the target. Here we do not care whether such function exits return any values. But when we do, we can inspect the return values like this:

probe foo -> (a, b)
    print("returning ", a, " and ", b)
end

This probe will output the first 2 return values of every return of the target Lua function.

Similar to the Lua function entry probes, the Lua vararg syntax (...) is also supported in return probes to inspect all the return values when the number of them are not fixed or not known ahead. Below is such an example:

probe foo -> (...)
    for i = 1, select('#', ...) do
        print(i, ": ", select(i, ...))
    end
end

One typical output is like this:

1: 171
2: hello
3: true
4: -3.140000

We should use the dump() built-in function in case the return values might be composite Lua values like Lua tables, as in

probe foo -> (...)
    for i = 1, select('#', ...) do
        print(i, ": ", dump(select(i, ...)))
    end
end

For Lua function return probes, all local variables visible to the corresponding Lua function return point, upvalues, and global variables can all be accessed. Note that the Lua VM might optimize certain local variables away if the return point does not really reference those local variables.

Any Lua primary expressions can be used as probe specifier, as in

probe (package.loaded["io"].open) -> (file_handle, err)
    if file_handle ~= nil then
        print("opened file as handle ", file_handle)
    else
        print("failed to open file: ", err)
    end
end

Or even with tracer-space built-in function calls, as in

probe (require "io".open) -> (file_handle, err)
    if file_handle ~= nil then
        print("opened file as handle ", file_handle)
    else
        print("failed to open file: ", err)
    end
end

Note that the parentheses are required to enclose the expression otherwise there is grammatical ambiguities.

Back to TOC

# C function probes

# C function entry probes

To probe on a C function entry point, we can just write something like below:

probe C:lj_cf_collectgarbage()
    print("foo: ", package.loaded.foo)
end

Here we probe onto the entry point of the C function lj_cf_collectgarbage and then print out the value of the Lua expression package.loaded.foo.

Back to TOC

# Standard probes

# begin

Identical to ylang’s _begin probe.

Back to TOC

# end

Identical to ylang’s _end probe.

Back to TOC

# process.begin

Identical to ylang’s _process.begin probe.

Back to TOC

# timer.profile

Identical to ylang’s _timer.profile probe.

Back to TOC

# timer.s(N)

Identical to ylang’s _timer.s(N) probe.

# timer.ms(N)

Identical to ylang’s _timer.ms(N) probe.

# Type system

The ylua language has the following value types:

• tv: for storing Lua values in the target process (or “TValue” pointers).
• num: for storing double-precision numbers in the tracer space.
• int: for storing 32-bit signed integers in the tracer space.
• bool: for storing boolean values true and false in the tracer space.
• str: for storing strings in the tracer space.
• nil: for the nil value (or void) in the tracer space.
• func: for the built-in or user-defined ylua functions in the trace space.

A ylua variable in the tracer space can take all the data types above except the nil type. When a ylua variable is initialized with nil, it is of the tv type. A tv typed variable can also be assigned to by a nil-typed expression.

When a function returns no value (like the built-in function print()), it takes the return value type of nil.

Every ylua variable can only take data type and its type must be determined at compile time and must not change at run time.

All the ylua variables for storing values in the target process space must be of the type tv. And naturally, all values coming from the target process take the type tv.

Back to TOC

# Embedding ylang source snippet

It is possible to embed arbitrary ylang source code snippets using the following syntax:

ylang [=[
    _probe _begin {
        use_ngx_stream_lua_module = true;
        _warn("Start tracing...\n");
    }
]=]

Basically the embedded ylang code is put into a Lua string literal (to save the trouble of escaping, long-bracketed string literals are preferred here).

Instead of using ylang to embed top-level ylang code snippets, it is also supported to embed ylang code inside any other contexts like inside a Lua function entry probe handler:

probe foo()
    print("foo called!")
    ylang [=[
        printf("from ylang...\n");
    ]=]
end

Back to TOC

# TODO

• Make the for k, v in pairs(tb) loop iterate through the array part of the table as well.
• Operator % should work on floating-point numbers directly.
• The and and or operator expressions should not return boolean values only.
• Support conversion from number-like string values to numbers.
• Lua’s string.format builtin function.
• Lua’s string.find builtin function.
• Lua’s string.byte builtin function.
• Lua’s string.char builtin function.
• Lua’s string.sub builtin function.
• Lua’s table.concat builtin function.
• Lua’s collectgarbage("count") builtin function.
• Lua’s math.* API functions.
• Lua’s coroutine.status() API function.
• Lua’s getmetatable() API function.
• Lua’s getfenv() API function.
• LuaJIT’s bit.* API functions.
• LuaJIT’s ffi API for manipulating cdata’s C data structures and C types, like cdata.field[index].
• New table.narr() builtin function.
• New table.nrec() builtin function.
• New dump_bt() and dump_full_bt() builtin functions for dumping Lua backtraces. Add support for debug.traceback() too, which is just aliased to dump_bt().
• New agg data type for aggregate variables, the <<< operator, as well as corresponding stats builtin functions like agg.count(), agg.avg(), agg.max() agg.min(), agg.sum(), agg.hist_log(), agg.hist_linear(), and etc.
• Support defining user functions in the tracer space (just like defining new Lua functions).
• Support probing on cdata functions like ffi.C.foo().
• New _Continue keyword and statement, similar to C’s continue statement for loops.
• Add ylang-style built-in hash table variables and array variables.
• Add built-in probes like begin, end, syscall.*, and etc.

Back to TOC

# Author

Yichun Zhang (agentzh) yichun@openresty.com

Back to TOC

# Copyright & Licenses

Copyright (C) 2018-2021 OpenResty Inc. All rights reserved.

This software is proprietary and must not be redistributed or shared at all.

Back to TOC