Important News: 26/10/2016 - Call for Presentations to Lua Devroom at FOSDEM 2017
Important News: 04/05/2016 - Community news #2
Important News: 11/12/2015 - Blog opening and contribution guide

Build a Standalone Executable

By Eric R. Schulz Mar 28 2016 12:15 Tools Comments

luastatic is a command line tool that builds a standalone executable from a Lua program. The executable runs on systems that do not have Lua installed because Lua is embedded alongside the program. Lua 5.1, 5.2, 5.3 and LuaJIT are supported. luastatic can be downloaded from GitHub or LuaRocks.

Lua is commonly used as an embeddable scripting language as part of a larger program written in another language. However, Lua can also be the primary language used to implement a program. Programmers using C are accustomed to building an application to a single executable that is distributed to the end user. luastatic does the same for Lua programs.

Below is a description of the arguments luastatic supports:

luastatic main.lua[1] require.lua[2] liblua.a[3] module.a[4] -Iinclude/lua[5] [6]
[1]: The entry point to the Lua program
[2]: One or more required Lua source files
[3]: The Lua interpreter static library
[4]: One or more static libraries for a required Lua binary module
[5]: The path to the directory containing lua.h
[6]: Additional arguments are passed to the C compiler

The shell script below shows how to use luastatic to build this program for GNU/Linux and Windows. The program uses Lua, LuaSQLite3, and SQLite3. I tested the script on Ubuntu 15.04.

#!/bin/sh

# download build tools (if necessary)
sudo apt-get install build-essential make mingw-w64 unzip libreadline-dev

# download program dependencies
wget https://www.lua.org/ftp/lua-5.2.4.tar.gz
wget https://raw.githubusercontent.com/ers35/luastatic/c810584/luastatic.lua
wget http://sqlite.org/2016/sqlite-amalgamation-3110100.zip
wget http://lua.sqlite.org/index.cgi/zip/lsqlite3_fsl09w.zip
wget http://lua.sqlite.org/index.cgi/raw/examples/simple.lua?name=0c08de88e066ef2d6cf59c4be3d7ce2aa7df32c9 -O simple.lua

# extract dependencies
tar -xf lua-5.2.4.tar.gz
unzip sqlite-amalgamation-3110100.zip
unzip lsqlite3_fsl09w.zip

# create build directories
mkdir linux windows
cp simple.lua linux
cp simple.lua windows

# build Lua for GNU/Linux
cd lua-5.2.4
make linux
mv src/lua ../
mv src/liblua.a ../linux
make clean
# build Lua for Windows
make mingw CC=x86_64-w64-mingw32-gcc
mv src/liblua.a ../windows
cd ../

# build luastatic using itself
./lua luastatic.lua luastatic.lua linux/liblua.a -Ilua-5.2.4/src
cp ./luastatic linux
cp ./luastatic windows

# build SQLite3 for GNU/Linux
cd sqlite-amalgamation-3110100
cc -c -O2 sqlite3.c -o sqlite3.o
ar rcs ../linux/sqlite3.a sqlite3.o
# build SQLite3 for Windows
x86_64-w64-mingw32-gcc -c -O2 sqlite3.c -o sqlite3.o
x86_64-w64-mingw32-ar rcs ../windows/sqlite3.a sqlite3.o
cd ../

# build LuaSQLite3 for GNU/Linux
cd lsqlite3_fsl09w
cc -c -O2 lsqlite3.c -I../sqlite-amalgamation-3110100 -I../lua-5.2.4/src -o lsqlite3.o
ar rcs ../linux/lsqlite3.a lsqlite3.o
# build LuaSQLite3 for Windows
x86_64-w64-mingw32-gcc -c -O2 lsqlite3.c -I../sqlite-amalgamation-3110100 -I../lua-5.2.4/src -o lsqlite3.o
x86_64-w64-mingw32-ar rcs ../windows/lsqlite3.a lsqlite3.o
cd ../

# build simple.lua for GNU/Linux
cd linux
./luastatic simple.lua liblua.a lsqlite3.a sqlite3.a -I../lua-5.2.4/src -lpthread
strip simple
cd ../

# build simple.lua for Windows
cd windows
CC=x86_64-w64-mingw32-gcc ./luastatic simple.lua liblua.a lsqlite3.a sqlite3.a -I../lua-5.2.4/src -lpthread
strip simple.exe

luastatic generates the C source file simple.lua.c containing the Lua program and runs the following command to build the executable:

cc -Os simple.lua.c lsqlite3.a liblua.a sqlite3.a -rdynamic -lm -ldl -I../lua-5.2.4/src -lpthread -o simple

If you are familiar with C you can read the file simple.lua.c to see the generated calls to the Lua C API.

If you find a program that luastatic does not build or where the resulting executable does not behave the same as when run with Lua, please report the issue or email eric@ers35.com with enough detail to reproduce the build.


Making music in Lua

By Elihu Garret Feb 18 2016 09:36 Art Comments

Writing code as an artistic practice

Live coding is a hard thing to define but essentially it is about writing code to generate improvised music and/or visuals. Frequently, all the code manipulation is projected or streamed. There is a big, diverse, friendly and a well organized community around this artistic movement. Also, it is inclusive and accessible to all, a lot of live coding environments can be used for free or you can write your own. Popular live coding languages includes: SuperCollider, Sonic Pi, Fluxus, ixi lang, Tidal, Gibber, etc.

Using Lua in a musical context

Why use Lua for live coding if there are cooler and more developed frameworks? Well, that was the first question that I had to answer when I decided to write a musical live coding environment in Lua, now named Moonlet. It is not the first Lua live coding framework, maybe the best known is LuaAV but the project seems dead. There is also Praxis and Worp, but you have to compile a lot of things and I hate to compile a lot of things, in this case Moonlet is portable (only for Windows users). Writing your own environment is like making your own musical instrument and Lua lets me to express easily (with a few lines of code) ideas and structures needed to create the music that I want. I will explain that in future posts entries and how developing some musical concepts in Lua is easier and more readable compared with other languages.

Experiments

I made a few videos showing how Moonlet behaves with diverse Lua libraries.

String interpolation experiment

Lua is a minimal programming language, and many "popular" features are not included in it. But I think that the "less is more" Lua philosophy is great, handling with a restricted environment makes you more creative as everything in the language is well designed and not just a pile of features. When I found the f-strings module made by Hisham Muhammad that implements python-like string interpolation in Lua, I decided to make something with it:



In the video basically Moonlet sends MIDI messages in real-time to an external synth, the script is updated when saved. Moonlet can read numerical notes (61) and traditional notation (c#4). Note patterns are represented as long strings, then the method "string:n()" converts it to a table. Representing note events as strings let me use the f-strings library to manipulate the pitch of each note individually.

-- numbers are notes and underscores represents silence
local a = ( F[[
    60 _ 72 {67 + 12} _ _ 63 63
    67 _ _ 63 _ 77 65 74
    60 _ _ 67 _ {63 + 7} 79 60
    _ _ 65 _ _ 60 _ 67
    ]] ):n()

S-Lua experiment

I love functional programming languages and, obviously, parentheses. Lua has the coolest functional features and the LuaJIT ffi library lets you write and call C stuff within Lua code. If you think this is really great just hold on and check this. Yes, s-expressions for Lua, you must also read this: Experimental Meta-Programming for Lua by Max Rottenkolber. In this experiment notes and sound samples are represented as s-expressions:



This snippet of code is a lispy step sequencer:

-- the semantic of the language does not change.
-- is like writing in Lua a,b,c,d,e = " ", " ", " ", " ", " "

-- letters are sound samples 
-- "x" -> kick drum

sLua[[
    ($ (a b c d e) 
        !(c3 d#4 g4) 
        !(c4 d4 g4) 
        !(x _ _ _ x _ _ _) 
        !(_ h _ h p _ _ G) 
        !(_ d#5 _ c5 f4 g4 g4 c3 ))
]]

Conclusions

There are a lot of things that I left out about Moonlet, like how it works or how to install it. I will write about that in the near future. The purpose of this post is to show that Lua is fully capable for music handling in a live coding context. Here is an example of a complete track written entirely in Lua and recorded live:



For this piece I live-coded hardware synths with Moonlet using MIDI protocols. You can listen more of my tracks in soundcloud. Feel free to make whatever you want with them, they are open source. But... where is the source? Well, I am making/developing a pretty little album and it will be released with the source code.


Continuous Integration with Lua

By Enrique García Cota Feb 16 2016 17:40 General Reblogged Comments

I recently gave a talk in FOSDEM, which is a great event if, like me, you are into open source.

Direct link to slides

You can watch the whole talk in video format. If you prefer a text-based approach, read on.

If you want to know what Continuous Integration is, or why it is useful, you could start by reading the article I wrote about that.

I manage several open source long-lived Lua projects. I have continuous integration in almost all of them, and it has been of great help. What I am about to explain works for my kind of projects, but it can be adapted to other options.

a) The environment

Continous Integration takes place in a server. Since we're talking about Lua here, the server must have a way to install some version of Lua, and some version of Luarocks. This is what I call "the environment". I want to test my libraries in Lua 5.1, 5.2 & 5.3, as well as LuaJIT 2.1 & 2.2. So I need my CI server to install all that.

I choose travis-ci as my main CI server. It is free for open-source tools, and it integrates very well with github. It is also very easy to configure: you just need to add a file called .travis.yml on the root of your repo, and travis will use it to run all the checks.

Travis supports a wide variety of languages. Setting up the environment for the supported languages is incredibly easy. For example, the following .travis.yml file will install 4 versions of ruby:

# .travis.yaml
language: ruby

rvm:
  - '1.9.3'
  - '1.8.7'
  - 'rbx-2.1.1'
  - 'ree'

Lua, unfortunately, is not one of the supported languages, which means it requires more work. I have seen lots of people going this route:

# .travis.yaml
language: C # does not really matter

env:
  - LUA=lua5.1
  - LUA=lua5.2
  - LUA=lua5.3
  - LUA=luajit2.0
  - LUA=luajit2.1

before_install:
  - source install_lua.sh

What the env section of that file does is setting up environment variables. It is basically saing: run one time with the LUA environment variable equals to "lua5.1", another time with "lua5.2", etc. In travis, each of these "times" it called a job. The before_install step, which is repeated at the beginning of every job, executes a script file called install_lua.sh, which in turn will install the appropiate version of Lua depending on the value of the LUA environment variable.

This solution works. But there are several things I don't like. install_lua.sh is usually a big file written in shellscript, usually several hundreds lines of code, which require maintenance. It also must be included on each repo which uses this approach for CI.

Fortunately, a better solution was presented to me. And it came from an unlikely place: Python!

There is this Python project called hererocks. Hererocks is a python script which can install any of the major versions of Lua and LuaJIT in a folder, with no "global" dependencies. It's maintained by Peter Melnichenko, and since Python is supported by travis, we can set up the environment like this:

# .travis.yaml
language: python
sudo: false

env:
  - LUA="lua=5.1"
  - LUA="lua=5.2"
  - LUA="lua=5.3"
  - LUA="luajit=2.0"
  - LUA="luajit=2.1"

before_install:
  - pip install hererocks
  - hererocks lua_install -r^ --$LUA
  - export PATH=$PATH:$PWD/lua_install/bin

What this does is: since Python is supported by travis, we can set it up in the language section. This makes Python's package manager, pip, available. Then we use pip to install hererocks. And then we use hererocks to install the version of Lua which is required for each job in a folder called lua_install.

b) Specs

As I said before, specs are "extra code which makes sure that your regular code works as you think it should". A usual term to refer to them is automated tests. I prefer specs because it is a bit more specific (there are lots of "tests" which can be "automated").

To implement my specs I use the library busted, by olivinelabs.

A quick taste of how specs look like in busted. Here's an extremely simple Lua library:

-- mylib.lua

return {
  add = function(a,b) return a+b end
}

And here's how we could write the spec for that library in busted:

-- spec/mylib_spec.lua

local mylib = require 'mylib'

describe('mylib', function()
  it('adds numbers', function()
    assert.equal(5, mylib.add(2,3))
  end)
end)

busted declares several global variables: describe makes groups of specs. it creates one spec. assert is monkeypatched so in addition of its usual role it can be used to specify assertions. When these assertions are met, the specs containing them are passed (green). If any assertion on a spec "fails", the spec becomes "failed" (red).

Here's how you install & execute busted in travis.yml:

# .travis.yml

install:
  - luarocks install busted

script:
  - busted --verbose

The --verbose option gives a little more information when a spec fails.

Just by adding this to your .travis.yml, you should be able to "run your specs" through Travis. It integrates with github, and will give you some niceties if you use it as a platform: it will run the specs automatically every time anyone pushes code to any of the branches, or sends a pull request. It will also mark all the commits and pull requests with a green flag (when the specs pass) or a red cross (when they don't).

Travis Github integration

c) Coverage

Now that you have code which tests that your code does what you think it should do. What else is there?

One of the things that you can do is getting the coverage of your specs.

Coverage is a number - usually a percentage. It tells you how many of your lines where executed when the specs were run. So, if coverage is 100%, that means that all the lines in your code have been executed at least once. If your coverage is 50%, only half of your where executed. Etc.

A tool which I find really useful is the coveralls.io website. It is not a coverage calculation tool. But once you get your coverage data to it, it can present it in a very human-friendly way. And it doesn't limit itself to showing a single number; it presents multiple reports, per build, job, file and line. That last one is the one which I like the most:

coveralls file report

The picture above is coveralls.io showing the source code of one of my libraries, middleclass.

  • The lines in wite(ish) are blank lines, comments, and lines with syntactic value but no semantic value (such as the end finishing an if or a function). These are not interesting for the coverage, and are ignored.
  • The green lines are lines which have been executed at least once when the specs where run. Notice the number to the right of each one: that's the number of times each line has been executed.
  • The red line is did not get executed at all during the specs. This usually means that I am missing some spec for testing it (it could also mean that my code has lines which are never executed).

In addition to all the reports, coveralls.io also has "github hooks" - it will "turn red" the pull requests which lower the coverage, and "green" the ones which make it bigger (or the same).

So, how do we get the coverage info appearing in coveralls.io? In my case, I generate that info in travis, and then upload it to coveralls.

The standard Lua tool for generating coverage information is Luacov. Luacov can be used to generate a file with almost all the information coveralls.io needs. A second tool, called luacov-coveralls, translates this local file into a format coveralls understands, and does the upload.

Both luacov and luacov-coveralls are installable via luarocks, so setting them up in travis is very simple. We do something similar to this:

# .travis.yml

install:
  - luarocks install luacov
  - luarocks install luacov-coveralls

Once we have the local file generated by luacov, sending it to coveralls is also very easy. Since I want to send the coverage information only when the specs run correctly, I call luacov-coveralls on the after_success section of .travis.yml:

# .travis.yml

after_success:
  - luacov-coveralls -e $TRAVIS_BUILD_DIR/lua_install

Notice that I add a -e parameter to luacov-coveralls. This makes it ignore anything happening inside the lua_install folder. If I didn't do this, the coverage information would include busted's source code, too. This is only necessary because I am using hererocks to set up the environment.

The only missing part is generating the luacov file. It turns out that busted already has a command-line option for it, called --coverage. Add it to busted, push the changes to travis, and it will start sending information:

# .travis.yml

script:
  busted --verbose --coverage

If you are not using busted, luacov can also be included when runing Lua or LuaJIT from the command line. Check luacov's documentation for more information.

d) Static analysis

Contrarily to specs or coverage studies, Static Analysis occurs *without executing the code it analyzes. It turns out that lots of useful information can be extracted from the source code by just reading it.

For static analysis, I use luacheck. This great tool detects common defects in Lua code:

  • Declared but unused variables or function parameters
  • Local variables obscuring other similarly named variables in their scope
  • And the one which probably is more useful, it detects global variables.

Luacheck can be installed with luarocks and executed from the command line. Adding it to travis is straightforward (I run it before I run the specs, on the script namespace):

# .travis.yml

install:
  - luarocks install luacheck

script:
  - luacheck --std max+busted *.lua spec

The only interesting bit is the --std option. The value max means "accept all the global variables which are used in any Lua version". This means that pairs, ipairs, etc will not be detected as variables. But so will pack (even if it isn't a global variable in Lua 5.3) or jit (even if it's not available in vanilla Lua). If you want to detect global variables with more detail, you might want to change this setting by a value adapted to your environment. The +busted part adds the global variables declared by busted (like describe or it).

e) All together

Here's the complete .travis.yml I use in my projects now:

# .travis.yml

language: python
sudo: false

env:
  - LUA="lua=5.1"
  - LUA="lua=5.2"
  - LUA="lua=5.3"
  - LUA="luajit=2.0"
  - LUA="luajit=2.1"

before_install:
  - pip install hererocks
  - hererocks lua_install -r^ --$LUA
  - export PATH=$PATH:$PWD/lua_install/bin

install:
  - luarocks install luacheck
  - luarocks install busted
  - luarocks install luacov
  - luarocks install luacov-coveralls

script:
  - luacheck --std max+busted *.lua spec
  - busted --verbose --coverage

after_success:
  - luacov-coveralls -e $TRAVIS_BUILD_DIR/lua_install

It's worth noting that I use this file as a template; each project might require some customization (deactivating one luacheck rule here, removing an unsupported Lua version there). I encourage you to adapt it to your needs.

Thanks for reading, and happy hacking!


RSS

Subscribe to Lua.Space by Email