Repository navigation
Need a testing stuite #2
Description
There's no responsible way to advance development without a testing suite.
A full test framework is great for general development, but here we focus on just acceptance tests. Either pass or not; no need for detailed failing reports.
some desirable goals:
-
make it very easy to write tests. Every bugfix should include a failing test. Since Lua is a very readable language, it's sometimes easier to express a problem in code than in English. It should be trivial to turn this into the test.
-
minimal requirements. Lua is very portable, LuaJIT also targets a wide variety of platforms. All of them should be able to run all relevant tests. Every external module, tool or language will be an issue for somebody somewhere.
-
make it easy to run one test. very important while developing a bugfix. ideally, a testcase would be run directly from the CLI, not only from a bigger environment.
-
clearly mark tests that are intended for some platforms or have specific requirements. try not to confuse users with tests about JIT on non-jit builds.
Some candidates:
-
an existing test framework, like Telescope, Busted, etc.:
pros:- very complete
- nice reports
cons:
- usually not minimal, might have heavy requirements.
- probably shouldn't be included in the repo, but marked as required for tests.
- very different styles, if you like one, probably won't like another.
-
the LuaJIT-test-cleanup
pros:- already existing
- while not very deep in testng, it covers a wide area of the language and included libraries.
- well known
- light requirements (really? not sure)
- nice 'tags' facility
cons:
- many warnings about not being "the test suite"
- still need cleanup? not sure about this
-
some other test-driver (like https://github.com/tarantool/test-run), or one used by other Lua(JIT) fork.
pros:- already existing and tried
- very complete (in most cases)
cons:
- extra requirements (Python in the case of test-run)
- not well known
-
a new, minimalistic tester (I've just hacked such a thing: https://github.com/javierguerragiraldez/luajit-tester):
pros:- no extra dependencies
- tailored for this specific use
- very easy to write tests
cons:
- untested
- limited functionality (to be improved with time?)
- not well known (but minimal learning curve)
Activity
Gerold103 commentedon Apr 13, 2018
Thank you for the investigation. Despite of the fact that I am Tarantool team member, I can not recommend to use
tarantool/test-runas is - it is huge and hard to support test runner, very linked with Tarantool in its code (yes, no direct dependencies, but in the code Tarantool is mentioned very often). But it has nice reports, built-in ability to run a test under lldb/gdb, and many other features. So it possibly makes sense to fork it and simplify by removal of Tarantool, fibers and other non-pure Lua things, msgpack.From another point of view we must think about not only test runner, but about existing tests too. LuaJit has a lot of code, and it is almost impossible to write all the tests from 0 or to convert them from one suite to another. As I can see,
LuaJIT-test-cleanuphas many tests, and it would be nice to get them.Telescopelooks to be neglected - the last commit was pushed 5 years ago.What about
luajit-tester- my opinion is that we must write our own test runner only as a last resort.igelhaus commentedon Apr 24, 2018
Hi all,
I may be late to the discussion, but here are my two cents. When we started our own fork of LuaJIT, we chose following approach:
This approach did result in a "heterogeneous" test environment, but on the other hand it brought us a considerable amount of Lua code to test against, and it did pay off: As far as I remember, adding each new suite helped reveal at least one stability bug.
Just in case, our first added suites were: the suite for vanilla Lua 5.1,
LuaJIT-test-cleanupand Lua 5.1 tests shipped with thelua-TestMoremodule.So I'd propose to split the issue into two:
javierguerragiraldez commentedon Apr 24, 2018
great tips!
those meta-runner scripts are easy to do as scripts, and I guess even
.batfiles, right? (i haven't done any scripting on windows since they used year numbers for home editions....)igelhaus commentedon Apr 24, 2018
We are Linux only, and yes, those runners are simple bash scripts (with some little fancy additions like output colouring), totalling 291 lines of code for running 6 suites.
I've also abandoned Windows development long ago, but I hope it is possible to implement the same functionality in
.bat, too.