Skip to content

Script to run sample files and compare against known outputs #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Zamua merged 1 commit into from
Feb 4, 2019
Merged

Script to run sample files and compare against known outputs #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ build/
.project
.settings/

tests/**/outputs/*.diff
tests/**/outputs/*.out
8 changes: 7 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,10 @@ Get gradle aliases from my .dotfiles repo: [Gradle Aliases](https://github.com/Z
4. How to run the tests in the most recently compiled file:
* `./test.sh`

5. If you would like to use Eclipse, then run `gradle eclipse` to generate project files.
5. How to run all sample files against previously-known output:
* `./dotests.sh -r tests`

6. How to set expected output for a sample file:
* `./dotests.sh -s <file-name>`

7. If you would like to use Eclipse, then run `gradle eclipse` to generate project files.
2 changes: 1 addition & 1 deletion compile.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash

gradle run -Ptestfile=${1}
gradle -q run -Ptestfile=${1}

129 changes: 129 additions & 0 deletions dotests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
#! /bin/bash

# Prints a help message to stdout
help() {
echo "Usage: $0 [options] file..."
echo ""
echo " Options:"
echo " -h Print this help message and exit"
echo " -r Recursively run all LJ files in the given directory"
echo " -s Set the expected output for the LJ files to their actual output."
echo " If -s option not present, LJ file output will be compared to the previous expected output"
}

# Takes the name of a file. If the input file is a LJ file, compile and run it.
# If the -s option is used, copy the outputs to the expected outputs file.
# Otherwise, show a diff between the expected and actual output.
# If the input file is a directory and the -r option is given, call test on the
# directory's contents
test() {
local file=${1%"/"}
if [ -f $file ] && [[ $file =~ .*\.lj ]]; then
parent=$(dirname $file)
base=$(basename $file)
name=${base%.*}
outdir=$parent/outputs
base=$outdir/$name
mkdir -p $outdir
echo "Compiling $file"
./compile.sh $file 2>&1 | grep -wvi time > $base\_compile.out
echo "Done compiling"
echo
echo "Running $file"
./run.sh 2>&1 | tee $base\_run.out
echo "Done running"
echo
echo "Running tests for $file"
./test.sh 2>&1 | grep -wvi time > $base\_test.out
echo "Done testing"
echo
if $set_expected; then
cp $base\_compile.out $base\_compile.exp
cp $base\_run.out $base\_run.exp
cp $base\_test.out $base\_test.exp
else
touch $base\_compile.exp
touch $base\_run.exp
touch $base\_test.exp
diff -u $base\_compile.out $base\_compile.exp > $base\_compile.diff
if [ -s $base\_compile.diff ]; then
((changes_found += 1))
echo "Changes detected in compile output; use -s option to set expected output"
echo
cat $base\_compile.diff
echo
fi
diff -u $base\_run.out $base\_run.exp > $base\_run.diff
if [ -s $base\_run.diff ]; then
((changes_found += 1))
echo "Changes detected in run output; use -s option to set expected output"
echo
cat $base\_run.diff
echo
fi
diff -u $base\_test.out $base\_test.exp > $base\_test.diff
if [ -s $base\_test.diff ]; then
((changes_found += 1))
echo "Changes detected in test output; use -s option to set expected output"
echo
cat $base\_test.diff
echo
fi
fi
elif [ -d $file ] && $recursive; then
for sub in $(ls $file); do
test $file/$sub
done
fi
}

help_flag=false
recursive=false
set_expected=false

# Parse flags
while getopts :hrs flag; do
case $flag in
h)
help_flag=true
;;
r)
recursive=true
;;
s)
set_expected=true
;;
\?)
echo "Unexpected flag $OPTARG" 1>&2
echo
help
exit
;;
esac
done
shift $((OPTIND - 1))

if $help_flag; then
help
exit
fi

# Make sure there's at least one file given
if [ $# == 0 ]; then
echo "Expecting at least one file" 1>&2
echo
help
exit
fi

changes_found=0

# For each input file f:
for f in $@; do
test $f
done

# Show how many changes were detected
if [ $set_expected == false ]; then
echo "Detected $changes_found changed outputs"
fi
Empty file added tests/comp/outputs/3np1for_compile.exp
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/comp/outputs/3np1for_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Error: Main method not found in class Main, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
5 changes: 5 additions & 0 deletions tests/comp/outputs/3np1for_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
JUnit version 4.12
....

OK (4 tests)

Empty file added tests/comp/outputs/3np1while_compile.exp
View file
Open in desktop
Empty file.
1 change: 1 addition & 0 deletions tests/comp/outputs/3np1while_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1
5 changes: 5 additions & 0 deletions tests/comp/outputs/3np1while_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
JUnit version 4.12
......

OK (6 tests)

Empty file added tests/comp/outputs/abracadabra_compile.exp
View file
Open in desktop
Empty file.
15 changes: 15 additions & 0 deletions tests/comp/outputs/abracadabra_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
1 Abracadabra
2 Abracadabra
3 Abracadabra
4 Abracadabra
5 Abracadabra
1 Abracadabra
2 Abracadabra
3 Abracadabra
4 Abracadabra
5 Abracadabra
6 Abracadabra
7 Abracadabra
8 Abracadabra
9 Abracadabra
10 Abracadabra
28 changes: 28 additions & 0 deletions tests/comp/outputs/abracadabra_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
JUnit version 4.12
.E
There was 1 failure:
1) initializationError(Main)
java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:191)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:128)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:416)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:84)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:65)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runner.Computer.getRunner(Computer.java:40)
at org.junit.runner.Computer$1.runnerForClass(Computer.java:31)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:101)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:87)
at org.junit.runners.Suite.<init>(Suite.java:81)
at org.junit.runner.Computer.getSuite(Computer.java:28)
at org.junit.runner.Request.classes(Request.java:75)
at org.junit.runner.JUnitCommandLineParseResult.createRequest(JUnitCommandLineParseResult.java:118)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)

FAILURES!!!
Tests run: 1, Failures: 1

Empty file added tests/comp/outputs/different_compile.exp
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/comp/outputs/different_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Error: Main method not found in class Main, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
5 changes: 5 additions & 0 deletions tests/comp/outputs/different_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
JUnit version 4.12
....

OK (4 tests)

Empty file added tests/comp/outputs/fizzbuzz_compile.exp
View file
Open in desktop
Empty file.
21 changes: 21 additions & 0 deletions tests/comp/outputs/fizzbuzz_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
1
Fizz
Buzz
Fizz
5
FizzBuzz
7
1
Fizz
3
FizzBuzz
5
Fizz
7
1
2
Fizz
4
Buzz
Fizz
7
28 changes: 28 additions & 0 deletions tests/comp/outputs/fizzbuzz_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
JUnit version 4.12
.E
There was 1 failure:
1) initializationError(Main)
java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:191)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:128)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:416)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:84)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:65)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runner.Computer.getRunner(Computer.java:40)
at org.junit.runner.Computer$1.runnerForClass(Computer.java:31)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:101)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:87)
at org.junit.runners.Suite.<init>(Suite.java:81)
at org.junit.runner.Computer.getSuite(Computer.java:28)
at org.junit.runner.Request.classes(Request.java:75)
at org.junit.runner.JUnitCommandLineParseResult.createRequest(JUnitCommandLineParseResult.java:118)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)

FAILURES!!!
Tests run: 1, Failures: 1

Empty file added tests/comp/outputs/oddgnome_compile.exp
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/comp/outputs/oddgnome_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Error: Main method not found in class Main, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
5 changes: 5 additions & 0 deletions tests/comp/outputs/oddgnome_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
JUnit version 4.12
.

OK (1 test)

Empty file added tests/failing/outputs/add_compile.exp
View file
Open in desktop
Empty file.
3 changes: 3 additions & 0 deletions tests/failing/outputs/add_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Error: Main method not found in class Main, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
43 changes: 43 additions & 0 deletions tests/failing/outputs/add_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
JUnit version 4.12
.E.
There was 1 failure:
1) test0(Main)
java.lang.AssertionError: expected:<3> but was:<3.0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at Main.test0(Main.java:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)

FAILURES!!!
Tests run: 2, Failures: 1

11 changes: 11 additions & 0 deletions tests/failing/outputs/list_remove_compile.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
generated/Main.java:12: error: cannot find symbol
public static T getb()
^
symbol: class T
location: class Main
generated/Main.java:14: error: cannot find symbol
T b;
^
symbol: class T
location: class Main
2 errors
3 changes: 3 additions & 0 deletions tests/failing/outputs/list_remove_run.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Error: Main method not found in class Main, please define the main method as:
public static void main(String[] args)
or a JavaFX application class must extend javafx.application.Application
43 changes: 43 additions & 0 deletions tests/failing/outputs/list_remove_test.exp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
JUnit version 4.12
.E.
There was 1 failure:
1) test0(Main)
java.lang.AssertionError: expected:<3> but was:<3.0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at Main.test0(Main.java:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
at org.junit.runner.JUnitCore.main(JUnitCore.java:36)

FAILURES!!!
Tests run: 2, Failures: 1

Loading