[ripleymj]

Not sure how serious you are about #33, but I was playing around with it to see what happened. This adds a pretty basic .travis-ci.yml and runs a few known-good test cases just to get started.

Build status would look like this: https://travis-ci.org/ripleymj/less-java/builds and someone would need to register JMU-CS with Travis.

[lam2mo]

Looks ok to me, but I have no experience with Travis CI. What all is involved in registering JMU-CS with Travis? It's listed as an approved third-party app in the org settings.

[ripleymj]

I think somebody with org access needs to go into Travis, make sure Travis understands you're an authorized org user, then toggle CI on for this repo.

[lam2mo]

Ok I think I have done that: https://travis-ci.org/JMU-CS/less-java

I guess this is ready for review; I'd like @Zamua to take a look before we merge it in.

Thanks for looking into this!

[lam2mo]

Also I think we may eventually want to use the script @hinescd added in 925e358. Both @hinescd and @Kelly4kc are supposed to be working on the test suite during their thesis work this Fall so this could be done then.

[ripleymj]

I looked at that script, but it seems to throw errors all over the place. I think the CI tools like seeing a lot of individual tests so they can assess what percentage are passing. The script gives you giant yes/no result, and I'm not sure as written that it would propagate a failure of one test back to Travis.

There are also a number of tests that depend on user input, so the test suite blocks waiting and those would need to be addressed in some manner. But long story short, I wanted to see if I could get anything working just for fun, and anybody else is welcome to modify it now that the integration is in place.

[lam2mo]

Yes, we definitely have tests that are currently failing with the script. Some of those might be buggy tests at this point but some are failing in the script because we have some tests that have test statements without a main (and so fail when you do run.sh) and some tests that have a main but no formal tests.

Also, some tests are written to expose the issues with our current type inference engine and won't actually pass until @hinescd finishes his thesis work. So we definitely don't want to use the top-level -r option; I was thinking more just running each test file separately. The benefit is that then we can also test things like I/O, which can't really be tested with the test statement and JUnit tests, at least with current capabilities.

In any case, the test suite definitely needs reorganization. Thanks for doing the legwork for getting TravisCI into the picture!

[Zamua]

Great work. We need to ensure that new tests are added to this file in it's current state, but is there anyway to tell travis to run a comand for all files in a directory as separate tests? That way we don't have to remember to edit this file everytime a test is added?

[ripleymj]

I didn't come across anything like that. There's one command to mark a group of tests so they can run in parallel, which would be fun, but I think would make a mess of generated/.

A few ideas then:

• As they're just shell commands in a yaml file, rework the current test framework to read lines from .travis.yml and execute them locally
• Find a more appropriate CI solution
• Add a script to build the travis test file, and have regular contributors add it as a pre-commit hook
• Rework the test script so that it can run without input, dies on first error, and reports success (exit 0) if it succeeds

[lam2mo]

Ok, I think that this pull request closes #33; however, #19 remains outstanding and probably will until the Fall. I envision a script-based test for external issues (invalid programs, etc.) as well as extensive set of embedded internal tests using the test keyword. Both of these kinds of tests should eventually be integrated into both pre-commit and CI-based test routines. I think this was the original intent from the time of Zamua's thesis.

Thanks @ripleymj!