Repository navigation
Fuzz testing #95
Description
For robustness, it would be good to have a separate test module that:
- is run with a random number of IPython engines (between 1 and 20 or so, inclusive)
- creates random distarrays with random sizes in each dimension (between 0 and 1e6, inclusive)
- with a random mix of undistributed, block distributed, and block cyclic distributed dimensions
- that tests ufuncs, reductions, and other fundamental stuff
We could test the edge cases in a separate test class as well -- size of zero in one or more dimensions, all dimensions undistributed, using a view with just one target /engine, etc.
Activity
cowlicks commentedon Feb 3, 2014
We would have to have a way of reusing the random data. So we would need test suite return whatever the initialization vector was.
markkness commentedon Feb 3, 2014
Would it be enough to save/return only the seed used to initialize the
random number generator?
On Mon, Feb 3, 2014 at 12:26 PM, Blake Griffith notifications@github.comwrote:
cowlicks commentedon Feb 3, 2014
@markkness Yes. That is what I meant random seed is a better word than initialization vector.
rkern commentedon Feb 3, 2014
It should take a
RandomStateinstance and use its methods to generate the random data. Don't use the convenience functions innumpy.randomthat use (and thus modify) the global state.bgrant commentedon Mar 10, 2014
John T. mentioned using the Python port of QuickCheck for this:
https://pypi.python.org/pypi/pytest-quickcheck/0.8
@kwmsmith : Is this still a possible task for @markkness ?
kwmsmith commentedon Mar 10, 2014
I've never used
py.test-- what's the consensus on it? Can we keep our currentunittest-based test suite and havepy.testrun it with the fuzz testing turned on as an extra?bgrant commentedon Mar 11, 2014
@kwmsmith : It seems to be pretty widespread, though I haven't used it either. TravisCI installs it by default. pytest will run unittest testcases:
http://pytest.org/latest/goodpractises.html#python-test-discovery
kwmsmith commentedon Mar 25, 2014
Putting on wishlist for now -- can move to milestone 0.3 if and when we get there.
bgrant commentedon Jul 31, 2015
Possibly consider: https://github.com/DRMacIver/hypothesis