When using class based tests, setUp is not called for each hypothesis test
I was just writing some test for a custom data structure and had a class based test with a setUp function that initialized a fresh instance for each test (so I don't have to copy the code in each test). Some test would fail randomly most but not all executions. After investigating this I found that setUp was simply not called for every hypothesis test which resulted in a "dirty" data structure and in term made some tests failing if the values came in the "wrong" order.
Here is an example:
class TestHypothesis(unittest.TestCase):
def setUp(self):
super(TestHypothesis, self).setUp()
self.test_set = set()
print "setUp called"
@given(unicode)
def test_example(self, text):
chars = [c for c in text]
for c in chars:
assert c not in self.test_set
self.test_set.update(chars)
print "test called with", textIf I run this, I get the following output:
setUp called
test called with
test called with \U0001bf50
test called with \U0004ac1e
test called with \U000d5c8f\U0002fb61\U00051be8\U000d5c8f\U0002fb61\U0002fb61\U00051be8\U000d5c8f\U00095c18\U0010a11f\U000d5c8f\U00051be8\U00095c18\U0002fb61\U000361af\U000d5c8f\U00019548\U000361af\U000d5c8f\U0010a11f\U000361af\U000d5c8f\U0002fb61\U000361af\U0010a11f\U0010a11f\U00095c18\U000361af\U000361af\U0010a11f\U0002fb61\U0010a11f\U000361af\U00095c18\U00019548\U000d5c8f\U000d5c8f\U00019548\U0002fb61\U0010a11f\U000361af\U00019548\U0010a11f\U00095c18\U000361af
test called with 0
[lots of other lines]
test called with \U00095c0f
Falsifying example: test_example(self=TestHypothesis(methodName='test_example'), text='\U00095c18')The reason for this is obivous: Since setUp was only called once the data structure got "dirty".
I'm not really sure if this is something that needs to be fixed but I think it needs to be documented that you should not use setUp in this way (which in my opinion is perfectly fine) when using hypothesis.
Source: HypothesisWorks/hypothesis