Multiple assertions are fine in a unit test

> One of those people liked to write 600 line cron jobs to do critical business functions.

I was a long-time maintainer of Debian's cron, a fork of Vixie cron (all cron implementations I'm aware of are forks of Vixie cron, or its successor, ISC cron).

There are a ton of reasons why I wouldn't do this, the primary one being is that cron really just executes jobs, period. It doesn't serialize them, it doesn't check for load, logging is really rudimentary, etc.

A few years ago somebody noticed that the cron daemon could be DoS'ed by a user submitting a huge crontab. I implemented a 1000-line limit to crontabs thinking "nobody would ever have 1000-line crontabs". I was wrong, quickly received bug reports.

I then increased it to 10K lines, but as far as I recall, users were hitting even that limit. Crazy.

Here are a few mistakes I've seen in other frameworks:

- Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt.

- It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to class inheritance, etc. Now, you have a tree of hooks. Document whether it is walked in breadth first or depth first order, then never change the decision (or disallow having trees of hooks, either by detecting them at runtime, or by picking a hook registration mechanism that makes them inexpressible).

TAP is better than some things, but it has some serious issues that I wrote about on my blog a while back - https://blog.urth.org/2017/01/21/tap-is-great-except-when-it...

Basically it's nearly impossible to fully parse it correctly.

While there is no strict need, sometimes assertions logically belong together.

Nah, it was some corporate project. But if you are interested, I could rewrite it. It would be a fun weekend project.

https://stackoverflow.com/a/700325 to write out the internal structure of a LambdaExpression.

If you just want the assignments then it's simpler:

Add to the Evaluate method a test for MemberExpression and then:

The variable name is:

((MemberExpression)expr).Member.Name

The value is:

Expression.Lambda(expr).Compile().DynamicInvoke()

FWIW, in Python, this is pytest’s entire thing (although it ends up grovelling into bytecode to achieve it).

Not quite the same, but available on nuget- 'fluentAssertions' gives you something akin to this. I've had decent success with having our juniors use it vs less verbose assertion libraries. I don't know about evaluating individual expressions in a line separately, but it does give you clean syntax and similar error messages that are very readable-

"GetProductPage().GetProductPrice().Should().Be().LessThan(...)"

I'm not aware of any related to testing, but perhaps you could use something like this in tandem with some tests to pull it off?

https://github.com/gvergnaud/ts-pattern

Not really, no.

I like to use Jest’s toMatchObject to combine multiple assertions in a single assertion. If the assertion fails, the full object on both sides is shown in logs. You can easily debug tests that way.

The only way to make it even possible is to do some eval magic or to use a pre-processor like babel or a typescript compiler plugin.

But if you find something, Lemme know.

It might not be the general solution you're looking for, but idiomatic Jest does the job for me.

   expect(width).toBeGreaterThan(0); expect(x+width).toBeLessThan(screenWidth);

You could parse the failing expression using babel and get an AST back.

power-assert

Even if the framework is fine, you can see something like an elaborate if-else tree, or even a try-catch block, and after it's all done, there's a condition check with `fail()`. So the point of failure could be manually detached from the actual point of failure.

Granted, this is not the way to do things. But it happens anyway.

Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened.

As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be harder to know what actually happened.

Just use you best judgement and you will be fine.

Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation.

Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution.

(This formulaic approach is also why pattern books tend to be a dry read).

Devious commenter was describing a (normal) scenario where a unit test is not precise. No need to follow up with an aggressive "so what you're saying is".

> So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?

You're falling prey to slippery slope fallacy, which at best is specious reasoning.

The rationale is easy to understand. Running 100 assertions in a single test renders tests unusable. Running 10 assertions suffers from the same problem. Test sets are user-friendly if they dump a single specific error message for a single specific failed assertion, thus allowing developers to quickly pinpoint root causes by simply glancing through the test logs.

Arguing whether two or three or five assertions should be banned misses the whole point and completely ignores the root cause that led to this guideline.

> It's the job of the developer to cut as many corners of those policies as possible to ship code ASAP.

I can't tell if this is supposed to be humor, or if you actually believe it. It's certainly not my job as a developer to ship worse code so that I can release it ASAP. Rather, it's my job to push back against ASAP where it conflicts with writing better code.

With 22 separate tests you have the possibility of knowing that only a subset of them fail. Knowing which fail and which pass may help you debug.

In Go, in general, tests fail and continue, rather than causing the test to stop early, so you can tell which of those 22 checks failed. Other languages may have the option to do something similar.

https://pkg.go.dev/testing#T.Fail

They are the same. I don't care, as I'll fix them one at a time, and if the fix happens to fix more than one great.

Tbh, one of the differences between xUnit and nUnit, is the way generated test cases work, like specifying test cases in an xml file. nUnit has the TestCaseSource attribute for this, while xUnit has the Fact attribute.

One of the key differences, is there are no test cases generated, nUnit tests just wont run, while xUnit will throw.

Since it's completely legal and sensible for a certain kind of test to have no entries in an xml, we needed to hack around this quirk. When I (and countless others) have mentioned this on the xunit github, the author berated us that how dare we request this.

So nUnit might be buggy, but xUnit is fundamentally unfixable.

How ironic, a unit test framework is so fragile and poorly covered by tests they are not able to determine if a change would cause any regressions.

We are using unittest.

But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases:

https://docs.python.org/3/library/unittest.html#distinguishi...

not really. one thing this is useful for is extracting out various attributes in an object when you really don't want to compare the entire thing. Or comparing dict attributes, and figuring which one is the incorrect one.

for example,

    expected_results = {...}
    actual_obj = some_intance.method_call(...)

    for key, val in expected_results.items():
        assert getattr(actual_obj, key) == val, f"Mismatch for {key} attribute"

> your example stops on the first one that fails, and does not evaluate the others after that.

I don't think this is a big problem; trying to focus on multiple examples at once is difficult.

It might be a problem if tests are slow and you are forced to work on all of them at once.

But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the caller).

this has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed.

Since you already have the object right there, why not do the latter approach?

I personally prefer seeing everything that I broke, not just the first one

Then you need to hire consultants to come fix it!

OP asked how any state change would be tested with a single 'assertion' and I provided an answer. Absolute rules are stupid, but our codebase has just short of 10k tests, and very few have more than one assertion.

The only reason I can really see to have more than one assertion would be to avoid having to run the setup/teardown multiple times. However, its usually a desirable goal to write code that require little setup/teardown to test anyways because that comes with other benefits. Again, it might not be practical or even possible, but that goes of almost all programming "rules"..

Please correct me if I'm wrong, but would a precondtion not just be the postcondition of the setup?

Invariants would either have to be publically available and thus easily testable with similar methods, or, one would have to use assertions in the implemention.

I try to avoid the latter, as it mixes implemations and 'test/invariants'. Granted, there are situations (usually in code that implements something very 'algorithm'-ish) where inline assertions are so useful that it would be silly to avoid them. (But implementing algos from scratch is rare in commercial code)

Unit tests should be cheap. Cheap to write, cheap to run, cheap to read, cheap to replace.

Near as I can tell, many people are made uncomfortable by this in practice because these tests feel childish and dare I say demeaning. So they try to do something “sophisticated” instead which is a slow and lingering death where tests are co corned.

Lacking self consciousness, you can whack out hundreds of unit tests in a couple of days, and rewrite ten of someone else’s for a feature or a bug fix. That’s fine and good.

But when your test looks like an integration test, rewriting it misses boundary conditions because the test is t clear about what it’s doing. And then you have silent regressions in code with high coverage. What a mess.

When you say I 'forgot' an assertion, are you implying that test should include all possible assertions on the code? That would perhaps cover more surface, but my goal (read zealot ideology) here is to have the tests help document the code:

test "pressing d key makes mario move 2 pixels right" {

expect { keyboard.d() }.to change { mario.x }.by(2)

}

I could test the value of the d() function, but I dont because I don't care what it returns.

Didnt understand the "whole story invocation" and exception part, am I missing some context?

Sure property-based testing can be invaluable in many situations. Only downside is if the tests become so complex to reason about that bugs become more likely in the tests than the implemenation.

I've sometimes made tests with a manual list of inputs and a list of expected outputs for each. I'd still call that 1 assertion tests (just run multiple times), so my definition of 1 assertion might too broad..

When you get the suites nested and configured right, and the code decomposed properly to support it, each of these assertions is two lines of code, plus the description of each constraint. So you just write four or five tests covering each one, in descending likelihood of breakage.

I think it’s a sound strategy more often than not, it’s just that RSpec’s DSL can make those trade-offs unclear, especially if you use Rubocop and follow its default RSpec rules.

It just so happens that your tests become IO bound because those small tests in aggregate hammer your DB and the network purely to set up state. So if you only do it once by being more deliberate with your tests, you’re in a better place.

I'd argue that it's the unfortunate result of Ruby being at the center of the Agile and XP scene back when it first became prominent (the manifesto etc) - because that scene is also where the more cultish varieties of TDD originated.

I think even with integration tests they should still be treated similarly - at the end of the day you are setting expectations on an output given a certain input, there’s just a lot more going on in between.

There’s no avoiding it though when you want something end-to-end, or a synthetic test. You’re piling up a whole succession of stateful actions and if you tested them in isolation you would fail to capture bugs that depend on state. In that sense, better to run a ‘signup, authenticate and onboard’ flow in one test instead of breaking it down.

Here's a trivial rewrite that satisfies your dogmatic requirement without any meaningful difference:

    setup()
    
    exercise(but_not_for_the_last_time)

    was_blah = (state.currently == blah)

    state.do_other()
    something(state)

    assert was_blah && state.now == blegh

It shouldn't be dogmatic, but I think it should be something to think about when writing the test or reviewing one. A test should be single responsibility too, in order for it to not be brittle.

I disagree, it can be a good starting point for most cases. You should be able to condense your test in 3 steps (arrange, act, assert) each one a single line, and even if you do not do it because setup is too complicated, it's not worth it for that single test, you assert more things etc., I think the mental exercise to try to think: "can this be made in a 3 line test" is invaluable in writing maintainable tests.

> is that then one assert logically for any N? At what point does it stop?

This is one of the hard things about good tests, they are a little bit of art too. Maybe you can apply the single responsibility like I said before: the test should change for one reason only. By one reason I mean one "person/role": it should change if the CFO of our clients want something differet, or if Mark from IT wants some change.

I am not stressing or enforcing single asserts too much, I feel like tests allow a little bit of leeway in many ways, as long as the decision enhances expressiveness. If the extra lines are not making the test clearer, if a single assert would be clearer for the story that the test is telling, then it should go into a single assert. If I can break the story into multiple stories that still make sense, then I do that, such that each story has it's own strong storyline.

Wait what's wrong with XP and TDD? Where I'm from those are absolutely considered best practice for most situations

There's plenty of sensible advice in there it's just that he argues for the sensible stuff and the idiotic stuff with equal levels of conviction and if you are junior you aren't going to be able to distinguish them.

It would be easier if it were all terrible advice.

Chapter 9 talks about unit tests and there is a paragraph called 'Single Assert per Test', where he says it is a good concept but that he is not afraid to put more asserts in his tests.

That paragraph is followed by 'Single Concept per Test' where he starts with: 'Perhaps a better rule is that we want to test a single concept per test.'

So, technically he doesn't say it.

It feels as if folks are splitting hairs where a haircut (or at least a half-hearted attempt at grooming) is required. Use good judgment, and do not blindly follow "rules" without understanding their effects.

As MikeDelta reports¹, the book doesn’t actually say that.

I’ve come to learn to completely disregard any non-specific criticism of that book (and its author). There is apparently a large group of people who hate everything he does and also, seemingly, him personally. Everywhere he (or any of his books) is mentioned, the haters come out, with their vague “it’s all bad” and the old standard “I don’t know where to begin”. Serious criticism can be found (if you look for it), and the author himself welcomes it, but the enormous hate parade is scary to see.

1. https://news.ycombinator.com/item?id=33480517

Heh yeah but it can be used to write tests that check your assumptions about a 3 party api. Granted, It'll only fail once the test are rerun without the cache, but it can still be a valuable technique. It can be valuable to have a testsuite that a) helps check assumptions when implementing the connection and b) helps locate what part of it later starts to behave unexpectedly.

For mocking, the parent comment means "swapping out the external calls with dummy calls", not "laughing at the developer"

Oh dear