Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind?
Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on.
But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?
Different applications require different tests, so no measure is going to please everyone. If you’re making embedded devices for an airplane, the buyer might ask you to provide a formal proof that the program works. In contrast, web apps tend to simply use end users as testers, since it’s cheaper.
So true lol. Mgmt just announced a directive at my work last week that code must have 95-100% coverage.
Meanwhile they hire contractors from india that write the dumbest, most useless tests possible. I’ve worked with many great Indian devs but the contractors we use today all seem like a step down in quality. More work for me I guess
I.prefer to count and report total tests run as part of each build. We get impressive large numbers, but there is no way to put any specific goal on the exact number, we can always go higher.
But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?
Formally? No, this is basically impossible by Rice’s Theorem. There is not even a guarantee that if you have 100% test coverage, the program is good (the tests could be flawed).
This is just a natural limitation of turing completeness. You can’t decide these properties while also having full computational power. In order to decide such things, you need a less powerful mode of computation (something not turing complete) that can be analyzed more thoroughly and with more guarantees.
That makes sense, thank you. Yes, it’s specifically “test quality” I’m looking to measure, as 100% coverage is effectively meaningless if the tests are poor.
Pit testing is useful. It basically tests how effective your tests are and tells you missed conditions that aren’t being tested. For Java.
https://pitest.org
This is really interesting, I’ve never heard of such an approach before; clearly I need to spend more time reading up on testing methodologies. Thank you!
I’d never heard of mutation testing before either, and it seems really interesting. It reminds me of fuzzing, except for the code instead of the input. Maybe a little impractical for some codebases with long build times though. Still, I’ll have to give it a try for a future project. It looks like there’s several tools for mutation testing C/C++.
The most useful tests I write are generally regression tests. Every time I find a bug, I’ll replicate it in a test case, then fix the bug. I think this is just basic Test-Driven-Development practice, but it’s very useful to verify that your tests actually fail when they should. Mutation/Pit testing seems like it addresses that nicely.
We are running the above pi tests with an extra (Gradle based) build plugin so that it only runs mutations for the changed lines in that pull request. That drastically reduces runtime and still ensures that new code is covered to the mutation test level we want. Maybe something similar can be done for C or C++ projects.
I think this is a good rule-of-thumb in general. But I think the best way to decide on the correct coverage is to go through uncovered code and make a conscious decision about it. In some classes it may be OK to have 30%, in others one wants to go all the way up to 100%. That’s why I’m against having a coverage percentage as a build/deployment gate.
There are tools to detail the code coverage if your tests. I’ve worked with Istanbul in the past, and it’s helped to point out parts of the code that could use more attention
I use coverage tools like nyc/c8, but I can easily get 100% coverage on buggy, exploitable, and unstable code. You can have two projects, both with 100% coverage, and one be a shit show and the other be rock solid - so I was wondering if there’s a way to measure quality of tests, or to identify code that really needs extra attention (despite being 100%). Mutation testing has been suggested and that’s really interesting, I’m going to give it a go tomorrow and see what it throws up!
I am not sure what the common/agreed upon rules are. Seems like it depends on the team lead or manager to decide. Some orgs have better engineers, experience, systems and others don’t.
I used to follow the 100% coverage because I was told to do so in my start. I found myself chasing semi-colons rather than null references. Luckily, I had a team mate with which we argued a lot about what we did, do, and will do and he helped me. (In a friendly manner, not like Dinesh and Gilfoyd from Silicon Valley).
Now, I start my tests by going over how the user will use it, e.g. the happy path. Then happy path away. It seems to cover most cases. It helps if you know the business too. (Think messaging system that is intentionally and strictly simple, or one that has a lot of Unicode and language support… fucking emojis hurt me cause I forget they exist even though I use them all the time, I always forget).
Alas, no matter what, I always miss some test case or a very imaginative user will find a way to show me how wrong I am.
In the end, I think the best, no matter how big or small the project you’re building is, to do many small PRs (with tests) to your team. This way, things are tested in increments and helps prevent PR burnouts. This I need to get better at myself.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !programming@programming.dev
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person’s post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
Follow the programming.dev instance rules
Keep content related to programming in some way
If you’re posting long videos try to add in some form of tldr for those who don’t want to watch videos
Different applications require different tests, so no measure is going to please everyone. If you’re making embedded devices for an airplane, the buyer might ask you to provide a formal proof that the program works. In contrast, web apps tend to simply use end users as testers, since it’s cheaper.
So true lol. Mgmt just announced a directive at my work last week that code must have 95-100% coverage.
Meanwhile they hire contractors from india that write the dumbest, most useless tests possible. I’ve worked with many great Indian devs but the contractors we use today all seem like a step down in quality. More work for me I guess
I.prefer to count and report total tests run as part of each build. We get impressive large numbers, but there is no way to put any specific goal on the exact number, we can always go higher.
Formally? No, this is basically impossible by Rice’s Theorem. There is not even a guarantee that if you have 100% test coverage, the program is good (the tests could be flawed).
This is just a natural limitation of turing completeness. You can’t decide these properties while also having full computational power. In order to decide such things, you need a less powerful mode of computation (something not turing complete) that can be analyzed more thoroughly and with more guarantees.
That makes sense, thank you. Yes, it’s specifically “test quality” I’m looking to measure, as 100% coverage is effectively meaningless if the tests are poor.
“When a measure becomes a target, it ceases to be a good measure”. — Goodhart’s law
On top or, better, in addition to mutation testing, some amount of property-based testing is always great where it counts.
Additionally I like roundtrip tests.
For example we have two data formats and support conversion between both of them.
So I have tests that convert from A to B and back to A. Then I can go and call assertEquals on them.
It’s a very cheap test, that tests all functionality of the conversion itself.
Pit testing is useful. It basically tests how effective your tests are and tells you missed conditions that aren’t being tested. For Java. https://pitest.org
This is really interesting, I’ve never heard of such an approach before; clearly I need to spend more time reading up on testing methodologies. Thank you!
I’d never heard of mutation testing before either, and it seems really interesting. It reminds me of fuzzing, except for the code instead of the input. Maybe a little impractical for some codebases with long build times though. Still, I’ll have to give it a try for a future project. It looks like there’s several tools for mutation testing C/C++.
The most useful tests I write are generally regression tests. Every time I find a bug, I’ll replicate it in a test case, then fix the bug. I think this is just basic Test-Driven-Development practice, but it’s very useful to verify that your tests actually fail when they should. Mutation/Pit testing seems like it addresses that nicely.
We are running the above pi tests with an extra (Gradle based) build plugin so that it only runs mutations for the changed lines in that pull request. That drastically reduces runtime and still ensures that new code is covered to the mutation test level we want. Maybe something similar can be done for C or C++ projects.
Does something like this exist for Python?
https://mutatest.readthedocs.io/en/latest/
80%. Much beyond that and you get into a decreasing return on the investment of making the tests.
I think this is a good rule-of-thumb in general. But I think the best way to decide on the correct coverage is to go through uncovered code and make a conscious decision about it. In some classes it may be OK to have 30%, in others one wants to go all the way up to 100%. That’s why I’m against having a coverage percentage as a build/deployment gate.
There are tools to detail the code coverage if your tests. I’ve worked with Istanbul in the past, and it’s helped to point out parts of the code that could use more attention
https://istanbul.js.org/
I use coverage tools like nyc/c8, but I can easily get 100% coverage on buggy, exploitable, and unstable code. You can have two projects, both with 100% coverage, and one be a shit show and the other be rock solid - so I was wondering if there’s a way to measure quality of tests, or to identify code that really needs extra attention (despite being 100%). Mutation testing has been suggested and that’s really interesting, I’m going to give it a go tomorrow and see what it throws up!
Maybe fraction between money spent on writing code versus money spent on testing code?
I am not sure what the common/agreed upon rules are. Seems like it depends on the team lead or manager to decide. Some orgs have better engineers, experience, systems and others don’t.
I used to follow the 100% coverage because I was told to do so in my start. I found myself chasing semi-colons rather than null references. Luckily, I had a team mate with which we argued a lot about what we did, do, and will do and he helped me. (In a friendly manner, not like Dinesh and Gilfoyd from Silicon Valley).
Now, I start my tests by going over how the user will use it, e.g. the happy path. Then happy path away. It seems to cover most cases. It helps if you know the business too. (Think messaging system that is intentionally and strictly simple, or one that has a lot of Unicode and language support… fucking emojis hurt me cause I forget they exist even though I use them all the time, I always forget).
Alas, no matter what, I always miss some test case or a very imaginative user will find a way to show me how wrong I am.
In the end, I think the best, no matter how big or small the project you’re building is, to do many small PRs (with tests) to your team. This way, things are tested in increments and helps prevent PR burnouts. This I need to get better at myself.