So today's topic is mutation testing, um, which I think we've, we've teased already this semester a bit. I think I spent a few minutes on it and we discussed it last week. Uh, today we're gonna spend the whole class on it. So, we're going to go into a lot of depth, but 1st, as usual, uh, there will be reading quiz. Let's see if we can go. And, uh, I'm going to put a, uh, 2 minutes on the clock.
There are the questions mutation testing developers.
Part of your current view is exactly correct. Right right. So that was hopefully the easy 1.
A question too, we're at a 1 line snippet of code. That's obviously error as defined by the paper. What does error to me? What is the paper to find? Error? Does.
Yes, uninteresting in particular uninteresting to mutate. All right so mutating. It wouldn't help.
It's not going to help uncover bugs.
Or write new tests. So what are some examples of things that the paper suggested are arid? Yeah.
Yeah, print statement is an excellent answer are the things that anyone wrote.
I thought of logging.
Yes, logging or debugging code. That's an excellent example.
There was 1 other example that they gave, I think, at least once maybe twice.
Was the, uh, uh, the, the initial size of a collection, right? If you initialize in a ray list as having 16 elements or 64 uh, at the beginning, it doesn't make a big difference. It's only going to affect performance. Right? So, uh, anything like this. But anything that, uh, we believe is error that you wrote down will get full credit. And I hope Cosby is listening here.
Anything that's even remotely reasonable is full credit. All right. Okay. Any questions about this, we'll come back to the, the idea of later in the lecture.
All right, if not, then we will press ahead. So today we're going to talk about mutation testing and we're going to go through a mutation testing and quite a bit of detail. I've opened some relation and I'll review at least all the definitions that we talked about before, um, and probably will cover a few more mutation testing for whatever reason. It's super heavy on vocabulary. So, I apologize in advance for that.
Then we'll talk about some of the assumptions mutation testing and also the implications of those assumptions. Right um, and all of this will be with the goal of getting towards.
How practical is this in real life? Like, how do people actually use it? And so, towards the end of the day today, we're going to talk about the Google paper that we read.
All right, so let's get on it. Uh, so, does anybody know what this means? Has everyone seen this phrase before.
Does a surprise test of your liberal arts education? Does anyone know what language that said?
Flatten yeah, definitely Latin. Um.
And we'll have a guess, at what am I mean? Yes that's exactly right, right. Who watches the waters right? Or who watches the watchman? Right? Um, and it was said by this, this guy juvenile, who is a Roman satirist in the 1st century 80 that's not very important right? What it is is who watches the watchers.
And this is fundamentally the, I, the idea behind mutation testing basically boils down to, uh, how do we apply this question.
To, uh, to testing, right this is a very famous question that, uh, you know.
If you take a liberal arts class, you might encounter. Right? This is a famous phrase in the realm of things like politics. Right? But in testing, it's also the motivation for.
Why we do mutation testing right? And it's notable that this question is recursive. Right? Whatever the answer is, we can ask the same question about it, and we'll come back to that, uh, as we go further today. Right? This is part of the reason this question is interesting for to, for example, politicians, because.
You know, if, even if you appoint someone to watch, like, then the question is who does that? But of course, now someone is watching them. So now you, it's sort of there's a chain of watchers all the way along.
So, what does this have to do testing as I said, this is the main motivation for mutation testing right? Is that the sort of core idea of mutation testing is that we need or we want to have a, um, a system that can keep an eye on our tests right?
Um, and so, uh, the key question we need to ask ourselves here is, how do we know that our tests are actually good right? And the, the reason that this matters that test also programs, right? When we're writing tests.
We're trying to check whether our program does the right thing, but the test themselves are programs and therefore they can have bugs in exactly. The same sort of ways that other programs can have bugs and so they can also be wrong. Right? And so, you know, if we, if we knew that our tests were correct, then we wouldn't need to worry about.
Testing our tests, but because we do, uh, we don't know that, like, the same reason. We don't know that, uh, our programs are correct.
Uh, we do actually need to worry about how do we check whether the test themselves are right, right so, to some extent, today's conversation is going to be about, like, sort of a meta view on testing. All right. Like, this is the question about, how do you check whether tests are actually doing the right thing?
All right. Okay, so we've already discussed 1 possible answer for how do we check whether test survey.
And that's coverage, right? We talked that we said, like, coverage is a a way to tell whether 1 test suite, uh, does a better job of covering the, uh, the, the space of the program, right? Uh, than another. And, and how much is a useful measure, but it's a limited 1, right? So, you know, better coverage could be meaning better tests, but that's not. We've seen examples of.
Cases where that's not always true and this is because coverage is an imperfect metric right? Fundamentally, it's only about inputs. Right coverage doesn't tell us anything about the quality for example, of the Oracles that we're using. Right. You can achieve high coverage with an Oracle that just says don't crash.
And you're going to achieve high coverage with a set of very good Oracles but coverage can't tell those 2 test suites apart.
And so average is it best a rough guideline to the actual quality of a test suite? The actual quality is some, like, unknowable, uh, you know, subjective thing. But I covered is a rough approximation. Right? It's a rough guideline.
Um, shouldn't like.
War coverage does not necessarily mean more better in all situations. Right?
And so our question for today is going to be, how can we do better? Right? How can we have a measure of test suite quality?
That takes into account more things than just inputs. And in particular we want to test suite, quality metric that naturally considers the quality of the input. The way the coverage does right. Coverage is basically a measure of how much of the input space have you, uh, have you covered.
Um, and also, uh, the quality of the Oracles, right? We want something that is naturally aligned with both.
And that's what mutation testing is going to be 4.
Okay, so there's a general technique for solving who watches the watcher style problems, which is basically, like, test the waters.
So, intentionally what you do is if you have a set of watchers, and you want to know if they're watching correctly, you intentionally introduce some problems and you check how often they find this. So, for example, if you've heard about, like, people running tests against the, the TSA, like the guys who do security at airports, right where they like, sneaky, try to sneak a gun onto a plane. That's an example of this.
Or they're effectively like, there's a known attack that's being carried out on purpose but the watchers don't know.
Right and then the question is just, does security catch you with the gun when you try to walk through if you're 1 of these, like, special people who test security right?
And so this is the sort of the core idea behind this sort of general technique for solving these sort of problems.
And so the idea is that we want to see whether the waters actually detect the known problems.
And, of course, this general technique can also be applied recursively. Right we can add some fake known problems. Right? And see if, uh, our testers are being honest and say all right.
And this is nice, but it it is generally very expensive. Right? We always need to add another level if we want to make sure everyone's being honest. And so we need an infinitely large structure of people watching. Right? And so this is not ideal, right?
But this is the sort of the same sort of style of defense that mutation testing it's.
So, how can we apply this technique to test it? Well.
Uh, in this analogy, tests are the watchers and they're watching for.
What what are tests watching for.
Bugs bugs yes, definitely. Bugs, right they're watching for bugs right and then, in the analogy, then what are, what are the fake things that we're going to? So, as you apply this general technique, right? We need to introduce intentionally introduce some known problems in the system and see if the waters detect them. So, what that means then, is that in the analogy here to the general technique, um, the known problems here are going to be.
Big box, we're going to introduce a set of fake bugs and see whether the test detect them. Right? And this is the core of the mutation testing in general.
The core idea of the technique, right? And if you just remember, uh, you know, what 1 thing about mutation testing this is it right that mutation testing is introducing fake bugs and seeing whether the test detect them that's the most important like, high level bit. If you want to explain it to your grandmother, that's what it is.
Okay, so we intentionally introduce some changes to the program, and we expect them to cause the test to fail. Right? And what we're going to measure is.
Do the tests actually fail when we expect them to? That is the, that is the core idea.
Okay, any questions so far on this sorry to hit you with the, like, aggressive liberal arts, uh, reason for why we care.
I had it's too much fun to put light at the beginning of the class.
All right, moving on then this is the idea this idea is the essence and this is the core.
All right, so I think we've seen this definition before, but here it is again, mutation testing limitation analysis is a test suite adequacy metric in which the quality of a test suite is related to the number of intentionally added defects that it finds. All right, this is the, the core technique that we're looking at here.
Right so informally if you claim your test is really good at finding security bugs, then I will just intentionally introduce a buffer overflow and see if you find it.
All right, that is the core idea. And if you recall, I used a truffle sniffing peg analogy to introduce this.
A couple of weeks ago, right? I remember the trouble sniffing pigs.
Yes, hopefully it was insane enough that it stuck out in a few people's minds. Right, and the idea here was that, like, you have some trouble standing pigs, and you want to see if you had some troubles in your backyard. And then the best thing is the 1 that find the most of these products.
All right, and, uh, this is the same kind of thing that we're doing the mutation testing, just with bugs and tests.
The key 1 of the key problems that are going to immediately have to confront here, though, is.
Are the bugs were introducing realistic? That is to say very similitude. I know this word.
Verisimilitude part of this before it just means like.
Relationship to reality, or how close to the truth is right.
So, in this trouble, stuffing pig analogy, right? If I hide every truffle in my backyard next to a smelling red flower.
I picked it finds all of those might actually not do well in the real world. Why.
What's wrong with this situation? Let us know the cloud.
Right. There's a CO founder. I know the word Co founder right? I've heard of that this, like, within, like, statistical analysis, or doing science in general.
A few people. Okay, so, let me just briefly. Explain so that, uh, we're on on the same page, right?
If you're going to do, like a scientific experiment, right?
Um, 1 of the things you really have to worry about is controlling for possible. Confounds.
What that is to say, like, if you're trying to measure, you know, whether, uh.
The temperature changes, because you do a funny ritual, right?
But you happen, you happen to do this during, like, right in the middle of spring, when it's getting warmer every day, it will be the case that you will observe that, as you do the ritual more if you do the ritual more each day, the temperature doesn't fact, go up and you'll say, oh, it worked, right? I can you can draw a statistical correlation. Everything looks great.
The bad news of course, for this is that there is a compound it is spring and the temperature is going up every day by chance right? Because of time not because of the thing that you did, right? You did the same experiment in the fall. You would get exactly the opposite affection and so the, the presence of time not being controlled for properly it's called a compound within the.
The literature on experimental design. Okay. Any questions about that? Does that make that idea? Make sense?
So, the compound here, is this not a red flag right? The pig is noticing the smaller red flower and not actually the truffles.
Then it's being, you're measuring the wrong thing, basically. Right? Your, uh.
You're you are, uh, being confused by this, this contract. Okay. So how is this related to the idea of, like, testing whether you're, you're testifying fake bugs?
Does anyone see the connection? I see some sort of maybe.
Paychex.
Yeah, so, uh, this is going in the right direction, right? Like.
So, in this analogy, right? The, the troubles are the bonds, right?
And the pages are tests what is the smelly red flower that.
Well, it's some other property of the code that might cause the test to fail or.
When cause the test to fail when you go back to right?
Um, and basically, the idea here is that the trouble placements that you make up might, or might not actually be indicative of real world truffles, right? That's another way of framing this. Right. That is to say where I put the truffles in my backyard, when I'm hiding them on purpose is not necessarily the same as where I'd find them in real life right out in the woods. Similarly.
When I edit the program to introduce fake bugs.
Well, I always put them in the same places that bugs would actually crop up in the wild.
Well, obviously, I can't right, because if I knew where bugs were going to appear, I would simply fix them. Right? So, the key question then is, uh.
If I add a button, like, how do I add big bugs to the software in such a way that they're it's actually informative, right? It actually corresponds to how, uh, humans would make mistakes right?
And so if I add folks to my software that are not indicative of real world.
Bugs the humans with right then the result of mutation testing wouldn't be very helpful. Right? We might be able to find all the fake bugs, but none of the real ones.
And so, as we go through, I want you to keep this in mind, especially when we start talking about things like mutation operators. Right? And as you're thinking about homework, 6 as well, like, once you start working on homework, 6, um.
In homework since you're going to be asked to build a mutation testing tool right? A very simple 1, but a mutation testing tool you're going to have to choose the mutation operators, right?
When you choose mutation operators, try to think about how close this mutation operator actually is to the kind of bug that you might write. Is it closed or not? Right? I mean, you're probably like whether or not it is, it's not going to necessarily change how well you do on the assignment.
All right, but that's the sort of thing that we're going to be looking for you to reflect on in the right.
Okay, so the implication here is that mutation testing requires us to know what robots actually look like. And how good of a model do you think we, as, like, the software engineering community? Have of what real bugs in the world actually look like.
Like, if you had to guess pretty high meeting sorry is it good model or bad model? Let us just say, like, do we have a good understanding of what kinds of bugs show up.
I think you're overestimating the strength of our discipline at this point. Yeah.
Um, so thank you for actually being brave enough to answer, but I would generally argue we don't really have that good of an idea. Right? The kinds of bugs that show up in real life are difficult to characterize each 1 seems to be unique. Right? I mean, of course there are categories that we can talk about, like, buffer overflow or resource leaks or the, like right but the root cause, like, why did he like.
How did a human make that mistake is not well understood right? And so that's the core problem here, right? To do mutation testing effectively. We have to make the same kinds of mistakes that humans would.
But we don't fundamentally know why humans make the mistakes that they do. Right this is sort of a like, we need to understand how human brains work problem right? Which is just something we don't really understand yet. Right? And so, um.
What our best guess here is going to be what we can do right? And so this is, this is why, in part the mutation testing is both quite effective.
All right, like, it can be quite effective, but when done poorly, it can.
Really be misleading right? And so you have to be very careful about exactly sorry to be very careful about the mutation operators you're using.
Be very careful about how you interpret the results. Yep. All right. Any questions about that does that make sense?
I wish I had a better answer for, like, why do humans write bugs?
But if we knew, we would simply teach you not to write bugs anymore and then we wouldn't have to have this conversation about testing.
All right. Okay so then we're going to call it defects seeding the process of intentionally introducing a defect into a program. Right? Um, and typically what we're gonna do is we're gonna try to make it similar to the defects that are introduced by real developers. But, uh, like.
This typically here typically unintentionally or doing a lot of work in the sentence.
All right, because this is what we are trying to be as similar as possible to us.
But whether or not, we're succeeding is very hard to to.
All right. Okay. Um, so, typically, we're gonna we're gonna do seating by changing the source code. Uh, that's what they're doing homework 6. um, it's also possible to do mutation testing say at the bite code level, right? And instead of writing a mutate that targets see, programs, you write mutate, that targets assembly programs. Right? The problem with doing that, is that the assembly.
Programs are typically much more brittle and so any single change to them tends to make them much worse. Right? By contrast if you, like, make a small change to a C program often, you can still compile it into something sensible.
And then, uh, typically, what we're going to do here is, uh, doing defects hitting automatically. Right? So, the, the tool you're writing homework, 6 is a tool that automatically does mutation testing.
Um, and so we give it a model of what human bugs look like, in this case, you're going to write some mutation operators right? And give them some probabilities right about when to apply them.
And, you know, that will be that's how it will work. However, you, you can still do lightweight mutation testing yourself.
And so, for example, you can think of both regression testing and test driven development as forms of manual mutation testing. So think to understand this think about, why is it, the case that in test driven development when we were discussing it? I said, you must always write the test 1st and ensure that it fails.
The reason is that we want to show that the problem that you're introducing there, the problem that you're, you're writing a test for, and that you're going to then fix actually exists.
Right. And so that is to say that the test is actually a good model of that problem. And so you can view these things as a sort of lightweight forms of manual mutation testing. But when we talk about mutation testing, this is not what we need.
I want to be clear, right when we were talking about mutation testing, we mean, this automated version that does it at scale.
Any questions about this.
All right, so then I've said the word mutation operator, a number of times, let's talk a little bit about mutation operators, right?
So, mutation operator is a way of systematically changing a program, right?
Um, typically an invitation testing what we try to do is model them on historical human defects, but I say this, but I, I, I'm not even sure I really believe it. Right. Fundamentally, like historical human defects are hard to collect and hard to reason about. And so the examples I'm about to show, you are not really modeled on historical human defects.
They're really modeled on, like, typos.
All right, and so I want to be careful with this. This is this is the operational what they're supposed to be.
Right. And then the bottom of the slide is going to be like, what you actually see in practice and what you're actually probably going to do in homework 6 right and homework. 6, if you can find a list of historical human defects, and then manage to, like, make mutation operators out of them, Bravo to, you'll probably do better than everyone else in the assignment. Right?
Probably though you'll write a D, right. Mutation operators like this.
I changed a lesson into a lesson rebuild and equal to.
Well, swim lanes, et cetera. Reimbursing functions.
I change technical wise, I think. What's that?
Um, what is what do all of these things happen, huh?
Yeah, they're really simple, right? They're simple mistakes, right? They're they're basically typos, right? If I write less than less than or equal or vice versa like.
That's probably not like a, it's not a logic, but in the sense that it's not like, I have the wrong model of the world, and I wrote the wrong program. Right it's like, I have a decent idea of what I'm doing. And, uh, you know, I got 1 edge case wrong.
Um, and so in practice, a lot of the things that mutation testing is doing is like.
Making these little changes, right? And we'll talk later about what's called the competent program or hypothesis. Right um, and, you know, I'll have I have a whole several slides on that later, but the reason that we think that this is okay, is the competent program or hypothesis.
And as we talk about the competence program or hypothesis, uh, hopefully, it will become clear that this is like, not.
Necessarily true that these are actually good models of historical human defects.
And just to give you a sense of, like, how far we have, or have not come, this is the list of mutation operators in from.
Opera this was 1 of the 1st mutation testing systems that was actually made practical. I think it targeted. I want to say 4 trend.
Anyway, it doesn't matter. Right um, but these are the, uh, what the invitation operators were, right? And as you can see here, uh, if you read through these briefly.
They're not that different from the ones I showed here right? They're very simple. You know, like replacing 1 constant with another. Right?
Uh, you know, changing the label on I go to right these are all basically, like, 1 statement alterations to the program.
So, why do you think it is that people prefer when writing mutation testing system?
To choose mutation operators like this, instead of like, trying to model the kind of mistake that a human would actually make.
In reasoning that our program, what are these things all have in common.
Besides being, like, looking like, typos, what might motivate the designer of mutation testers.
Yes, that is exactly right they are simple. Right? They're easy to come up with and more importantly, they're relatively easy to implement. Right? Like, if you wanted to do if you want to automatically replace every constant in the program with a different constant.
Like, as you'll see in homework, 6, that's not that hard, right?
If you want to rearrange the entire program to imagine that a human had written in a totally different way using a different set of data structures, like good luck, like, even an outlet, I don't think can do that now. And that's the best system we have for arbitrary rearrangement of programs. Right? You know, people haven't actually, this is a total aside, but like.
There is a feeling, let's say, among the research community that LMS might actually be a huge benefit from mutation testing. Right just because the mutation operator of call the and ask it to mess this up is way better than any of this. All right. And so we'll see whether how that.
That goes, but I've talked to a few people who are expert, like, true experts in mutation testing, as opposed to me. Uh, you know, who is.
Was way too much, but it's mostly a novice. Um.
And they seem very excited about this prospect. So that's something to look out for coming down the pike. Maybe we'll be better than these kind of mutation operators. We'll see.
Right so that aside, does anyone have any questions about these operators? Does it make sense why these are the ones that historically have generally been used.
Also, by the way, and just because I think it's fun in homework 6, when you're doing mutation operators, if you want to, you're welcome to include a call the, and ask it to mess up this program component. That is totally, totally allowed.
All right, it's up to you if you want to try anything and saying like that. But if you do, we'd love to hear about it.
Okay um, yeah, so the key questions here in mutation testing about these operators are what operators you use.
And how often do you use each? 1 and I'm not, I mean, you intentionally not really discuss these because these are what homework is supposed to teach you.
Right the main, the main pedagogical goal of homework 6 is not, like, learn how the library and Python works. It's instead to figure a to get a feeling for the like, how effective these operators actually are, and when to apply them. Right? And so I'm, I'm not going to give you, uh, too many details on the, the core ideas there. Instead I want you to figure it out.
That's right, because I think you'll learn it better that way. All right.
So then we've talked a lot about, um, mutants. Right? But here's the official definition. Right? I'm muting is when we run a mutation operator against the program, and get a new program out. Right? Um, and I think I gave this definition before as well, like, the order of the number of, uh, mutations that are applied to.
In this order, and a quote unquote, higher order we need is 1 that has 2 or more orders. Sorry order 2 more.
So, for example, if we make 2 changes to this, uh, this original here, right we switched the less than 2 less than equal and we switched the, uh, the plus minus. Right? That would be a 2nd order of hierarchy, right?
All right hopefully this is a relatively straightforward definition.
Yeah, so I'm going to use this definition a bunch of times, which is why I wanted to put it up again, just to make sure it's in ever in the forefront of everyone's fine.
Okay, so then next piece of terminology here is that a test suite is set to kill or detect, or reveal a mutant. If the union fails attached to the original program passes.
What happens if the mutant passes, the test that the original program fails.
If the if the original fails, the program, sorry if the original fails and the, the meeting.
If both of them pass, then yes, we're going to come back to equivalent meetings in a bit and, uh, we'll it's a good thought like, hold that 1.
Possible right yes. Uh, so, yes, that's exactly. Uh, uh.
That's exactly right. And that is the intuition behind an entire research area called automated program repair. That doesn't really work. All right so, there's a lot of work that's gone into trying to make this happen, but it doesn't usually happen. The key problem is basically the tests are not indicative enough of, uh.
Whether or not the program is correct, and it's easy to pass the tests while breaking the program and exciting ways by applying mutation operators in in ways. That would surprise you. Okay. So that was a total of total digression. Apologies. Right? But we were killing used by immune failure and the original ask.
So, killing maintenance is generally considered good, right? And mutation testing proceeds by making a lot of making a bunch of mutants.
And then measuring the fraction of them that a test suite kills right? And this is called mutation adequacy score or just mutations score. And that's the main outcome of mutation testing generally is a rating of 2 different test suites based on their mutation adequately score.
Right. Um, as we'll see as we get into practicality, though, we can also use this technique to just derive new tests right? Or by, like, telling humans like, maybe you should write a test for this. And, in fact, like, as you may have seen from the paper, that's the main way that, for example, Google uses mutation mutation testing. They don't primarily use.
For the mutation adequacy score, which is traditionally what it's thought of as the most useful thing that comes out of mutation testing.
They use it as a way to bully developers into writing better tests. Right? But the mutation out of practice score is sort of the traditional traditional output metric mutation test.
All right, so then the idea is that it has to meet with a higher score is better.
And as I said, sorry, for all the vocabulary, this is not quite the end of the vocabulary. But at least this is the end of the only vocabulary section of this talk. So I apologize.
All right any questions here, right?
So then let's get into assumptions and implications here so suppose that I have 2 programs and each of, which has its own test suites these are 2 different programs from a test suite. We do mutation testing them to determine if it has an 80% mutation score and program B's test suite, we test with mutation testing and we determine has a 50% mutation score.
Yes, yes.
Yes, in general, right? Because.
So, it depends on the situation, right that that doesn't actually come up in most deployments mutation testing. Right? And the reason is that you typically only have passing tests for a program.
If you have a feeling test for a program, probably you're going to fix bugs rather than test where the test suite is good.
That said, um, you know, if you do happen to have a program with failing tests, then yeah, you would you like.
If for whatever reason you're doing mutation testing on it, the way that most mutation testing tooling would work is it would count it as a, a mute and kill. If the output of any test is different. Think about it this way the mutation testing system has no idea what the correct answer is.
It doesn't know whether the mute got the right answer or the original program got the right answer. So it just assumes the original program is right all the time.
And so anytime it differs, it's called.
Good question though. All right so coming back to the slide.
Suppose that we have these 2 programs programmatic front of me. Um, can we say anything about the 2 test suites which program has a better test for Airbnb?
Yeah, okay.
No, no, not particularly, but it doesn't really matter right? Where the this question is really about, uh, like, are these values compatible right? Is this 80% better.
50% wasn't compatible. Yeah. Why not?
Yes, that's exactly right. It's because the programs are different, right? So these test suites are not related to each other. Right? So, um, the answer is, we don't know. Right and the important thing here is that mutations work are not comparable across different programs.
Right similarly, like, uh, coverage scores are not directly comparable across different programs. We want to remember why.
Think about P. G. and the number, the amount of coverage we're getting on that in that homework versus, uh, like the for your homework that we did at the beginning of the the course and how much coverage you could achieve there.
Right wasn't the case that the had better testing because, uh.
You could get 90% coverage on it.
Not really like, you couldn't really say that the true was better tested, because they're just different right? And similarly, here with mutation testing has the same problem. Right? The score is not directly comparable across programs. If anyone ever tries to convince you to compare mutation scores across programs, you just tell them they're wrong. Right? It's an easy fallacy or easy mistake to make. And so I would just want to get it out here. 1st, and.
Headed off of the pass never compare mutation scores. Right? And if, uh, if you see someone doing it, you have to tell them they're wrong. Yeah. So, um.
The standard setting here is that what we want is the same program.
A different passwords, right? If we have 2 tests fits for program.
But it's the same program, then we can actually use mutation tester.
And that's the case where the higher mutation score is better but mutation score is not like an absolute measure that we can compare.
Uh, across different programs, right? Yes.
Is the, uh, it's exactly this the fraction of meetings that are killed by the test suite.
Hi, Hello?
Yes, yeah, yeah. Uh, the divided by total number of meetings.
Yeah, so in particular, like, suppose we generate 10 meetings and the test suite. Uh.
Kills 5 of them, that is to say that for 5 of the meetings, the test suite, uh, the test fails right? But for the other 5, all at the same test pass, in that case, then the mutation score would be 50% because half of the meetings were killed because of the tests.
For these different answers, and half the mutants survive, because it has produce the same answer.
Okay, any more questions here does this all make sense? Yeah, yes.
My situation.
So, we can do something like that. So, this is a good, good intuition right? You can do something like communication center where you flip the same indifferent and this, uh.
Uh, statements here, right? If you have different programs that are trying to achieve the same thing, and you had 1 test capability done, right?
Then you could try and mute hitting the password and seeing which of the programs actually is more like, you know, breaks more often. When you pass a different tests.
This is not a standard technique. Unfortunately, the closest thing among what we've talked about is differential asset right? And differential testing in that setting is going to be much more effective.
If anyone has not received candidate just by speaking today, uh, just.
Start waving at some point tonight. I'll talk to that to you. I'm very easy with the candy. All right any other questions here before we move on.
All right, so then I mentioned the competent program or hypothesis, there's 1 other important 1 that, uh, this whole notion of mutation testing relies on for its effectiveness, and it's called the coupling effect hypothesis and I'd like to go into both of them in some detail here. Uh, to give you a sense of, like, how true they actually are right? And also the implications.
For that, or the implications of these, these assumptions for how to interpret mutation testing scores, mutation source. So.
I just want to also give a hint here, which is that, uh, like, uh, I'll mention at the end, the exam is coming up in just a few weeks. Right? And a common style of exam questions will highlight task is consider some assumption that we discussed. But some technique that we have talked about at length, uh, assumes what would happen if that assumption wasn't true. Right tell me about.
Examine that, and that's that's a common style of question that I would like to ask generally when writing an exam because it makes you think about, like.
Both, how true is that assumption and also demonstrate, like, makes you demonstrate that, you know, how the technique actually works.
Okay, so, 1st, the competition program or hypothesis so the competition program or hypothesis holds that program faults that is to say bugs are syntactically small and can typically be corrected with just a few keystrokes.
All right, so this is the reason.
Like, this is the idea that if we change, you know this to this, that's actually fixing a real box.
So, um, the idea then I like the underlying argument is the programmers generally wrote programs that are mostly correct programmers, get their higher, like the high level idea, right right. And then when they're when they're making mistakes, because they miss edge cases, basically.
Right. They, they, they should have thought of even less than than we did less than, or equal to or they just forgot to test a case and they type equals equals instead of not equals. Right? They meant equals equals, but, you know, their fingers left and they were not equals instead. Okay.
So, if the test suite is good at catching artificial meetings that simulate these kind of keystroke mistakes, smoky strokes, it will also be good at catching a unknown, but real false.
That actually will exist out in the wild. Right and so this is the way this is the formalization maybe of why using these kind of small operators like mutation operators might actually be good at catching bucks in the wild right? Or it might actually be a useful measure of.
How good a test suite is so, how true. Do we think this is.
True. Very true. Why do you say that?
That's true. Yeah, I made plenty of typos. Right. Uh, so that's a great point. Right? Um, it is the case that, uh, almost always when you're programming, you are going to make at least some type of.
Right. Does anyone, uh, like, want to make a counterpoint about that?
I think there is an excellent 1 in particular like.
When you're programming in the modern world, are you doing it in a plain text editor? Or is there a spell checker or auto correct? Or something looking over your shoulder all the time?
It's more like the latter situation, right? Like, there is an ID if you say something like that, obviously isn't correct like, if you write, you know, how many how many of you have written something? Like, if X equals equals novel.
X dot do something. All right. We've all written this before. Right how many of you actually executed this code?
I mean, I have, but that's because I grew up in time before IDs and also, like, at some point, I was made to, like, program in a place where I couldn't use an ID, just a terminal. Right? And so I have made this mistake directly. However, most of the time, if you're programming an ID, it will immediately say, are you sure are you sure you didn't mean not equals here? Like, I mean, surely, if you're referencing.
It right away like this is supposed to be a null check right? Like, this is these are backwards.
You get the little you get a little squiggly line here. All right. Actually, I think this squiggly line usually is going to be here, but.
Regardless you get the little red squiggly line and then you say oh, yeah oops. And you fix it.
So, in the modern world, many of the kinds of defects that mutation operators simulate.
Are unlikely to make it into production programs right? But even more than that, it's not just that they're unlikely to make into production programs. Right?
If we're doing mutation testing, what does that suggest about.
Our dedication to testing, let's say, would you say that if we're doing mutation testing our dedication to testing is.
Below average, average, or probably above average.
Above average, right probably mutation testing is considered a relatively advanced technique still. Right? It's relatively expensive to apply. Most places aren't going to be doing it. Right?
Maybe they should, but you can definitely get away without it. Right?
So, that has a an implication, but implication is that we probably already have a decent test suite.
If we ever execute this line, what will happen.
The program will immediately crash. Yes. And so we don't even need an Oracle to know that this is wrong, right? If we even cover this, we'll detect the problem.
Right. And so this kind of simple mistake is unlikely to actually make it any further than the test suite executes. If you have even remotely reasonable coverage.
All right, and so mutation testing isn't really like, this is not a particularly useful fault to simulate for mutation testing. Right?
And when we start talking about detectable versus productive mutants and getting into, sort of some of the details from the Google paper later on today, like, this is, this is the point that they're getting at.
That this is a mistake that you basically wouldn't make in real life right now right?
My contrast though, if I write something like.
If X greater than 0 do something with X.
All right, what are the odds that I accidentally should have written that.
Will I detected less likely? Right? Unless I write a test case that specifically checks, whether X is like, X equals equal 0, it goes 1 way or the other. And so, 1, uh, there's a good argument that this is a much more realistic buck.
Have like going from greater than greater than or equal right then going from equals equals and all to non external.
So, the conclusion here, what I want you to take away from this is that is this is this hypothesis true.
Well, sort of, right um, so, yes and no, right? So it is true that humans often make simple mistakes and some of those use filter, right? And some of them do lead to really serious bugs right?
But humans also made much more complicated mistakes and.
Those complicated mistakes are not always, uh, something that is easy to tie back to a simple mistake like this. Right? So, for example, um, there are a number of famous bugs related to, uh, converting between units. Right? So, for example, I think there's a famous spacecraft loss. I want to say this was the Mars Polar orbiter spacecraft that was supposed to be.
Sending to some foreign planet's surface right? And 1 of the components reported the altitude, I think in feet.
And the other was expecting it in meters or vice versa and so it failed to deploy parachuted the appropriate time and crashed into the surface of Mars and exploded. Right? And everyone was very sad.
Put a mutation testing system have detected that, or that there was no test to check for that situation.
Is that a bug that's due to a simple typo? Like, not really it's really a bug that's about, like, the specification of different components.
Right. It's at a higher level than the kind of little small bugs that mutation testing is simulator. Right?
How could you simulate such a bot it'd be difficult to do, right? Because you, you sort of need to imagine, like, different different implementations of components. Right? Uh, and.
It turns out that, like, implementing entire components is very expensive and you can't really do it automatically.
If you could, we wouldn't need programmers. Right? And so that's not really possible. So, um, the, the thing about the, the competent have our, uh, program, our hypothesis here is that we kind of have to believe it to think the mutation testing is useful right? Like, mutation, testing's usefulness relies on this being true.
But we all kind of know that it isn't true right? And so this, this limits, the usability or the usefulness of mutation testing.
Intention testing is quite good at simulating these kind of small mistakes and it can be very useful for finding like.
Edge cases that you failed the test, right? But it's not a panacea. It's not going to fix all of your problems. There are still plenty of bugs that mutation testing is not going to be able to detect.
And this brings us to the other of these hypotheses, which is the coupling effect hypothesis.
Which couple of effective as the claim is that complicated faults are quote, unquote, couple, simple vaults in such a way that it last week they will take all simple bots.
Uh, would also detect all the complicated ones and.
As I was just explaining at length, this is not really true. Right. Um, so I say, is this true? Um.
That said many people claim that it is, and they use evidence like this. Like, for example, that's the defect. Simple mutants. We're also able to take over 99% of 2nd and 3rd or review historically. Right? This is a big study that was done.
Does this make us believe that the coupling effect of processes is true?
Why, or why not it is it possible to do something.
Not as because I.
Yeah, well they're also just a 2nd or 3rd or mute is just.
2 or 3 small changes right? Is that it's 2 or 3 small changes the same as.
1, complicated change, like, 1, specification bug.
Well, we still don't know that. Right? This doesn't actually address that question at all this answer the question. Do we need to do 2 mutations or is 1 and now.
Right but it doesn't answer the question. Like, can we just have complicated bucks? Right but this is the best evidence that we have for something like this.
And that's why I'm going to argue that. I don't think this is true.
I don't think that the coupling effect hypothesis is true. This is a controversial position that you can find, like, professors at eminent universities, who will disagree with me about right? But I find this evidence on convincing. And the reason is that I don't think that a 3rd room is particularly close.
To the kind of real fault that mutation testing is going to miss anyway.
Yeah, sorry just.
It's great. What what what you say it again or.
Yeah, uh, the of the coupling effects hypothesis or of this.
Well, okay, so the problem of hypotheses is arguing that it, uh, mutants, like, simple.
Are going to if you can select all of them, you should be able to check all the complicated bugs to.
Basically that, um, for every complicated bug in the world, there's some small mutants that if you could detect it, you would also detect the complicated book. That is to say, if you write a test that detects the small mistake.
That test would also fail if the complicated mistake was made.
All right, this is the argument that is made by components of this, uh, this hypothesis that because, uh, higher order meetings can be detected, uh, by the same test that can detect low order meetings.
This must be correct. Okay.
I am arguing right now that, uh.
Detecting 3rd or Newton's is not sufficient, right? Yes. This means that it's not necessary to test for our meetings. We can test single remittance instead, but this doesn't tell us anything about bugs that are so complicated. We can't express them in terms of meetings.
And depending on how big a percentage of the bug is in the real world, you believe that is.
That should change your belief in how, how much this matters right?
If most of the bugs in the world are so complicated that no combination of mutations of these sort of small mutations that we're talking about, could ever produce them. Right? They're more like the, uh, I give you, uh, the altitude and feet and you expected meters kind of bugs, right?
You think that's the most bugs then mutation testing but this is this is false. And mutation testing might not be as useful.
All right, if you believe that very few bugs are like that and most bugs are typos.
The mutation testing is likely to do. Well, right and so, uh, like again, I'm, I'm.
Being clear that this is like a point of debate within the community and so I don't want to like.
Tell you that, like, 1 of the answers is correct, because I don't know. Right but I, I am not a believer in this, because I believe that more bugs are like like a complicated thing that involves a specification mistake or like, human miscommunication.
The bugs that actually matter in the real world, I should say, right? Rather than typos yes. For a type of bugs and mutation testing is excellent for finding them. Right? But it's limited to just that.
So, yeah.
Our, uh, depending on the hypothesis.
Yes, yes, so, in in particular, uh, if you read the extremely tiny, uh, uh, citation down here, right? The title of the paper here is investigations in the software testing coupling effects.
Right. Which is exactly this couple of weeks head office. All right.
Um, and that that's the claim that this is evidence that that is true.
Sure, yeah. Mm. Hmm.
Uh, I think because they still believe in mutation, like, they still believe that this is true, right?
I would say, I mean, I haven't specifically asked Jeff offered about that, but I suspect that he would say, yes.
Yeah, that he believes mutation testing. He continues to publish it now. You don't have to be, right? Yeah. Well, I mean, I think I think it's it's it, this is a genuine point of of of debate right? Like, this is the, uh, the evidence that I know. But I'm not I want to be clear I'm not the expert in this field, right? Like.
In particular Jeff office, the guy who publish this, right? And who, as I said continues to work mutation testing is a professor, a senior professor at George Mason in Virginia. Okay. Um, I suspect that he probably has better evidence than this right now.
But I have not seen him or anyone else who's depending this make a convincing argument better than this 1.
That's, uh, the couplings effective. This is true.
I also want to be clear here, though, that this does not mean the mutation testing is useless. Right? And that's an important takeaway here. Right? You shouldn't think that just because the coupling effects hypothesis is not necessarily. True. Does not mean that mutation testing is useless. It just limits its applicability. Right? Jason testing can still be useful for finding small mistakes.
But it's not going to uncover, like, bigger and more complicated faults. Right? At some point, a human has to be responsible for writing this specification.
Okay, any other questions here I think I have 1 more note here. Yeah. Yeah. So, uh, this is basically the argument that I've been making our hire, or maybe it's a good proxy for real complex bugs. I argue. No 1 could argue. Yes.
I know the word proxy, by the way surprisingly a few of you. Okay. Um, so just, uh, for anyone in the room who doesn't know this word, a proxy is a, a measure for something that we know is not necessarily a perfect match, right?
So, for example, I know about right.
Is a good measure of whether someone is healthy or not.
Not particularly right, but but it does have some correlation. Right? And so, like, BMI is a good example of a proxy for how, uh, how healthy you are, you can have a BMI that suggests that you're very unhealthy and still be healthy. Right? You are a big weight lifter and so you're gigantic on top right? Um, but you can also if you have a very high BMI, it might mean.
You're very overweight, right? So, like, some of the time, it's good. Not all of the time. If you go to the doctor and you are a weight lifter and then they, they tell you that because your BMI is high you need to eat less.
You should say you're not a very good doctor, because you don't understand that this is only a proxy and not the underlying truth.
All right. Okay. So, yeah, so this, this, uh, measure is proxy measure, right?
And real complex faults are the real truth.
But as I said earlier, like, we don't really understand what all the real complex bots are, we don't really know or we don't have a great model for them because it's still an open area of research. Right? People are still working on this. So.
Anyway, this is the fun part of being in an, in an area that has a.
Ongoing active research. Okay. Any questions on this slide before I move on.
And to double check the time okay, we're doing fine. All right so then, uh, yeah, jury is still left. Okay.
Mm, hmm.
So, I, I might, but I wouldn't expect you to, like, give, uh, I like as as good of an answer as, for example, I would expect myself to give given a day of time. Right? Um.
I might ask you, like, did you believe me when I made this argument in class.
All right about this specific argument, like, or was there a point that resonated with you or not? Right?
But I probably wouldn't ask that as an exam question if only because it's an opinion.
Right I would ask you to, like.
Like, about an assumption and say, like, do you believe this, like, justify why or why not but I'm not asking for you to come up with, like, a citation in order to defend your position right? Then I will be basically, like.
I will evaluate an answer basically only on your understanding of.
Like, the thing you were arguing for.
Like, for example, if I say, like, imagine, the coupling effect is false, and then you give an answer, that's totally not about mutation testing at all. Like, that's not a good answer. Right? If you have an answer, that's like that says, well, the coupling effective process is important for mutation testing. So so mutation testing will change in the following ways.
And those are reasonable ways that make sense given what we know about the effect hypothesis, then you'll get a good score. That's the sort of question that I'm talking about.
All right, so we've talked a lot about theory here. Yes.
Yeah, so a 4th quarter mutant means that we've applied to mutation operators to it.
So, for example, um, if we have, uh.
If something like, if X is greater than 0, um, Y, equals X +5 right? A 1st, sort of meeting might be changing this to greater than equal to. Right. A 2nd order meeting is then.
Making 1 additional change so, for example, we might change this, this 5 to 1, right? And now it's a 2nd order meeting and then we might change, uh, this, uh.
This 0 to 3, and now it's a 3rd meeting because we made 3 changes.
And so on, and so the order is just the number of mutations that we've applied.
How would the sentence to me? That's.
Well, uh, the argument here is that, uh, if you can detect a single, a single order mutation.
At the same time, we'll detect, uh, 1 orientation and orientation and the priority mutation. And if we imagine the fall bugs are made of many mutations.
Then, maybe this argument extends indefinitely and so any bug that's made of 100 mutations will also be detected by 1 that.
Detects just 1 yeah, I mean, and to some extent, that has to be true. Right? Any bug is made of some number of mutations. The question is, are they the kind of mutations that.
Mutation testing tool will will produce and also, uh.
Is there ever a break in that chain? Good question.
All right. Okay so, um, we, we've talked a lot about theory and I want to before we go too much further. I want to talk about a specific concrete example here. The reason I didn't do the concrete example a little bit earlier was that there are a couple of more complex things that I'm going to talk about inner leave with this example, right? Including.
I believe equivalent meetings, which you mentioned earlier right? So consider this program. We have a minimum a minimum program, right? This is written in Java. It takes 2 and B, and it returns a is listen B and B otherwise.
All right, everyone understands this in this, uh, customer call on the syntax.
This is a front of a printer operators. It's just basically an ancient statement form.
Okay, so, um, well suppose that we have this original program, let's further suppose that we're going to make some mutants right? So suppose that our 1st meeting deletes the less than B, part here, and just returns a all the time. Right? So that's more than 1.
We have mutant too that I also just returns. B, right?
We have 3 which switches the less than to a greater than or equal to.
All right, so, uh, the effect here is that basically it's going to return a, when we originally were trying to be and return D when the original word would return.
All right, and then we have this 1, uh, which changes the less than 2 a less than or equal to.
Okay, so, um, we have these 4 meetings, right? And now, um, I want you all, uh, to, uh, turn to someone scenario and take 5 minutes or so.
And provide a test case that detects each of these meetings.
Hello.
And this is an excellent excuse for me to run to the bathroom.
Okay.
Hello.
All right, I have a test case that can detect and 1.
Surely some of you at this point no.
To any test case to detect.
I think we've given it enough time, so let's let's go ahead and, uh, and start considering it.
All right, so, um, here are, uh, let's let's fill in some, some test cases here, right?
Ah, darker color here we go.
All right, so if someone's in goes to me.
Values for Andy that was the title is 1 thing.
Phase 1 phase 2 sounds good. All right. What is the original compute? And 1.
Right so what how does.
1, it returns a, but does computer.
2, and we're trying to see what does compute recall and the 3 is.
Right so to, and then 4.
Wow, okay. So which means that we detected well, which means different from the original.
Yes, so if you take to them to detect them 3. excellent, good work. Right? So, uh, if this is our entire test date, what's our mutation score?
Which means it's probably worth it, so.
We'll get that yes, our mutation score here at 50%, but just this test this just this test case. Right? And the question was, is that does that mean it's a good reason or a bad view. This is a good test case or a bad test case right?
Well, it doesn't, it's hard to say, right and, uh, we're gonna go into, like, through this in quite a bit of detail, like, further on in the lecture when we talk about actually using this in practice. Right? But I would argue that, at least right now, uh, this the fact that these 2 meetings are killed was probably a good thing. Right? This test is doing something useful.
All right, you sold them 1 of them 4 left. Did anyone think of a test case that will kill? Em, 1 around 4.
I still wasn't 1 to 1:1:okay. Let's try 1 1.
All right, so on 1:1:uh, what is the original? Do.
Mm, hmm.
1 from 1, right? 1.
What does that morning? Yep, cause it records I, what does that mean? If you do? 1 again? What about.
1, also 1 yes, so this, this detect any minutes.
Yeah, right and, uh, this is, uh, this kind of thing is quite common right? So, uh, when I saw this table in a minute, you'll see that the 1st customer I have here is just 1:1:example. Right? This 1:1:example is a good 1, because it doesn't actually fill anything.
All right, um, it doesn't mean it's a bad pass necessarily, but it's not a useful test in terms of mutation, cluster mutation customers that the value of this.
Again, that doesn't mean it's times, but at least here, we're not seeing any value from it. So it depends a lot on what the meetings are. Right? Whether they are the same data center.
Alright, I heard some other suggestions. What is the technical kill either for.
2 and 1. good. Okay. So, uh, in this case, we're going to get what.
1 error right and 1 is going to return 2.
Right. And then we know these are dead. But, uh, what about this? 1.
1.
So, it's less than or equal to itself. Okay. Um, so, uh, in this case, uh, we can kill them 1. all right but we still felt the California.
Why is it that we keep failing to for?
When number sequel.
Please please stop so.
Yes, you're getting close to the answer, but you're giving an argument for the answer rather than saying the answer directly.
So, why is it why, what is what is wrong with him for? What is the problem with him for here? Yes, it's equivalent, right? That's right. Yeah. It's semantically equivalent because the technical term, right? That is to say, uh, it's meaning it's always the same output will never get from the original, right? Because we're either returning a, or B and the only case for this.
1st, is when a is equal to be, and so as, uh, with being on here, I'm.
The answer here is never going to different and so this is an equivalent and it's semantically equivalent to the right.
I'm going to race, or I have here, but I'm going to spend fill it in using the, uh.
The slides and it will be the same set of test cases.
Because I correctly predicted those 3 type case would be the ones that we discussed.
All right, so, 1st, the all one's test case doesn't kill any meetings right then 1:2:all right? And you can see that it will kill, em, 2 and 3, right? That's the 1st 1 we showed.
And then 2:1:which will kill and 1 and 3, but will not kill, right?
So, did anyone find a test case that can affect them for? Does such a test case even exist?
No, all right and this is what's called the equivalent mutant problem. This is 1 of the key problems in mutation testing in general is.
The problem equivalent means so suppose you have and why, and you swap those 2.
What does that do to the the program's output? Sorry? No. You smoke both the swap the 2 statements.
Right so 1st, you do why it was and then you do.
Perfect right if you just reordered the, the order of operations, but they're the same operations. And so the result is the same at the end all the registers have the same values, right?
Yes, so this is an excellent example of an equivalent mutation, right? And there are many it is very easy to accidentally mutate a program in such a way that its output does not change. Right?
So the resulting program would be immune, but it's semantically equivalent to the original and therefore cannot be killed.
Right it is. It is immortal. No test case can kill it because it by construction, it will always have the same result as the original.
Like, we can construct a proof that it does the same thing, right?
And 4, in the previous example, was also like this, right? And when you all are building your mutation tester for homework, 6, this is something you should definitely be on the watch for you will certainly create some right? And you have a limited budget of mutants to deploy against the, the grading server. And if you spend them all on equivalent meetings, you.
Not make very much progress at distinguishing the test suites, right?
So, um, effectively what this does is it dilutes the mutation score, right? Um, if we look back at the previous example, right? Our mutation score here with these 3 test cases, or even just is 2 is 75%.
Right. We can kill 75% of these minutes, but there's that's actually the top limits that the upper back we can't do better than that. There's no task for killer for. Right?
So that means that, like, just achieving 75% like that, that sounds not so good, right? But it's actually impossible to do better.
So a priority you don't know what the upper bound of mutation score is.
All right, if you're just looking at a program, right? And you're thinking about mutation testing, there's no way for you to know whether you've actually achieve the best you can without, like.
Carefully reasoning about each of the mutants and checking, whether it's equivalent to the original.
Because you don't know whether each each meeting is an equivalent, or whether it's a meeting that you just haven't been clever enough to kill yet.
Yeah, so for the assignment, you're going to be creating buttons, right? The assignment this week. Uh, is that, uh.
I have a, you are given a simple, a Python program, right? And I have 5 test suites for it.
You will see you receive full marks if you create a mutation testing tool that, uh, gives different mutation scores to the 5 test suites in the correct order based on how good. I think they are.
In particular, the 5 test suites are subsets of each other. Right? So.
Test a is strictly larger contains strictly more tests than test suite B and so on, right? For.
And so, basically, the way the scoring works is that if you're if you say, for example, that you say that, uh, your, your mutation, uh, things does this.
Something like, uh, you know, 80 for, like, you give mutations for 82 tests, a mutation score 70 that has to be mutations for 62 tests with C mutation score or 50 to and 42 8 you would get full marks. So, the challenge is.
Great test cases. Oh, no. So these are these are test suites. They contain test cases, right? So your challenge is to to figure out mutation operators that will.
Show that these tests are different. Yeah, yeah. Yeah.
No, you're going to write a program Yeah Yeah, you will write a program and, uh, in particular these test suites are held out.
So you don't get to see what's in them, right? So you have to build a mutation testing system. That's effective in general. All right.
Okay, so this dilutes the mutations for right? Uh, we don't know whether 80% is the best we can do. Or if we write better tests, we could get higher. Right?
And I understand the confusion here. Right? I'm talking about both tests and.
Like, building better mutation testing, right? And both are valid goals. Right. Uh, so 1 reason you mind you mutation testing is that you want to distinguish test today from let's just say you have and you're not sure which 1 is better.
Right. Then you might care about building better mutation operators and so on. Right?
Or you might be in a situation where you have only 1 test suite and then you'd like to write a test to increase your mutation score because you want your test suite to be better.
All right, and that's the situation I'm talking about when I say it, they lose the mutation score.
If you just have 1 test lead, and again, this is not this this is not what's going on in the homework. Right? But if you just have 1 test suite.
And it has 80% mutation score. Is that good?
Because of equivalent mutinous, there's no way to know right?
All right does that distinction make sense? Mm. Hmm.
Yep, okay. I'm sorry for the confusing terminology I know that mutation testing is like this that's.
Any questions on mutation scored dilution for a move on.
Yeah. Okay. All right. So detecting these equivalent means is a big deal, right? We would like to avoid the invitations for our possible, but if we can, we'd like to remove our equivalent meetings and not count them.
All right, if we could just say, oh, notice that these meetings are equivalent, remove, remove them from the denominator, then this would be a problem.
Right. So, uh, how hard is this.
Quite hard why.
Or put another way, how would you demonstrate that? It was hard?
So, 1 option would be that you could go off and try to do it and then you could come back to me and say, well, I spent 6 hours and I didn't succeed. Right? But let's consider plans. That will take less time.
Is there a way you can demonstrate that this is hard.
So, we could run the test suite and we could see whether it gives the same result. Right? But that's a.
That's not going to tell us the equivalent means all the time.
All right, that's the best in approximation. Right because for the same reason, that testing can only show the presence of bugs, not their absence, right? There could be some other test that would distinguish them and we wouldn't know. Right so I'm, I'm thinking about this in the sort of formal sense of this problem, detecting equivalent meetings.
How hard is it if we know the program you think.
I would put it to you that that is not knowing the program is not going to be sufficient.
Right and here's why so I've mentioned reductions in a few of our previous classes. Right? This is an excellent time to do 1. anytime you see a question.
How hard is this problem should be a hint, but the thing that you should be doing is thinking about other problems, and trying to figure out which of them, you could solve if you could solve this 1.
So imagine that I give you a machine that can automatically tell whether to mute sorry? Equivalent.
Sorry, the other 1 is equivalent to it's original program. Right? Mutation is equivalent.
How how could you use it to? How could you use that machine to solve some other problem? You know, to be hard.
So the answer is that is undesirable, right? I'm just going to give you that. Right? There's no algorithm for. It will always. They'll always give the correct answer. Okay.
The way that we prove this is with a reduction to some problems that we know is undecidable.
What problems do we know that are? undecidable? Sorry?
Yeah, so you're suggesting Boolean satisfiability right? Yeah. Willing. satisfiability is computable though. It is.
Exponentially hard, but it is computable. It's going to be complete.
What's the most famous undecidable problem it involves? Whether to pro or whether a program does something whether it's the program.
Related to this yes, whether the program terminates or not the whole thing problem right? So we can do this by direct productions to the whole thing problem right?
So, consider that we have a magic machine that will always return, whether P1 is equal to right this is our promotion assessment. We then right? The following program, right?
Uh, we can find if you want us, uh, in both, then we return otherwise we call ourselves again, right?
This program holds exactly when our equivalents tester returns. True.
Right. And so this is a, uh, like this.
Little program, it is a proof that, uh, this is equivalent to the whole problem.
Right because I, if we could, if we could solve this problem, this, this bot, the problem, right? You could tell when this is going to be true. We know whether this program hotspot.
Okay, I, I, I understood that.
Basically nobody. All right, so, uh, can someone asks me a question that will hopefully, uh, push us in the right direction here?
Every time with you, what is it.
Right, so the whole thing problem is, uh, given a program, uh, does it hot.
Right or does it run forever? You have to tell me my quick as he can.
Right so it does this program hold or not.
Right, well, it depends like so the reason the corner is interesting is because whether it's filtering on depends exactly on whether we know the, whether P1 is equal to P2. Right? And so, if we have a secret machine, that can prove that is equal to P2 all the time, or not instantly, then we will solve the question question right? By simply client.
Here, when you say, oh, well, we know if you want is equal to and so this will work from 0 or we know that is not able to be to. And therefore this will look forever.
So, we could instantly solve this problem. We can certainly solve the whole problem on this program and therefore.
We know that this problem, the equivalent of problem of telling her.
1 mutant is semantically equivalent to the original program right? Is undecidable. Right? Let's just say it is.
Uh, it it is as hard as solving.
I mentioned you're also, um, this is a, uh, something that will come up later, uh, quite a few times when we talk about static analysis, right?
Racist theorum says that any non trivial semantic property of a program. Right? That is to say it's true of some programs and false of others. That's what non trivial means.
And semantic means it's about what the program does not about, like, whether it's text contains a particular string.
All right. Um, so is there any program or any property that satisfies those 2 properties, uh, that programs? It's statically undecidable just to say without running the program, you can't figure it out.
Without, uh, he also being able to solve the whole day problem in all instance.
The, the, the proof of is basically exactly this, but if you put whatever the, uh, the property that you're interested in, in this in this position.
So, when we start talking about, like, does this program have no point or exceptions we want to get an answer? Yes. Or no, the reason that that is hard is.
That is also a non trivial semantic property for program. Just like, is this program equal to this other program?
Okay, so this was a deep cut on program analysis theory, but this is important. Right? So, the 2nd, equivalent means is very hard, right? In particular, like, theoretically, it's just as hard as doing the whole thing problem and so we're not going to be able to solve it easily. Right there not be an algorithm. We can just run that will tell us like yes. Is this a new news equivalent or not?
So, um, we still would like to know right? And there, in fact, many times to detect equivalent meetings.
These are all going to be approximations. Right? They're all going to detect some equivalent mutants, but they're not going to promise to detect all of them.
Or something like this right? And this is a, I think, a good time to bring up that this is like, um.
A problem that's undecidable is to software engineering or programming languages researcher like me. Exactly. Like a pot of honey sitting on the table in the summertime is to a bunch of flies right? They immediately appear. Right? So, um.
Undesirable problems, attract software engineering researchers are researchers who are trying to find a good approximations. Right this is the key. Currency of our research area is finding undecidable problem and then get as close to it as we can, uh, while still giving some promises about the result. And W, when we get into static analysis, it will become clear why I say this.
This is a sort of a hint about what the 2nd, half of the course will be about. Right?
So, I'm going to I'm going to discuss 2 here for the equivalent muting problem, just to give you a sense of the space. The 1st is called.
Uh, trivial compiler equivalence, which is a rough. Approximation is very cheap to compute and the 2nd is a more precise, but more expensive, uh, approximation that you can do the introduction to because we'll have some stuff in common with the discussion. We had a symbolic execution a few weeks ago.
Um, so 1st, uh, actually, so before I go on, uh, does this, like.
High level thing about what software engineering or programming languages researchers like to do make sense.
This is, this is the thing that we do, the coin of the route.
Okay, so then let's talk about right so, uh, TC, is an equivalent detection, uh, technique, uh, that shows the 2 programs are equivalent, uh, by, uh, running an Optimizer compiling on both of them and then getting the, uh, the minor.
So, uh, if we have if I have an original and I have a.
And then I pass them both 2 with Dash or 3 flags.
Jesus is going to try its best to remove any extraneous computation that that program does.
And then produce a fast binary right? But has had many years of engineering put into it.
So that when you run it with Dash 3, the binaries, it produces are semantically equivalent to the input program.
All right, and so if we take say 2 C programs, 1 of which is a mutant of the other.
And then we run them both through GCC with the dash 3 flag and we get the same binary out. What does that mean?
It means that prove they were equivalent. Right? And so that's basically how, uh, works right? If the compiler optimizes the way the difference you introduce.
In the meeting, then those programs must be the same, they must be semantically equivalent, because it must be the case that the compiler has proved it under the hood. Some clever program analysis researcher did all of your work for you, and you don't have to do anything right? Um, embarrassingly this was invented in 2015.
Right. Um, like, this is what I mean, by like, uh, program analysis, like, is a, uh.
It's a fun domain to work in, because surprisingly easy things have not been tried. So this is the key idea of a trivial compiler equivalence. Right? And this is taking advantage of existing analyses that are built into compilers means. It's cheap. Right? All you have to do is from pilot programs, which if you're going to execute the test against these Newtons, you must have already have done. You already have to be planning to compile.
If you were going to suppose, you have a mute and you're about to run the test suite on that meeting, right? If it's a C program, you need to compile it 1st before you can execute it. So you're going to compile it anyway.
So you may as well while they're added.
Check, whether the binary that produces is the same as the, uh, the input program's binary or, uh, any of the other binaries that you've produced before for any of the other meetings you've looked at. And and this is the embarrassing part in in experiments tcp detects about 30% of all equivalent means right in a controlled.
Experiment where someone did approve for each of the equivalent mutants right? And looked at every meeting that was produced.
This is a shockingly large number, at least to me, right. Uh, that something as simple as as TC will detect 30% of the equivalent meetings. It's pretty good. Right? It shaves off a pretty large portion of them.
Yep um, and this will also detect for on meetings and when we come back for redundant meetings.
Should keep in mind like, how would you say you detect that? Does it make sense?
Yeah, I would say that this is the, uh, the state of the practice in mutation equivalent meeting detection in the sense that, uh.
Like, if you are going to do mutation testing in in real life, like, it would be foolish not to do this.
Right. Everyone does it the stuff that we're going to talk about on the next slide.
Is much more sort of speculative, right? I, you know.
People doing research, because they think it might work out rather than like, what a company would do right now.
Okay, so another option here is that we could try to prove that meetings are equivalent by doing reduction to satisfiability module experience, right. To say we're going to try to call a soul. Right? We talked about solvers.
We did, yes. Okay. We should. We did right but as a brief reminder and solve our solves bully and satisfiability, even in the presence of a.
Things like linear arc MTech or a bit vector or or something like this right?
Okay, so how would we do that all this is similar in spiritual symbolic execution right? Like symbolic execution we, uh, we also were doing mostly by reduction, right? But instead of asking this over, like, is an input is there an input that lets us reach this particular branch?
We ask, is there an input that causes these 2 related programs to divert? Right?
So imagine that we have a, we have something like again something like if X greater than 0 dot dot dot.
All right, and then we changed that great. I greater than to a greater than or equal to.
Uh, basically, what we're going to do here is we're going to take, uh, the kind of path constraints that we would have used to, uh, generate an input that would reach this branch.
And we're going to pass them to a solver along with this, uh, constraint and say, is there 1 that would work for this 1? But not for that 1.
Does that high level idea makes sense so, in particular suppose that we have these 2 this example right? We have X rated 0 expert and equal to 0, right?
The solver is going to have to come up with some input that causes X to be 0.
Right. That's the only way it can force these 2 to diverge. Right but effectively what we're going to do is solve for the path constraint of both of these together right with the additional constraints that it has to go into the 1st, 1 without the 2nd.
Or vice versa, right? And again, like, this is via the same conversion process that we used, when calling the solver in smart execution, right? We'll talk about exactly what the details of that conversion process like, in, like, week.
The 2nd, to last week of the semester and then, I guess, okay.
But, for now, just understand that, like, what we're, what we'd be doing here is we are just trying to force this over to take us down 1, but not the other of the of the 2 paths, right?
Okay, so the key problems with this 1st applicability, right? Many mutations are difficult to reduce to, in particular things like modifications to the heat. Like, if the mutant instead of writing to a field, uh.
Like, chooses not to write to that field. How do you, uh.
Encode that in all right? You can't really encode that as a constraint easily.
If the mutant is modifying a like a conditioning statement, it's pretty easy to encode it as a path constraint. If the mutant is modifying, like some value that's stored by the program that's very far to include as a path constraint, because it's actually not path related at all. It has nothing to do with control flow. Right? And so this is.
This is a key problem here. Right? And this is a problem that has basically not been solved yet.
So this works well, for certain kinds of mutation operators, like, for example, uh, changing greater than integrated or equal to inside mid statements, it works great. Uh, very easy works every time right? But for other kinds of mutations, like delete a statement, it works basically never.
And secondly, I s, T solvers are slow.
Uh, solving is not, uh, like, they're fast on most examples, but this is a pretty complicated example. Right? Effectively what we're doing is double, the amount of of clauses. That you get in symbolic execution plus some. Right? Because we need, uh, the path constraints for X, greater than 0. we need the path constraints for.
Or equal to 0, and we need a constraint that says the result, the answer of these 2 can't be the same. Right? So, it's about 2:2:times 2+a constant.
For how much how expensive is relative? Sorry? For how expensive? Symbolic execution would be.
Right that means since we know that some more execution is too slow. Sometimes.
That this will be too slow about price is off right which is not great.
Um, fortunately though, there is a, there is a trick, uh, and that's the citation here is this trick. Uh, that will allow us to to get around this at least some of the time, which is, uh, uh, these 2 sets of constraints. The, if X is greater than 0 and FX is greater than or equal to 0, have a lot in common, they share many pauses.
And so it's possible to do some clever stuff with the, like ahead of time before you pass it to the silver to eliminate duplication.
It's still very slow though. All right any questions on this does this high level strategy make sense?
Make a make a math solve, or do the problem for you. Okay. Um, as I said, this is still an area of active research. Um, I know for a fact that there are people working on this as far as I know. This problem has not yet been solved.
So, um, let's return to our example, and I've now crossed off. Right? I would assume that either or, uh, an solver or some other massive MySQL magic machine has told us that is equivalent and we're no longer going to consider.
I want us to say, is there a different question are all of equally useful right? In other words, is it possible to predict if any of them will be killed based on whether the others are.
And because it's cover up here, we're called for 20.
It was possible, obviously doing that number.
So, yes, all right. Um, so in particular, her returns always exactly the opposite of that.
So, if it's greater than the for friends, hey, uh, which is exactly the opposite of what, uh.
And 1 return alright, thanks. Everybody. Uh, the original 1.
But that's not quite the relationship between the 3 meetings.
In particular the relationship here is that everyone always reference a new models.
In a matter of time, and we can actually just ignore this condition.
And 3 always returns either a, or B.
Right. So, for any input, either a, was returned by, or B was.
Yeah, because we know that I'm 1 return day and to recurring fee.
So, if a process, if if returning a causes that can be built, then.
And the, for turning be positive, they needed to be held and.
And so is actually what's called redundant with.
Right that is to say any test case that kills and 3 must also kill either and.
Does that make sense to everybody? This is extremely important to graph before we move on because we're going to get into some.
Kind of like, more complicated things that are built on top of it.
Okay, so 1st, let's give the definition.
All right, so I said to be redundant, if its outcome can be predicted based on the outcomes of other meetings.
So, if it's possible for us to predict, that will always be killed, because everyone was killed or who was killed and we don't need them 3. we can just run, em, on an empty.
Redundant means tend to inflate the mutant detection ratio or mutation score. Right? That is to say, they make you make it look like we're doing better than we actually are. Right? If we go back to this original example here.
Right. Original mutations for here was 75%, but actually, and was, uh.
Equivalent so if we remove them for, then now, it looks like our mutation detection ratio is 100.
But if we go back again, and we don't know that I'm for equipment, right? At 75%.
Well, also is detected anytime that we have to detect either in 1 or 2 so we don't really need a 3 so we can get rid of them 3.
And then our mutation detection score, actually, if we include them before, it looks like 2 thirds.
All right or done for, it looks like 3 quarters which 1 is right?
Well, it's hard to say, right isn't actually providing any value, but it does raise the mutation score in the presence of equivalent of.
Um, so it makes it look like we're doing better than we are. Right um, and they make it hard to assess progress, right? Like, it might look like your mutation score is going up, but.
Actually, it, it, it's not making a difference, right? Because, uh, like it just because you like it if you.
For example, you write a new test like, let's suppose here we have just, uh, just this, uh, uh.
It's supposed to be done none of these tests. We had just the 1st 1, right? The 2nd 1 here and we kill 2 minutes.
We feel like we're doing very well. Okay. But actually.
Are we, we really only killed 1. interesting.
But it seems like we've killed it right? And so it's a deceptive it's deceiving us a bit about whether it's useful or not and we're wasting resources on it.
All right, so it's bad in 2 ways. Um, and so the, the question is, can we formalize this notion? Right um, and the hint here is that, of course, we can, uh, otherwise I would not be asking so let's do it right? The way that we formalize this notion of redundancy is with, uh, what's called mutants assumption, right?
So, uh, I'm muting. Em, 1, is that just assume another meeting if, and only if some test skills and all the tests that kill, em, 1 also kill em, too.
All right, actually, before we do, we know what this means.
Definitely, yes, yeah. Math words. Okay. So, um, what's wrong with this definition?
Well, okay, let me rephrase like, this is a fine definition, but what's going to make it difficult to tell whether its assumption is true between 1 and M2 and practice.
So, for example, to get it, why this is hard. I claimed.
That, uh, M1, and together in the the example here.
Subsume all right. How did I demonstrate that to you?
What did I argue.
But think about the argument I had to make, I, I came over here my main an argument that.
Well, uh, this part doesn't this part of the program doesn't matter the recurring values are in some set and then like, this, these 2 cover that whole set, right?
Importantly, it was an argument about program semantics. There's an argument about the meaning of the program.
And that should be a big hint and this is going to be a very hard problem.
Well, exactly the reason that we saw before we, the equivalent meetings, right? This is obviously going to be another case of that is to say, like, whether or not, uh, a particular moon is, uh, subsumed by other means.
And, uh, going to be a, and also an undesirable.
It's going to be just as hard as technical meetings are equivalent.
So this is this is going to end up being difficult. Um, and in particular, the hint that this is true is this all tests here?
Related to the right what are the, all the tests, man.
All possible inputs yes, exactly. And we know that that's not going to work out.
For us, right this is the extract coming back and saying, remember testing can only show the presence of logs, not the analysis, because we can't write all the tests.
All right, but this definition requires us to know all of the tests in advance.
So this is what we call a quote unquote. True assumption. There it is.
Um, true assumption is difficult to check in practice, whether there's another why we've already talked about it. Right? It is also undecidable right? All test means the set of all tests that could possibly, uh, exist and this set is usually infinite. When is it? Not infinite.
We've talked about some examples in particular. I'm kind of looking for a key word here. There's a phrase we gave to the situation when this is not infinite.
I'm starting to write it on the board. Yes. Hello.
Sample it's an age. Sorry oh.
It's he goes to Austin, exhaustive testing. Oh, Josh exhaustive testing is the answer yes. It's exhaustive testing, right? If there's a finite number of possible inputs, then we can test all of them.
All right, so, for example, if we have a function that takes 2 billions and returns something, right we can simply test all 4 combinations of 2, billions.
Very easy, barely inconvenience. Right. Uh, but, um, Internet, as it usually is right in practice, the steps are either inputted or still orders to be infinite. Right?
Then we're not going to be able to easily Checker. Right? We won't be able to do exhaustive testing.
Okay, so can we can we still make use of this idea of assumption in practice? Of course, the answer is going to be yes, right. So what do we do when we face undecidable problems.
Recall as a software engineering researcher, what am I going to do? I'm going to go straight attic. Right immediately. I'm going to jump straight for the undecidable problem. And what am I going to do?
Closest to the answer correct I'm going to try to approximate right and try to get as close as we can. So, what can we do instead? Um, so the technique that we usually use is what's called dynamics assumptions. So given a finite set of tasks.
The new 1 is set to dynamically subsumed another meeting number 2 with respect to see definitely some testing kill them 1 and all the test and T to kill and 1 also kill them too.
All right, what did we do here? Well, we just restricted the universe and possible test for some finite set.
And in practice, what do we usually have a finite set of tests.
We have that it's the test the test. Okay.
So this is quite useful, right? This allows us to keep track as we go of, whether our mutant has been subsumed so far.
All right, among all the tests that we've run right? And if we have a fixed test suite, we know that we're not going to add any test to it. Then this is good enough.
All right, it's a good enough for approximation and it turns out to be quite useful in practice. Right? So, um.
This is exactly a true assumption if, and only if, uh, T contains all possible tests again, that can occur only if you're testing exhaustively. Right? So we said, previously.
And here's a graphical representation of this, right?
So, if we have this, uh, a more of a set of all tests that we can't possibly get to right?
And then we have some tests that kill somebody named Jay and some testing meeting decline and some test to kill somebody with them.
What did we miss assumed by which other meeting in this example?
Yes, that's right. Um, actually, that might be exactly backwards. Uh, M. J. here's the right that is to say, if you can detect always hold the tax deadline.
Because every test it, uh, the other 1 yes, you are, right? Yes, if you can detect, we definitely are impacting that bit. Right? The smaller side. It's the 1 we want that's totally contained inside the app.
Confusing myself yes. So that's exactly right here.
We're correcting, not that anyone needs any more candy at this point, but, you know.
Well, we have plenty, so okay, so we can model this idea of meetings assumption with with a graph, which is called the mutants assumption.
So, the nodes in this graph, represent a maximum set of quote, unquote, undistinguished mutants that is to say mutants, we can't get Tele part right? And the edges represent this assumption relationship.
So, for example, if M1 s, uh, which then systems we could represent using this.
And 1 there, uh, and then since entry, and so if we're going to, uh, choose 1, new thing that we actually want, which 1 is it.
That's right. Yeah. All right so then a mutation testing tool as it runs, can then maintain what's called a dynamic maintenance assumption graph, or that keeps track of are actually subsumed or undistinguished over the test suite that's of interest. Right? So, if you're testing a particular test suite, you can check which mutants were actually.
Consumed over that test suite. Right? And the idea here then is that, uh, you can keep the ones that aren't subsumed.
And you can, uh, throw away the others right? And so this is an optimization. Okay.
I want to be clear here before we go deeper into this that I have no expectation that you're going to maintain a, as you build your mutation testing tool for homework. 6, it should not be necessary, right? If you do want to, you're welcome to. Right, but homework should easily be solvable without this optimization.
Okay, so Newton's here occupy node with any with in degree greater than 0. do we know what means when referring to the graph?
Yep, yeah, it's the number of edges that come in. Right? So this note has integration.
Yep, and then I.
In distribution Newton's would occupy the same note.
So, if we have 2 meetings that we can't yet distinguish or that, we can't distinguish over our test suite. They would just be merged into a single node. And we just choose 1 of them.
All right, so then, um, here is an example, I've stolen this figure directly out of another person's talk. So, sorry about that.
Um, we're proposing that we have a matrix over here, uh, using this on the top and 1 down 5 and tests, uh, test cases here. T1 2:3:4.
Right, and these check marks are suggesting, uh, which, uh, tests skill, which means.
All right, so, for example, a has been killed by both 252 has been built by teams, et cetera.
All right in that case, we would get the following, uh, dynamic meetings, assumption brand here right?
In particular, I want to point out a couple of things. 1st, I'm just going down 5 or indistinguishable.
I have the same behavior on every test base right? But they're useful for, uh.
Going around, because they allow us to point like, uh, we don't have these, this, this meeting in the 1st week then we might.
I'm sorry, we don't have these tests in the test week and we wouldn't be able to check this.
But we only need 1 of them to it very effectively. The same from our perspective, right?
Similarly is only detected by, right?
Um, and so, uh, it, uh.
Is this, uh, and, uh, tentatively for.
Does this all make sense? And so this is the technique that's used in practice to detect for abundance.
I wonder if that's the reason we're talking about this, this is how, uh, in practice, when we're building mutation testing tool, that rules out with 1 of the meetings and get rid of them, right? As VMs.
Build this draft and then only pay attention to the instead of parodies, outward notes, right?
These written notes are called the quote, unquote, minimal mutants. Right? These are the only ones that the mutation testing tool needs to keep around.
All the others are guaranteed redundant. Right? So this is a redundancy detection technique.
All right any questions on the dynamic maintenance assumption graph.
Sorry, uh, as far as I'm aware, uh, I okay I can't I can't say for sure. I'm pretty sure that, uh, like.
Pit and major probably both implement this. Those are 2 mutation testing tools for Java. I would guess that most of the existing like.
Modern mutation testing tools do under the hood.
All right um, but, uh, I don't directly work on them, so I can't say for sure.
At least in the literature, this is very prominent, so I.
People believe it is important. Lots of paper is published on this topic.
Sorry uh, so it's, it's based on which ones are the, the roots in scratch, right? So anything would integrate here.
Alright, so, uh, if the industry is 0, that has to say there are no edges into it in this graph. Then it's, uh, minimal.
When you have more than 1 reason in the same node for this particular interesting web share, uh, you would just cheap 1, right? Because the test, we can't tell these 2 apart.
From the test to be protected, at least 2.
Yeah, or with the circle and.
Yeah, so.
Oh, oh, you want me to draw something like this for that graph? Yes, we can try.
Uh, let's let's do it over here. Okay, so, um.
We have, uh, the the smallest things are going to be.
5 and 3, which are separate, right?
Then we know that is going to be going to fully contain him. 3.
Right. So let's lead this 1. M1 right and then we know the, the test, the detective are going to fully contain both.
This 1 and that, so it would look like this.
So, just to to explain what I did there a little bit more, right? Like, I derived this structure directly from the structure of this graph. Right? So, um, if we detect here, right this graph means that any test that detects in 3 will also detect. M1, and.
Right. And so that means that needs to be, uh, inside, but, and.
All right, so these are the universe of all tests that detect the maintenance labeled with right?
And all the tests that detect include some of the tech only on for some of the tech and 1 and them for, and some of the detect and M4.
And so, the idea basically, is that if you want to keep around the smallest number of meetings, but still detect as many as possible, what you would do is, you'd select say on the 2 and 3 here and then you'd cover these spaces.
Which would guarantee detection of 1 small identify sequel at 1 time too.
So this is, I want to be clear and approximate. This is an approximation technique, right?
Like, this is also, uh, nothing in particular the.
Oh, here yeah, so these are these are just double circle because they're in the middle of.
Right there was a, uh, I was mentioned office that I didn't include.
Sorry, so I, I want to be clear. This is an approximation checking right? Um.
If this only works, if we have a fixed universe of finance universe of tests.
We need to consider all the possible tests. This technique stops working.
Right, yeah, she those are the points. Yes.
Yeah, so I've shaded the written notes in this now and the double circle ones, uh, in the Sprint.
What was the reason that service and then yes, that's right. Yeah. They don't have other smaller circles inside. Yep.
Any other questions here is this.
So that I can put more slides. All right.
All right, so that's all I'm going to say on the theory of mutation testing, and we're going to spend the remainder of the lecture today and oh, okay. Well, we're, we're doing worse on time than I had hoped.
So, possibly the remainder of class today on the practicality of mutation testing, and how these, like, how these ideas are are useful in practice and what we might do with them. Okay.
So, the 1st, and most important thing here is a question of deductibility versus whether a mutant is actually useful or productive. Right?
Historically, and in the theory that I've talked about so far, we've basically been arguing that the technical meetings are good, because we can re, create a test to kill. So, and equivalent meetings are back. We cannot create a test that kills them. And therefore they're going to mess with our, our numbers. Right? But this is a a very black and white, let's say view. Right? It's a very.
Absolute view and in practice, it is more nuanced than that. Right? This is a bit too simplistic. Right? Um, deploying mutation testing in the real world makes it clear that detectable versus equivalent is too simplistic of a definition and we need something that.
Uh, is, uh, takes into account how useful the, the test it, it'll list it is, uh, limits it, right?
To say there are detectable, detectable mutants that can be useless. Remember my example of if X equals equals now do something with X at the beginning of class right? And their equivalent new mutants that can be useful right? They might not be detectable, but they still might be useful to a programmer. Right? If we show them to the programmer and ask them write a test that detects this, they'll think about.
1 thing that's interesting.
And so, and we need a better and more developer centric definition of usefulness. Right? And this is the core argument of the paper that you all read today.
Right. Was that a developer centric, um, definition of usefulness in particular they call this productive versus unproductive mutants? Um, is, uh, is better than this sort of traditional, detectable and equivalent, uh, dichotomy.
And I'll try not to spend too much time on these slides that we have a list a little bit of time to talk about homework.
All right, so this core question here is going to be is concerning technical utility and really this is possible. Like, we can have this argument about any test equity criteria. Right? So we can have this argument about coverage. We can have this argument about whatever anything we're, we're measuring test suites. Any number is going to be a bit too simplistic, right?
And the, the core question is always about.
Whether it's actually useful to developers or there, it actually convinces developers to do something useful.
So then, um, the definition of of productive, uh, then is 1 that improves the quality of the software under, uh, under test, or of the test suite, right?
By eliciting some behavior from the developer. Right? So this is a very human centered approach. Right? But this notion is very like, this is a very fuzzy and subjective notion. Right? Quality here is very, very hard to define. Right we don't have a, a.
Like, Chris definition for it, because it's, it's.
Fundamentally, all of the things we've been talking about for the most part are measures of quality, but imperfect ones, which is to say they are.
A word that we just we defined earlier today.
They are proxies, right?
Mutation adequacy score coverage, et cetera are all proxies for test suite quality. Right? Testing quality is some unknowable like.
Truth that we can't really get at. Right exactly. All right. All these things we're discussing our ways to approximate it. Right? And, you know, if we allow.
Any particular metric to become our goal, and we lose sight of the sort of more subjective, fuzzy notion the reason why we care about that metric. It's easy to become focused on only the metric and not and lose sight of the main goal. Right? And so, um.
What we're really going to be talking about in this section of this lecture is how not to lose sight of that with respect to mutation testing.
So, the, uh, the Google folks claim that it is going to be productive if either it is detectable and it elicits an effective test. That is to say, uh, there is a test that can detect it.
The test suite doesn't currently contain that test and, uh, the developer decides to actually write that test because it's actually a useful test to have.
Right. Or the meeting is equivalent, and it advances code quality or or knowledge of the code in some way. Right? That is to say it has some other side effects on the developer themselves that is actually beneficial. Right?
All right, and so what I want to do next is kind of go through a series of examples of meetings.
Right and discuss whether they're detectable and whether or not, they're actually useful, right as I want you to be, like, put yourselves in the shoes of an engineer who was asked to try to detect each of these mutants and think can it be 1st can it be done.
And 2nd, if it could be done, is it actually going to be useful to do so.
So, let's start with this 1. right? So, let's take a minute to think about this.
And read over it, read over this program and, uh, write down an answer right? Is it detectable? 1st, like, can you detect it? And 2nd, is it useful to detect.
Okay.
Yes, and feel free to discuss with those near, you.
Be good to try to reach consensus here before we get to it together.
Yes, yeah, yeah. And so to be clear detectable here means, like, we can find a test that kills that.
There exists a test that kills it is not equivalent.
So, in particular, uh, when I ask you all a, in a moment, whether this is detectable, uh, I will expect those of you who say, yes. To provide me, like 1 of you to provide me with a test case.
So you should have 1, if you want to claim it's deductible.
By contrast, if you claim, it's not detectable. I'll ask you to provide me with a proof.
That might be harder to come by.
Sorry, the, uh, the way.
Yes, that's true. Right. Yeah. So for sure. So, um, this is definitely detectable, right? And we know that, because if we choose a a.
Uh, an array that contains a 0 part way through right? 1:2:3 00:4:or 5, right? That will have very different behavior. Uh, if we, uh, use the 1 on the right then the 1 on the left.
All right I think some stores 0. mm. Hmm.
Oh, yeah, great point. So, keep it.
You know, there is.
So like any non 0 input yeah. Great point. Yeah.
You're right so, yeah, so this is very easy to detect. Right? So, the question then, is it productive? Is it useful to have a test case that distinguishes these 2?
So, if so, for example, suppose that you're an engineer at Google.
And you are trying to commit the code on the left. All right.
You've written this program and during code review, the mutation testing system says.
This test, or this muting was not killed by any of your tests.
Is that a useful piece of information for you to have? Why not?
Well, that's true but.
Status the same guy.
Yeah, so the 1 on the right does the meeting does return the same value no matter what. Right so, if this isn't killed by your test suite, what does that say about your testing?
So, it must be really awful. Yeah, it must be really, really bad. So, actually, this meeting is quite productive.
Because, uh, if if this is if the case that this meeting didn't get killed, then you, as a developer have really messed up, right? Like, you should have not, like, written any tests that actually exercises that your program doesn't correctly.
Any tests, so, in fact, this is a useful mute shop, right? If you, as the designer of mutation testing system are trying to elicit useful tests from the developer. This is a good example of 1 that you should surface.
Because it's almost instantly available. All right so if they'd written any tests, they should have done this and you want you want to actually write tests. So yeah, this 1 is definitely productive. Right? And this is what we're getting at here when we say, like.
Like, the difference between, like, de, like, detectable and productive is actually quite, like, subtle. All right. Um.
This is so easy to detect. All right it's so easy to detect.
But the fact that it wasn't detected is very interesting.
All right, it's very useful information to the developer and so, uh, this is the sort of subtlety that, uh, the Google folks are getting at when they talk about this notion of productive versus unproductive. So some of you can actually, like, if they're not detected are quite interesting and ought to be surfaced, even though they seem like they're not very interesting on, like, on the the sort of traditional.
Detectable versus not like, is this 1 detectable or equivalent? Well, easily detectable done right? Not very interesting.
But the Google folks that aren't need this one's very good. Okay.
By country, let's let's look at a different example, though. Like, so this one's quite similar, right? But instead we're dividing by Lynn at every stage.
Okay, so is this 1 detectable.
Yeah, that's right. You're not returning that.
So, it's not detectable, right? There's no way to check this 1.
However, is it productive? Right? This one's definitely equivalent.
I want to be clear. I mean, well, it's equivalent in the sense that there's no, there's no way for this change to affect the output.
But is it productive? What's the point of this variable.
I couldn't hear you all. Mm. Hmm. Yeah.
Yeah, but but for what I heard, it does not do anything that sounds promising. Mm. Hmm.
Yeah, yeah, well we're returning some of our lives, right?
And so this average variable is only ever used in computing itself.
Right. It never flows into some and therefore never flows into the output.
And so the fact that this Newton can't be detected is a hint that this is dead code.
Right and that the developers should look at this and say, oh.
Well, in that case, like, let me just backspace this entire line and also that right? I don't need anything all. I guess this is the original. I don't need this. I don't need that.
Those will be the lead or if you need any on the call.
This is the whole purpose. This is the whole function and this is a all right well, yeah, but if you're doing, like, a.
Continuously like every part of program then.
You know, later iteration, you might need it.
So, 1 of the beautiful things about sourcing roles that you can always go back to the other day.
All right, so, yeah, we have like, simple all that. Yeah, but, um.
So, this is getting into a non testing based quality or a software engineering, hot topic, but it is generally agreed among professional engineers but that should never be committed. And well, or should not be committed anymore should always be a room.
Because this is wasted competition. That's fair. Great. How are we gonna address.
Yes, but, uh, in general, uh, another, like, good rule is don't leave commented our code and, uh, in in the country.
Delete it, you can go over here in the source control system later if you need it. Right? But it usually it turns out that they've done.
And this is like an empirical, a empirical observation that.
Engineers now, pretty much all agree on is that you should not comment out this code. You should definitely believe it.
And so this, this product, this meeting is quite rough, because it, it points out the fact that this code can be deleted properly. Right?
And, uh, 1 of the, uh.
Uh, 1 of the things that engineering often are happy about, uh, when, you know, working on very large systems, it's making the system smaller because each line of code that you delete is 1 fewer line of code that might cause a bug.
Right. Um, let me give you an example of this. Okay. Actually. Sorry let me ask this question.
Oh, are we supposed to find close those data?
I would argue that decode is above because, uh.
Engineers would like to remove it, right?
It's considered a bad practice to how they voted in. You're in the system it's better to get rid of it.
So, no, it's not a bug per se, right? In the sense that it's not going to cause a well.
Hold up on, it probably is not going to cause a problem with that.
Time right but, uh, we still don't need it. It's a total waste. It's at least the performance though.
So, when we run the same program is the output.
Of the program gotta be output into, or is the fact that that code is probably going to be done.
Neither right, so, um, when we're running mutation testing here, what we're going to get is we're going to get the output that this meeting hasn't been killed.
All right, and then we're going to ask the developer to think about Y.
And when the developer thinks about why the outcome of that process is going to be that.
And then the developer can improve the code we're moving.
I understand this requirement. I get 1 level. Yeah, but but when you're doing religion, you already have to think about this.
Yeah, however I also philosophy.
We never want to waste any time, right? Yeah. So we don't want to waste time on computers. Right? If we compute this variable, and then, for a way to resolve, that is a definite performance but it's also the case that, uh, if we have that code like this and in practice, we can have some pretty serious problems that resolved.
Think it was the European Space Agency, the Boston rockets, because they had a computation that, uh, rent like overflow and pause their machines shut down for a variable that wasn't being used.
Because they had reuse the flight controller on the override.
And so if this overflows, it causes this whole thing to crash.
We're gonna be very upset because we're not going to use it as a result.
To be very embarrassing for us when somebody's like, why it is you're wrong.
So, we definitely want to get rid of the stuff. I think that's very important, but I don't see it.
Yeah, do we have to at least from.
Sorry uh, so we don't we don't need to remove comments. Right? I think the the only thing I was saying there was that, uh, it is better to delete this code then to commented out.
Because there's no, like, commenting it out is the same as deleting it accepted clutters the code.
Uh, I mean, no comments don't affect anything, right? Or this they shouldn't.
When because we want to be able to.
Yes, yes. So if we did, we didn't imagine that we do this. We see this and we say, okay, we don't want to engage you anymore. Right? We've decided that is dead. We're not going to include it in the code. Right? We have 2 choices either. We can delete it or we can comment it up right?
We delete it, then it might be harder to retrieve it later. If we do end up meeting.
Right, but if we commented out, then every time we look at this code, we have to look at that comment and say, why is that there?
And it turns out that that 2nd cost is much more expensive than the occasional need to dig into the source control history and find the code that you deleted.
So, it's purely a Readability is like human time optimization question.
Yep, okay. So the, the answer for is this productive the answer is like, actually, yes, it is despite the fact that this.
Uh, meeting is not detectable, right? There's no, no test that could kill it.
All right, finally here's the 3rd example. What do you think this detectable.
Uh, it is going to be a different size.
Um, but this gets, uh, okay, so, uh, there was a topic that I had intended not to cover, but I'm going to cover now because, uh, I, it is relevant here, which is the, the notion of a week versus strong mutation.
We've been basically exclusively talking so far about we are sorry strong mutation, right? Strong mutation is, uh, our strong mutation testing. It's only looking at the outputs of the program, right?
Week mutation, the testing is looking at both the outputs of the program and its internal state.
All right, so effectively, if at any point during the program, like any variable state changes, then the maintenance considered killed.
Versus instead in strong mutation testing we've been talking about the meeting is only considered killed if the output is different.
All right, um, in this case, I.
It is, I think impossible for asset, I think does not let you even the chart, like, determined capacity here without reflection. Like, you need to, like, break the in order to get this piece of information back from the.
This is purely a performance number, right? If you put the right number here, then the hashtag will perform a bit better, but the wrong number nothing particularly bad happens.
And so I don't think this meeting is too technical, right? At least not without, like, deep introspection into exactly the layout of the, the memory of the machine, right? There's certainly no, no possible way that this should well, this should never affect the output right?
Well, I mean, okay, so it is detectable, right? As you said.
But it's not very interesting. So, is it productive?
Is it useful to know that no test detects this output?
So, we are, we can, we can deeply introspecting the machine and tell it apart, as we said, right if we're very careful about our Oracle, we can see that these 2 program executions are slightly different.
Does that matter would you want to be shown that your test suite doesn't kill this? This test.
Your test doesn't kill this me. Yeah.
But, yeah, I mean, it's, it's technically detectable, but it's like.
It's not very interesting, right? So this 1 in particular is not not.
Not productive. It would be very annoying. If you were bothered with this 1.
And, okay, so the summary of all of this is that, like, when you're doing this kind of mutation testing and practice, you know, I'm trying to elicit good tests from developers.
Like, just whether a mutant is equivalent or a detectable or not is not sufficient. Right? You need another set of heuristics and the, the whole point of the paper that we read.
Uh, for today's class was google's discussion of the logistics that they actually use. So we look at.
Like, what what it looks like at Google, when you're doing mutation testing, it looks something like this.
Like, you have written some code, like, on this run me uh.
Right, and you made a code review comment that looks something like this that says.
If you change it, this 1 line to not equal to be here, this does not cause any tasks Japan right consider adding a test and, you know.
Like, here's the reason why we did it. Right? And so note here that this, this shows you, the mutant.
Right. And then it also has 2 buttons at the bottom.
All right, uh, please fix and not useful. Does anyone know these 2 buttons? Do.
What would happen if you press the please pay.
Or who launching? Yeah, that's what it means, right? Isn't the button is intended for the, uh, the code review.
Owner of your press this button if they want this to not be dismissible until a new test case is.
All right, so this is feedback about the original program. This is a, to the dev who's coded under review that this is a good mutant that they need to pay attention to.
All right, by contrast the not useful button, you would press if this is a bad mutant, right?
And so this is feedback to the mutation testing tool developers that this meeting was not productive in this situation. Right? And in particular.
The, uh, for each location operator, what Google does under the hood is that they're computing the ratio please fixes to, uh.
Not useful and making sure that that ratio was as high as they can get it.
All right, and that's the, the ratio that they care about internally when determining whether a mutation operator is a good idea or not right?
Okay, so does this does this make sense in terms of, like.
How this is actually deployed as far as I know Google is the only like large company that's deployed this at scale, but also don't have like, there's certainly the only ones we've talked about it in public. Um.
But it's possible that others are using it as well. Yeah.
Okay, so Google didn't counter some interesting problems when they deployed when they did this deployment. And this is the point where I want to mention this idea of, of which I think is also an interesting 1.
So, for example, isn't a good idea to mutate logging statements right? Um, I understand what logging is.
I've encountered it in daily life, so at a very high level, right? When you're writing a program right? Especially a long running service that's going to be deployed on 100,000 machines in the cloud. You want to keep track of what's going on right? At some point that that service is going to go down and you're going to be paged at 3 in the morning because you're doing dev ops and you're going to.
We asked to figure out what's wrong with it and you're going to be very happy if the logs contain some information that's useful.
All right however, is it useful for our mutation testing tool?
Mutate the logs no right no, because we're not going to test the output of logs right? Probably not. Anyway. I mean, maybe the well, but that's like, not that's their problem. Not ours, right as developers. So, uh, we probably don't want to mutate the logging statements.
Similarly, then suppose that we write something like this.
If Eva or dot blah, blah, blah.
Do we think we want to mutate this debug thing and make it if not debug.
Then do the login also. No right and so the idea of, uh, of is that it, it populates up.
Right so any if statement that only contains error things is also itself Eric and so on and they have a like, if you go back to the paper, you'll see they have a whole sequence of these rules. Right?
But they're basically common sense, right there they're basically rules for things that are always unproductive from the, from their engineers perspective. Right? And so an error code statement is a statement that if mutated, we'll always lead to unproductive meetings.
And Google basically has a list of these internally. Right? And then their tools are designed to avoid creating these meetings in the 1st place. They try very hard to avoid this.
And this is the sort of thing that 1 has to deal with 1:1:is actually trying to deploy these tools in real life.
And so we can be thankful that the Google people are willing to publish this because now, if you want to go off and deploy such a thing, you won't have to worry about it either, because you can just do this. Yeah.
It doesn't matter right? I mean, so, for for, like.
Like, this, if not debug, like, yes, it's easy to detect it. Right because if you're just consider the, the, the log part of the output.
Then not logging at that thing that you would be or logging a new thing would definitely be a change in observable behavior.
Right. But, uh, mutating the logging statements is not the developers will not be happy with you if that's what you do. Right? If we go back and look at the previous slide.
Like, they're very pressing is not useful 1 all the time. It was here like, blah, blah, blah, blah.
In fact that that's basically never going to be used.
Okay, so any questions on on before we move on.
So, um, just to summarize, uh, let's talk through some of the pros and cons of mutation testing, just to make sure that, uh, you all like, I know we went very into the weeds here so I want to just step back up a level and talk about this like at a very high level, what are the pros and cons mutation testing? So, 1st, it has the potential to subsume other test adequacy criteria, but it's just.
It can be very, very good. All right it is quite good at telling apart test suites a well designed mutation, testing system.
Is 1 of the best ways to determine whether 1 test suite is better than the other all right.
However, it's difficult to do. Well, right? You need to answer questions like, which mutation operator do we use? Where do you apply that? How often do it by them?
You do that at random like, do you buy the randomness like.
Well, it means to be productive or not right? And so on right? And these are the questions that you're going to be answering it from workspace, right? Or you're going to try to answer on how it works it's on a relatively small example, right?
Finally mutation testing is also very expensive. It costs a lot of machine time. Right? And the reason is, if you make a 1000 reasons, you not need to run your test suite a 1000 times. Typically your test, which is pretty slow.
Especially if it's large. Right? And so every meeting you make basically adds 1 more run of the tests that you used to do.
And so mutation testing is the sort of thing that can, as I said, can be quite effective.
But it typically is pretty machine intensive and so you probably aren't going to be doing it unless you happen to have a lot of machines with spare capacity often.
When are you in a situation where you have a lot of machines with spare capacity often?
Yeah, if you're Google or Amazon or Microsoft or the like, and you have, you are, you are a cloud provider, right? Or you otherwise have a like a large service.
That it has a peaks and valleys of demand right? So if you have a ton of machines, and sometimes you need them, because.
Everyone is trying to watch Netflix at the same time, but sometimes you don't need them.
And you physically own the machines right? Mutation testing is often for you right? If you don't physically on the machines, instead you run them from Amazon.
Mutation testing might be for Amazon, but probably is not for you. So, mutation testing is most effective when you have either otherwise idle machines.
All right, any questions about these high level pros and cons. All right. So then, um, with that, uh, I wouldn't go through, uh, 2 announcements and a discussion with.
So, 1st, I recall that there was an exam, uh, during the 1st class after spring break. Right. Um, and I want to announce something that, uh, you know, you could have known already if you were looking at the website. But I think it's important to put it in front of all of you, you are permitted to bring 1 letter sized piece of paper with handwritten only notes. And, uh, I'm excited that you are permitted to if you take notes on an iPad of the.
To print those out as the answer, right?
The only thing that's important is that your handwriting must match what you do on the exam so if you type them out, that's not permitted. Right? Because you're going to be turning in the notes and we will look at your note sheet and check that. It's similar to the handwriting on your exam.
So that also means you can't use someone else's note sheet that you photocopied.
That is also cheating right? You must write it out yourself. That is the important thing, right?
We'll also note I will also note that you, you are expected to prepare this by yourself right or at least in small groups. Right? And, you know, you certainly welcome to discuss it but I do expect that everyone's note sheet will be different.
Right, if you're noticing is exactly identical to someone else's, both of you will be have a discussion with me. Right? They should not be exactly identical. They should at least be somewhat different.
Say you, you're welcome to discuss amongst yourselves about what's important what you want to put on it. That's fine.
But you shouldn't copy anything someone else's you exactly. Copy it. He'll be in trouble.
Any questions about that frequently if you ask me, why am I making you? Do this.
Good there you go, you spoke, um, I'm making you do this because I have found that this is 1 of the most effective ways to force students to study for exams and students who write a note sheet tend to do better. That is the purpose.
All right, um, if I ask you a question, it also means that I feel not so bad if I just included a definition as an implicit part of a question.
For example, I ask you to say something about the competent program or hypothesis.
I don't have to worry about like, this this question will punish those who don't remember what the competent program or hypothesis is, is because if you're struggling to remember that you will simply write it on your note sheet.
All right the definition, right? So you should have the definition of product, or you should remember your review.
Will it be a review? I will typically hold a review session, but only if there is sufficient demand. Right? And, uh, when I hold a review session, it will be like, I will prepare nothing. I will just stand in front of the whiteboard and answer answer questions until I stop being asked.
Before I would plan to do it, uh, probably 1 of the evenings after the break in the week, leading up to the exam.
I will survey the class.
Probably, I would prefer to do it in person, but is extraordinarily annoying to as our classrooms.
The, uh, open ended verbal.
The exam mix yeah, you can take a look at the exams that I gave in my undergraduate software engineering class over the last couple semesters. The format will be broadly the same.
Yep, those are linked on the exams page or the course Web site.
Uh, so there's no textbook for this course, but, yeah, it will be. It will only yeah, so it will only cover the material before spring break.
Yep, I will expect that you have read the readings and so I might reference them, but you won't be like, directly asked the reading.
Yeah, so like, for example, I might say, like, you know, like Google says this about, uh, you know, mutation testing, like like, why did they argue that like, uh, what, what was their evidence right? That? Or like, you know.
I might also say something like, you know, considering google's opinion about how to do mutation testing, argue that error statements are a bad idea. Right? Something about this right? Like, something like that is allowed. Right? But in general, I'm mostly I'm going to draw design the questions based on the content of lecture and the stuff you were supposed to learn from the hallmarks.
So, things that, that will cover like that you did in the homework.
Expect to be emphasized more uh, I will, uh, an hour and a half at 90 minutes, right? I will design it to be done in 60 minutes.
But I will give you 90 minutes for it. The reason for that is that I am very bad at designing exams and they always take longer than I expect.
So, just multiply 1 and a half yeah.
We voted on this at the beginning of the semester, and I don't remember the result. It will be either 6 to 7:30:or 7:30:to 9, depending on what people said in the.
The background survey at the beginning of class, There'll be a lecture in the other section, the other half.
I'll do a lecture for the other half of the, uh, at the time.
Yeah, no, it will be the introduction to the 2nd, half of the class.
No, no, it will almost certainly end early. Yeah.
It will probably be only about 45 minutes if I had to guess. Right? And then.
I will do 90 minutes for the exam, and then you can will be done early and it'll be fine.
Any other questions. All right. Um, and then, uh.
So you have 2, uh, another announcements, you have 2 weeks for the next homework homework 6, right? Uh, the reason for that is that it requires quite a bit more programming in particular, um, you should get started on it this week. Right? It requires a fair bit of programming, and it may require you to interact with programming concepts that you might not have encountered before and we want to be able to help you with those. Right? So.
Next week's lecture will be relatively short. I will not continue talking until 8:30:it will probably only go for an hour and a half because I expect most of you will have questions about homework. 6.
All right, and that we will spend most of the next hour and a half. Uh.
Helping you with those, right? So, that means that you should definitely get started on it this week. All right. And then when you encounter the fact that you don't know what a visitor pattern is, or you're confused about or whatever, right prep those questions and bring them to bring them to class next week. And we'll, we'll go through them all together.
Yeah, there's a free service. Yes, you want to use that.
You can do whatever you want, right I provide that information as a courtesy to you. Right. So if you are getting weird behavior, like, your when you do it locally, everything seems to work fine, but on the grading server or something like it, you never get results. Right? It always says that your system crashed that might be because, like, there's some version difference or something that does.
Yeah, please. All right.
5 times today. Yeah, so that rolls over the next.
Yeah, so that's intended as a soft limit, right? So, I'm not going to enforce it directly in the sense that, like, there's nothing in the that stopped you from submitting too many times.
However, this is an assignment that reward can be beaten by, like.
Trying the same thing repeatedly, like hundreds or thousands of times and getting lucky right? If you do that.
You will automatically fail right? So, the idea is that you probably, if you're working on it, like a reasonable amount, you shouldn't need to submit more than 5 times a day.
If you need to submit 8 times, it's not the end of the world, but like.
Try to keep your don't use the as your.
Primary metric for that you're succeeding.
Instead, like, test yourself and when you think you've made a serious improvement, then call the auditor.
Yeah, so basically what we're gonna do is we're going to just look at, like, after when we're grading these, we'll look at your submission history. If there's hundreds or thousands of submissions, we'll be upset. If there are like, we're not going to count, like, we're there 5 or 6 in this 24 hour period.
Other questions.
Sorry.
So, the short answer is, no, the longer answer is that I've already done so, uh, which is to say that, um, I've already posted the, uh, the exams I gave my undergraduate software engineering. Course it covered some of the same material. So, for example, there are questions on some of those exams about, uh.
Like coverage, or mutation testing or the like right and yes, it doesn't it covers other things as well and it doesn't cover some of the.
Uh, the questions, like some of the topics that we covered in this course, but there's enough of an overlap that I think you should be able to get a sense of what kind of questions I might ask and that's all you're going to get from practice questions. Anyway.
So, I'm not going to prepare a separate practice exam for for this course.
But that's why, so I urge you to review those exams and pick out the questions that are relevant to this course.
Basically, anything about testing any other questions.
All right, 1 last announcement, which is that causes office hours this week will be slightly shorter than usual. They will end at 4:30:because. I have made causes to do something else at that time. In particular. We have a faculty candidate who's meeting with students, and I think it's important for causing in particular to talk to that faculty candidate because they have research interests in common.
All right any other questions here before we end.
Alright, I urge you just to use the next half hour to get started on homework 6, and, uh, you know, ask us any or at least read this back and ask us any questions you have about the speck. The speck is quite long. Uh.
So, you know, spending next half hour reading it and asking us any questions you have about it is a good way to resolve those.
Um, if not, I I will see you on next week.