2 minutes on the clock and whoever has sound on please for now.
1, true or false C Smith users, both diagnosis and runtime checks to avoid undefined or unspecified behaviors and the programs that it generates.
True. False. True. Yes. True is correct. Uh, so, yeah, uh.
Much of the paper was about these 2 topics. So, uh, hopefully it's fairly easy. 2nd question.
Uh, the main example in section 2, the icon paper, uh, was, uh, which of these.
A stack. That's correct. Yeah. It was a stack. Uh, it was not a C program. It was a Java program.
All right, any questions on either of these before we jump into it.
Okay, so today's topic is Oracles and we're going to go into a fair amount of detail about several techniques for Oracle generation, but I want to start out with some high level motivation.
So, we've been talking about generating input for the most part over the last few weeks. We've talked about fuzzy and we've talked about.
Uh, other input generation techniques, like symbolic execution, but generally input is only of.
You know, limited value if we don't know what the program is supposed to do with that input. Right. Fosters can detect crashes, but they're gonna struggle to detect anything more. Interesting.
And our topic today basically is how do we get them to detect something more interesting?
And so the key question, if we can generate and input for a given path, like, for example, using the techniques we talked about last week, how do we tell if the program actually behaved correctly when that path was executed? Right?
And in general, this is hard. This is a very hard problem, including for humans right? So.
Many people believe that formally writing down what the program to do is just as hard as writing it. Right I've heard this claim before.
Okay, I've tried to write a formal specification for any program.
A couple of you. Yeah. Um, it it is very hard, uh, when we get into the static analysis later this semester, um, you will see exactly how hard it is.
All right, so to formally specify, even a relatively small program.
It is genuinely, very hard programs. Have a lot of behavior. Modern computers are very complicated. Um, there's a famous, uh.
Famous, I guess paper is the right word, but really more like like 960 s blog post from a famous computer scientist named Tony W***. I've heard of Tony hor.
No, 1, okay. He's a really famous person in compile, like, early compilers, but he wrote this paper, which, as I said more like a 160 blog post, where he went through all the work, he did in order to prove a very simple program. Correct? And it's like, 10:15:pages, I considered making you all read it for this lecture because it's useful for.
Depending a lot of this stuff, but it's actually like a nightmare to read. So we're not we're not going to actually do that. But if you want to see the detail there that.
Uh, that paper is called, uh, I believe proof of a program find right? But it's for a very simple program.
And, you know, he spends 10 or 15 pages telling you all about that and that's for our program written in the sixties. So it's not a very complicated program. Right? Proving something about a modern program is much harder. Okay. So formally written down what the program should do people often like many people believe is as hard as writing the program in the 1st place.
This is called the Oracle problem in general and so if we refer to the Oracle problem, what we're talking about is the difficulty and cost of determining what should a program do for some given input.
So, the key question, what should the program do and so, what we're going to talk about today are various ways that we can try to, uh, especially paired with random input generation, or, uh, improve our ability to detect bugs that are beyond just.
Uh, crashes and this is expensive for both humans and permissions. Right? You know, you've probably had to spend some time thinking about what a program has done is supposed to do at some point in the past.
And you probably have a good sense that it's a pain. Computers are not particularly better at it than humans. In fact, in general computers are much worse. Right? So humans struggle a bit to come up with test inputs to computers are very good at choosing test templates. Mostly at random we talked about and we talked about other techniques for this. However, computers are generally not any better than humans. Really?
Uh, finding oracles, they can be good at enforcing oracles, so we can provide them and that's mostly what we're going to be talking about today is how do we find Oracles to make a computer enforced because almost all of this is gonna have to be done by humans.
Although the last technique we're going to talk about, but you all read a paper about icon dynamic. Conversion detection is a, a technique that's mostly automated.
All right, so, yeah, and for machines, this is sometimes impossible at some point a human needs to decide what the program is intended to do. All right, there's, there's no way for a machine to make that judgment on. So.
Okay, so we've already seen 1 kind of important Oracle when we talked about, which is what's called an implicit Oracle and the observation that motivates these is that there are some things that programs definitely should not do given any input. There's no input on which these things should occur. Right? And these are things like crashes or sag vaults buffer overflow.
A loop forever, right? I never terminate.
And these sort of implicit oracles are what fussers use we've talked about them before. Right? And so the key idea here is run the program and check if it doesn't any of these definitely bad things.
Um, and we can have a whole definition American historical, right? It's an Oracle that's associated with the language or the architecture and not with a specific program.
So, the way that we find implicit Oracles is we go and look at like, what should what, what must be true for all of this to, uh, for this to possibly work don't ever crash. No program should do that.
Okay, so these, these are useful, right?
But the key question we're going to be trying to address today is, like, can we do any better.
So, and this is what most is actually using the real world, right? You saw AFL, for example, right? Afl uses instead of implicit oracles. Right? But our goal today, as I said, is to try to look at what we can do, that's a little bit better.
And the reason that we care about this is the implicit oracles have a key weakness.
Anyone have an idea of what that is based on what I've said so far.
What's wrong with implicit oracles? Yeah, they're they're, they're not going to detect program specific bucks.
If you had to guess, like, just rough percentage.
How many bugs in real systems do you think are program specific?
Like, have a program specific Oracle is it.
Approximately non approximately all or approximately something.
Approximately all approximately some, it's somewhere between approximately 7 approximately. All right definitely. Um, certainly not none. Um.
Yes, programs do crash in the real world, but crashes are often detected pretty quickly.
All right, uh, they're often fixed, much easier to fix because.
I mean, it's easy to, uh, it's easy to observe for them right?
When the program is doing something subtly wrong, when it's producing the wrong output, it's often much harder. Right? And these are the kinds of cases that implicit Oracles are going to do very poorly at detecting, and that's why we're going to want the techniques that we're going to talk about today. All right. So, implicit oracles, the key weaknesses that they're limited to facts that are true about all programs. They cannot.
They cannot find a program specific box, right? Or in particular, uh, bugs that don't manifest practice for, uh, bugs or the program just does look like, get the wrong answer.
We're never going to do to get the wrong answer bugs, using implicit oracles, unless getting the wrong answer. Actually causes the program to crash or not terminate or something that we can actually detect.
Alright, so compare this to the way that a human would write a test like, think about how YouTube, how you write a test, you select an input, you select an Oracle and then you compare the 2.
And I want us to view this, uh, this technique as a form of sampling right we're selecting an input and an Oracle and, uh.
When we do this, we're doing it for it concrete specific examples. That's what humans are good at doing.
So, human testing, generally, when we write tests, what we do is, we usually sample the concrete behaviors of a program. We look at exactly what it does on some particular input and we say, yes, that's right. Or no, it's not right.
The question then, is can we do any better than this? Because like the obvious Oracle generation I'm very sorry.
My blood sugar is low, so we're gonna have a little snack as we continue this. I apologize.
We're all going to keep handy. So, the key the key question here is, can we do any better than sampling.
And, of course, the answer, or wouldn't be talking about this is yes.
All right, so the key idea behind all the techniques for producing better oracles that we're going to discuss today.
Is that each of those give each of them is going to give us a more general way to discuss what our program should do than a single concrete output.
These are called words, these are called partial oracles.
What are less specific about exactly what the program should do?
In a traditional human written article 1st of all human written articles. That's something like, give an input, uh, 5 produce output 3.
All right, these kind of oracles are going to be more abstract. They're going to say something like the output must be positive for any positive input.
And you can do with partial Oracles like, this is an abstraction over top of testing, right?
Um, the concrete traditional Oracle or something like that should be fine.
And a, a partial Oracle is an abstract version of this so it might say something like, for all X actually be greater than 0. all right.
And in general, this is what we're going to be talking about, we're going to be talking mostly about.
Formulas about what the program should be. Right almost everything that we're going to talk about today is related to this.
So there's going to be a fair amount of math in this lecture so I'm just going to warn you all ahead of time. This is going to be a fairly like, we're gonna be testing your your math background a little bit today because, like, this sort of partial Oracle is all about finding a way to.
You know, sort of symbolically represent what the program is supposed to do.
Right. And so if at any point, I mentioned, like a math word that you don't know, please do stop me. I'm happy to go to the whiteboard and to find them I've included at least a couple of places in the lecture where I'm, like, actually put a definition in there, but I might have missed some things that you might you all might know not might not know.
So, if so I apologize it, but please do please do stop me.
All right, so today's key theme is that we're going to combine random testing, put generation, like, buzzing.
With these kind of abstract, partial Oracles right? We know that.
For all program executions, extra, be greater than 0.
And if we run the father and ever produces something, where actually is not very in 0, we found about.
That's the key idea. All right so where can we get? Partial oracles? Does anyone have an idea.
Not by using the output. Um, so.
I think you're referring to something like icon.
Like, the, the tool we wrote about in 1 of the papers today. Yeah.
That's 1 option any other ideas.
Have you ever considered writing an article like this for a test that you've written?
Yeah, yeah, so, um, 1st, option here is definitely just ask a human right? So, certainly it turns out that humans are actually surprisingly good. This, like, asking human to do it here is going to turn out to be a reasonable like, reasonable plan.
And this leads, so it's called property based testing, which is the 1st thing that we're going to cover today.
Uh, the idea of property based testing basically, is that you're going to write a partial Oracle and then the system's going to enforce it. Right?
And surprising yeah, I think it is it is maybe a bit counterintuitive though, but humans actually do generally turn out to be pretty good at.
Like, this was not obvious, like, before people invented property based testing um, but it turns out that it actually works quite well to ask humans, direct partial oracles instead of concrete ones.
It's about as hard, uh, and, you know, anecdotally, uh, for a human to write a a quick check kind of a style, uh, parcel.
A partner Oracle using a property based testing, uh, tool as it is to just choose a concrete output. Like, usually, humans are only slightly harder, maybe, uh, to write the partial Oracle, but you get a lot more out of it because you can then fuzzy against it.
Um, okay, so that's option. 1 option 2 here is to, uh, exploit known relationships between different inputs or programs. This leads to what's called Metamorphic testing. And, uh, a subset of Metamorphic testing is the kind of differential testing that you read about in the other paper. Uh.
For this week, this is an example of a differential testing technique. All right. And differential testing is 1 kind of Metamorphic testing.
In particular C Smith is exploring the relationship between different compilers.
All CI, compiler should produce valid code. Right? And in fact, they should probably produce the same code. Right? If you give it the same executable code from the same C program if you give 3 C compilers the same CI program.
You get out 3 programs that do the same thing, right? If that's not the case, there's definitely a bug and that's differential testing, but we'll go through a bunch of other kinds of, or at least 1 other kind of anamorphic testing as well. Right? So.
Um, the, the difficult part about Metamorphic testing and general, I think human, you're gonna have to provide these relationships.
All right, it's relatively easy when you're talking about compilers and we know exactly that they should behave the same. All right.
But even see was pretty hard, right? You all saw all the work that they had to do in order to, like.
Build a buzzer that didn't trigger undefined behavior. Right. Um, and that is, I think emblematic of the general difficulty of Metamorphic testing.
Metamorphic testing is extraordinarily effective, but it's also a ton of work right? And it requires you to think deeply about exactly what the program is doing, or how different inputs or programs relate to each other. Right?
Okay, and then the 3rd option is icon right? Which is to say, run the program and automatically absorb what happens to be true, uh, about the program as it runs on a bunch of tests, and then try to enforce those on, uh, by the right. So, observe some invariants that happened to be strong human written tests.
And then, uh, let the father, uh, try to break those, right?
And if if something's true on all the human written tests for, and icon were generally assuming that it's true in real life.
And that leads to, as I said, dynamic in very in detection or icon, which is our example of dynamic conversion detection there are other dynamic. And, uh.
Other dynamic and variant detection techniques out there, but I'm not going to cover them. Um.
You know, I think icon is by far the most usable system that exists and also is by far the most famous.
Okay, so this is basically our agenda for this, uh, this lecture, right? We're going to cover these 3 key things.
Property based testing Metamorphic, testing and dynamic and very introduction. Right? So these 3 options.
All right any questions here before I proceed into property based testing.
I'll give you a chance to the rest of this candy.
I'll run it on the board.
D, a, I say, oh.
Oh, no, let me look for it. I don't know what I mean.
Uh, probably, I don't know if you know.
And so vegetable.
Say something yeah. Um, I have heard it compared to a carrot.
It's it's named after a particular vegetable. I genuinely don't know why.
I I, I know the person who is the main author of this tool, and I've never asked him and I probably should next time I talk to you maybe I will be able to report back later this semester.
All right any other questions before we move on.
All right, so then let's, uh, let's dive into property based testing.
So, I'm gonna refer to property was testing for the most part is here just especially in the slides. So, keep it short but.
Testing testing technique for a human rights of personal Oracle that is specific to the system under test.
And this bit about being specific to the system under test is key, right? It's not properly based testing. If you're writing, like.
The program shouldn't crash. All right, that is not properly based testing. That's just a fact that's true. Of all programs. Right? So we want, uh.
You can think of this as, like, blessing, but instead of using an implicit Oracle, like dumb crash, use it a human written program, specific Oracle instead.
Um, and I like this is basically how this works, right? The most effective variants of property based testing is pretty much. You write an, an environment like.
My stack should always have a positive number of items in it. Right and then you fuzz it. And if it ever has negative items or negative size, then you report a buck. You write, you write down whatever property you think should be true about your system. Plus the system and see if that property is in factor on all inputs are all the inputs that the files are generated.
Um, and I want to note here this does require that you have knowledge about the system being tested like this is in general hard, right? So, if you can, if you can apply some part for Oracle to any system under test, it's it's actually an implicit Oracle. Right? So the key difference here between implicit Oracles and property based testing is that property based testing you have to sit down and think about the system.
So, what it is that it's supposed to do, right? And that is the hard part.
Writing a property bit, like using property based testing, requires you to sort of formalize what knowledge you have about the system.
I, um, you know, it's, uh.
There there are many invariants that are true about programs in general. Like, there are many facts that that ought to be true and probably based testing basically is like.
Force you to write them down and probably be assessing is interesting, because it actually come. I'll get to this in a minute, but historically it comes out of the PL community, not the software engineering or testing community, or from industry right? It's been widely adopted in industry. Now.
Uh, but this, this comes from, like, thinking about how to write proofs about programs and effectively the, uh, the history here is that, at some point, someone was like, well, we're writing all of these proof techniques but, no, 1 is using them. What? If we just pair it with testing, how will that go? And the answer is very well, right?
All right, so what are the benefits of doing this? Actually, before I give a list, what do you all think are the benefits of property based testing.
Like, why would you do this instead of just writing a concrete Oracle staff?
Yeah, it gives you like some more flexibility. That's true. Right um, that can be both an advantage and the disadvantage, right? If you write something like X should be greater than 0, rather than X equals 5. all right.
As the example of from a previous slide showed that will constrain the outputs last. Right? But it's more useful in terms of finding like, it's useful in, um.
Giving you the flexibility to test many inputs. All right.
So so, yes, that flexibility can be a benefit, but it can also be a downside. Right? If it's the case that X really does actually, like, really should be 5 on this particular input. And that really matters.
Uh, then that partial Oracle isn't going to catch up that X changes to 4.
All right. Okay. Um, others.
Other ideas yes. And this is really important. That's a really excellent. Yeah. And in general, this is a big advantage of formally reasoning about programs in general. Like, if you formally reason about what your program to do, you're going to have a better understanding of it. Right. It's a forcing function for making you understand what your program's actually supposed to be doing.
All right, and yes, that is a very important benefit of property based testing. It's also an important benefit of using a Skype system or using a verification technique right? As we'll talk about when we go further into static analysis and the 2nd, half of this course.
Um, this is 1 of the most important benefits that you can get out a formula for, isn't it?
You get a better understanding of what's going on other ideas.
It should, uh, yeah.
Check with.
Let's see.
Yes, yes, and that's excellent. Yeah. So, um, this is a great idea, right? Um, concrete Oracles can be 2 strips right? If you, I, you know, I think a web page is a great example if you, like, take a screenshot of the webpage and it's just the, uh, the webpage muscles always look exactly like this.
Then if the user is on it, like, you tested on a different device that has a different size shape of screen, you might end up with something different.
Or some library, you're depending on changes and move some buttons around like, maybe that doesn't matter.
What matters is that the button is visible right? And checking the visibility of the button might be a good powerful Oracle.
All right yes. Excellent idea. Any other thoughts.
All right, these are all these were all excellent. Right? I have a few on here, and I think they correspond pretty neatly to the ones that we talked about, but let's go through them. Right? So 1st, um, tests can have a clear mathematical presentation. And this is both good for you, like, in writing them, as was suggested, uh, right about, uh, by.
I don't know your name. I'm sorry, Michael by Michael I, that's not, uh.
You know, it, it helps you understand the program better, right? But it also is going to make it easier for future developers to understand what you need. Right? If you write down the like, this thing should always be positive.
Uh, that's much more clear to a human then, like the output on this specific example it should be 5 right? That's the specific example. Being 5 doesn't communicate the fact that you expect the output to always be positive.
That more mathematical definition is something that is sort of a form of documentation that then your your testing is going to enforce right? So a future developer can actually trust that. It's true. As opposed to like a comment that you wrote, which may, or may not be true now.
Um, another advantage is that, uh, you can avoid, uh, finding and writing a case for each property right? Effectively you don't need to, like.
Uh, worry about the specific inputs, uh, that you're going to use, right? You don't have to worry about, like.
Uh, you know, choose X equals 5 or 6, you just write down the property you want to enforce and then the buzzer will use both. Right? And so this allows you to focus on what you actually want to test and not the specifics of how you're going to test that.
All right um, and this is also good for maintenance, uh, because.
Like, it turns out that, uh, the partitions that you imagine, when you write a software component in the beginning and you, if you're doing partition testing, someone remind you of partition testing is.
We talked about it earlier in this class.
Yeah, yeah defining the input into groups and, uh, right like writing 1 example test case for each. Yeah so this is a very common technique for humans. Humans often do this when writing test cases by hand, right?
The problem with doing it is that, like, as the software changes, those partitions that you originally imagined might not be true anymore and so you might end up with either redundant test cases, or parts of the code that aren't really well covered. Right?
By contract with property, based testing, you're you're, you're writing down what the partitions you're imagining are. Right that's effectively what you're doing when you're writing 1 of these.
Property is down, is you're saying, like, this is what I think the.
Uh, this kind of input should produce. Right? This is a fact that should be true about this program and, uh, that will remain true right? So you don't have to worry about the specific, like, choosing specific examples and worrying about. Will they become redundant later?
And so that's another advantage and finally, I mean, Ken decrease maintenance maintenance costs like that is to say, like, you're not going to have to worry about things like the, uh, the gooey testing example where, like, the web page moves a little bit and you have to update the test case.
Right, because the property, uh, will still be enforced. Yes, you occasionally you have to change the properties themselves, right? Because, you know, you might actually change what the program is supposed to do that does happen, but these are sort of tests tend to be much less brittle. Right, and the fact in the sense that they break less often for silly reasons.
Alright, alright. Do all of these benefits make sense to people.
I don't have a slide on this, but what do you think is the biggest cost of using property based testing? What are some we're continually think of a downside.
Yeah, I mean, especially if you're not used to writing this, uh, writing test this way, it can be harder. Right? It's a skill.
And so if you don't practice doing property based testing, it will take you longer.
In my experience, I would say that writing property based test is not really any harder than writing concrete test. Once you get the hang of it. But it takes some time to get the hang of it. Like, maybe for the 1st, month or 2 that you're using, this kind of technique is probably start a little bit, but once you, once you get going on it.
Most people can get the hang of it pretty quickly. I would say, though, that it does require you to be like, a bit stronger in math then concrete testing does right in particular. It's going to require you to be able to think about, like.
Quantifiers or formulas, or like, statements that involve for all executions of the program, blah, blah right it must be true. But, you know, if you have a computer science degree in theory, that's something you should be good at anyway.
Uh, any other costs that no, 1 can think of.
Okay, as I said, I don't have a slide on this, but I probably should.
All right. Okay so, then in practice, I think so. I mentioned that.
Property based testing him out of the PR community in particular was developed for hospital. I know hospital or no other possible.
Some people yeah. Okay so it's a functional language for the various types of stuff that, like, people would love to write, like, new versions of the compiler for. Um, and it was developed around 2000. it has the same kind of like, mathematical vibe that you get out of the hospital community. Right? Where, like, the expectation here is that you're always going to write a test in the form of what for all X dot dot dot.
Right, and this is the same kind of feeling that you would get from writing a half ago program too. I actually I've written a whole program.
I've written a program and any functional language.
Almost like, surprising a few of you um, I would strongly recommend at some point before you like.
Get a real job, try to sit down and write some non trivial programming and functional language. It is a very useful like mental exercise. You will see programming in a bit of a different way. Like, maybe it'll actually we'll say, no.
I wouldn't it's not quite the same, right? Like, try to write a traditional program, like a, you know, sort of standard algorithm like.
Some kind of Graph search or something like this in high school, or okay rather than writing it and see or Java or Python or whatever and you'll get a.
It it's different. Let me just say all right. Um.
So, I, I strongly recommend doing this at some point. Okay. And, as I said property based testing is the same kind of mathematical 5. and for a while, it was a very.
Niche thing, or maybe, like, 10 years or so I was really, really popular in the functional program in community and nowhere else. Nobody else had heard of it. And people talked about it as if it was something that was unique to functional languages. And, in fact, at the beginning, people thought it was right that you needed to be doing functional programming in order to get the benefits of property based testing.
But it turns out that that's just not true. All right, that was a misconception that people had and over the last, like, 10 or 15 years, a number of frameworks have been built for mainstream program.
I think the most fit to the 2 most payments are hypothesis that I've actually.
And Tuesday, but both of these are, uh, still like relatively modern inventions. I'd say hypothesis became, like, usable around, like, 2015 or so. And deep state, I think is even more recent than that. Um.
But, you know, these tools exist and you can use them and in fact, I would recommend that you do, right? If you're writing programs in these languages.
Uh, like, this kind of is is a great technique, right? Um, we're not going to directly use either of these in the homework.
So today's homework is going to involve using icon and using, uh.
A minimum for testing right? But manual for testing is sort of like the evolved form of a stronger form of so you'll, you'll get the, the, er, the, the chance to actually attempt this kind of thing. At least once in the works.
Are any questions here on, like the fact that these techniques exist or their history.
These tools taking that session, we.
He's a member of the.
Yeah, so that's true. All right. Um, it has some theoretical background, but it's not, uh, it's not so much that there's like a strong theory behind it. It's that it came out of the partitions.
All right, so it's the, it's the sort of thing that a theoreticians would think of that's maybe the right way to think of it. Right. Um, so.
Uh, the yeah, so property based testing does not provide any formal guarantees, right? It certainly can't because it's a form of file.
All right, and in test can only show the presence of bugs, not their absence, right? It's not going to provide guarantees even about the properties you write down. Right? Even if you write, you know, for all x X is greater than 0. right? Property based testing will never tell you. That's actually true. Right what it will tell you is that we ran a buzzer for 1 minute and try to violate that property and we couldn't violate it with any inputs.
All right, and that gives you some confidence just like testing does.
But it doesn't prove it. Right? However, um, the reason that, like, property based testing exists at all, is that there are a bunch of mathematicians basically who were writing actual programs and, like, in high school. And we're like, writing tests is really dumb. I'd rather write down a function instead.
All right, concrete testing, very boring for mathematicians 0 interest in doing it and instead they invented property based testing.
Because they would rather write down for all X, blah, blah, blah, then write down, like, actually be 5 on this input. Right? And that's what I mean, by it came out of.
Uh, like the theoretical community, but but it's it's as I, as a.
Pointed out here, like, this is fairly mainstream now and like.
You can use it on real programs and it's quite effective. Okay.
Any other questions. All right.
So, that's all, I'm going to say on poverty, based testing directly, and now we're going to switch into the Metamorphic testing. But Metamorphic testing is actually a form of property based testing the more specific for.
So this is a property based testing technique in which the Oracles are defined, what are called Metamorphic relations, which are going to be the, the, the subject of most of our discussion here, uh, between related inputs or programs, and these Metamorphic relations or relations in the math sense um, so I know what a relation is in the mass sense.
Can someone to find it for me? Yeah. So it's a property over sets. So this is a set theoretic concept right? And here's the formal definition that.
Pda this afternoon um, but basically the the way that a relation works is that it's telling you how, like, elements of 2 sets are related to each other.
All right, so, for example, or, I mean, or in elements of the same side okay. Um, but formally up to officially a few sets, right? Um.
For example, like, less than as a relationship, a relation, I should say over, uh, like the natural numbers right 1 is less than 2:3:is less than 4 right? But 4 is not less than 3 and the formal way that we define that math is generally that, like, there's a set of numbers that are, like, less than is true. And there's a.
Of that set is all the numbers where less than is not true. Right? So, the less than set would include like, 1 comma 2, but not 2 common 1.
Similarly, when we're talking about, uh, more Metamorphic relations, we're going to be talking about, uh, relations between either between inputs or between programs. Right? Um, and in particular we're looking for here is is, uh.
The reason this is going to matter is what we're going to want to do is we're going to want to find find either inputs or programs.
Uh, where we can relate their outputs to each other and the differential testing that you all saw in C. C. C Smith paper is an example of this. Um, and also, I guess, uh.
Yeah, a historical note here is that there are multiple names for this concept. Um, data diversity is another 1, uh, that was actually actually came. 1st, but minimum for testing, I think is, uh, a better word for explaining what this means or what this is. And it has a better formulas behind it.
Also, I know what the word Metamorphic means occurs to me that probably very few of you. Okay. So.
Does anyone know what the morphing departments break it down? Changes? Yeah.
Meta means it changes it. Yeah. Yeah. So, um, that's that's basically correct right? Um, yeah minimal thing. Basically, I, at least in this context means, like, we're, uh, we're talking about relations that, uh.
Allow us to relate something to itself or to something very similar to it. Right. Okay.
Okay, moving forward here. We're going to cover 2 kinds of Metamorphic testing. Right? In particular we're going to cover Metamorphic testing, over 2 related programs on the same input. All right, this is traditionally called the parental testing, and you read these minutes, which is a good example of this right?
And then we're also going to talk about 2 our programs where there were 2 related inputs.
I have some Metamorphic relationship I was going to say where the outputs, uh, of the program.
Uh, unrelated inputs should also be related, right?
And this is usually, this is where, like, Metamorphic testing was 1st, coined, describe this.
Uh, but, uh, I am arguing here and this is.
A slightly heterodox tick that these are actually the same thing, just over 2 different abstractions right? Whether it's the program or the input, all of these, I think, are Metamorphic testing, but I suspect you can find some computer science professors who would disagree with me about this. Right? So, I want to be clear here that at this point, we're getting into the, uh, the realm of, like, this is.
Research now right so, probably based testing as I just described it before is like, well, established.
Standard technique now, but like, relatively recent this stuff for the most part. I mean, different from testing also is I want to be clear. Like, you should always do differential testing.
But most of this other stuff, like the, the broader relationship here that I'm talking about, or the way that I'm explaining it is like, not, you're not going to find it in a textbook because, uh, this area is not fully settled yet. Okay.
Okay, any questions about that I also want to be clear that all the things I'm going to talk about are like.
Well, established as I showed them a previous slide like, a lot of this stuff was invented in the 90 s.
But, like, the relationship between them, like, isn't fully formalized.
That's the bit that is my editorializing.
Okay, so we're going to talk about these 2 in in sequence we'll start with differential testing and then we're going to talk about this sort of traditional Metamorphic testing. All right.
So, starting with differential testing, right? Can somebody tell me what differential testing is based on your reading of the paper?
But or what you think it is yeah so that's that's.
Basically correct, right. I differential testing is.
Testing multiple versions of a program that are supposed to be the same against each other. Right. Um, and so the observation here that motivates differential testing is that there are many programs that have similar or identical specifications.
Like, all see, compilers should produce the same.
Executable code for the same C program right? Can anyone else think of an example.
Thus, there are many.
So, I suspect many of you have a web browser open right now suppose that we're going to consider Chrome and Firefox.
Do you think that Chrome Firefox should display the same thing when you open the same webpage?
Yes, probably yes right. So of course, anything like a pair of web browsers you can differentiate test. All right and this is actually true for any pair of programs where they're supposed to do the same thing. Right? And, in fact, any, any situation where you have competing things that are officially doing the same kind of thing, you can usually do some kind of differential testing. You might need to adapt the.
To each other, or do something like this, but in general, you can almost always do differential testing if you have a 2nd implementation that, in theory is doing the same stuff as what you're doing. Right?
So, if we're building such a program, we can use the other implementation as the Oracle. Right? And you might think, well, you know.
I'm certainly not going to build another web browser there. There are plenty.
Uh, but in fact, it's often the case that you, you might be doing this, even if you don't realize it. So, you know, if you're writing a C, compiler, you can compare output to GTC but, like, we're not going to write a C. compiler. There are plenty, um.
Okay, so differential testing then, uh, officially is a technique for testing 2 related programs by comparing your output right? On generate test inputs.
Um, any difference in, uh, indicates nonperformance in 1 of the 2, right? This is the key bit, right? If there's any input that we can find for, which to see compilers produce different output, it means that 1 of them must have a bug.
All right, there's no reason for them to do that right?
So this is the idea behind differential testing right? So what what are the advantages and disadvantages of this technique? So it's only applicable unlimited situations. You need another implementation. How often do you think this is the case.
Let's go with the, uh, the traditional all the time none of the time, or only some of the time, uh, style of voting.
None of the time clearly not or we wouldn't be talking about it all the time.
Also reasonable, right? But some of the time, yes. All right. So it's only going to be the case some of the time, but surprisingly more than you might think right it's closer to half than it is to none. And in particular, um, you're often doing something like writing a fast version of routine because you're optimizing it um, or you're refactoring, right? Or your.
Doing something else so I know what refactoring is.
Hopefully most of you yeah, I'm just going to write this part on the board because it's an important 1 to know, and I will freely use it on exams.
Because I know what it means, and I think you should too. All right so if you don't know what it means, you should go look it up a refactoring means, like, changing a program to improve some nonfunctional quality while attempting to keep its outputs all the same.
Right. And so if you are doing some refactoring, like, let's say, like, changing a class hierarchy or, you know, improving the program and some other way, right? You want to keep the programs. Exactly. The same. So you can, you can differentially tests like the program before you make your refactoring and then the program after. Right? It's a perfectly valid thing to do to.
And then check that the outputs are exactly the same on all the input the Pfizer generates it is, in fact a well known technique for validating refactoring.
Right. So even if you're working on something that you think.
You don't have another implementation for it. You often do if even if it's just a different version of the same program, you're working on.
Okay, 1 of the other down or 1 downside here, is that a human is also needs to decide which of the 2 is correct?
Um, so C Smith sort of solves this problem by using more than 2 C compilers and voting amongst them. And this is a plausible technique that you can use, but it's very rare to have 3 implementations. Right? I would say, like, only in sort of comments that are important spaces. See, compilers do you actually have more than 3 implementations of the same.
Program it's very common to have 2 very rare to have 3.
And sometimes neither of them is correct. This happens a lot actually. Mm. Hmm.
If you can locate the input on which something like like, say, 2 C, compilers emerge, it's often the case that both of them are wrong.
Right and this can get very tricky if as a human trying to, like, look at these. So, if you're differentially tastic, like, again, to come back to the C Smith example, like I know for a fact that some of the bugs that they found in GCC, and all of them were present in both, and they found them, because both GCC, and did the wrong thing, but they did it differently. And in particular most of those bugs.
Were caused by ambiguity in the specification, right? Where the C specification is written in English.
And include, like, someone wrote an ambiguous phrase there like.
Uh, you know, there's some weasel work or something. That's a little confusing in terms of the wording. And so the Implementers of BCC interpreted it 1 way and the Implementers of interpreted a different way. And then when everyone was brought into the same room, and I, like, argued about what should actually be done, the decision was made that both of them were wrong and then they were going to rewrite this back. Right? And that happened.
Times because of C Smith, and continues to occasionally happen because of C Smith and related projects to this day I mean, C Smith and, you know, it's now a successor. I mean, it sees these papers published 15 years ago, but its successors are still being run on all the sea compilers.
Right like, I think Google pays for this just constantly right? Because it's fuzzy and it does occasionally turn up box.
Uh, but, yeah, so figuring out which of them is right is 1 of the most expensive parts of this technique.
All right, you can just differentiate test programs all day, but eventually, when you find a problem.
A human is going to have to interpret that. There's no real effective automated way to do. So.
Okay, do these actually drive? Yeah. Um, do these 2 things make sense?
Any questions here before I show the next bullet.
So, in general, uh, differential testing, though is a good idea. It provides it much stronger Oracle than most of the other techniques.
This is generally for Metamorphic testing all the minority testing techniques that we're gonna talk about here are generally much better than the alternatives. But are going to be hard to implement are only applicable in the limited situations.
Okay, um, I know what, uh, is meant by a stronger Oracle.
Okay, so let's let's take a quick assign to talk about strong versus weak with respect to particles. Right?
So, okay.
So, um, in general, what we're talking about, like.
Math math, formulas, math properties, right? We'll often we use the terms of strong week or stronger or weaker, but to describe the relationship between different credits.
All right, so for example, for all x X is greater than 0 and for all x X is greater than or equal to 0. I don't have a relationship.
Which ones do you think you would call stronger on screen?
Sorry, the bottom 1? Yes, the bottom 1 is per X. Y.
It covers more, it covers more. Um, so that's, I.
Intuitively are informally. That's correct. Right. But the technical reason is that the cardinality of the set described by this property is smaller. Right? That is to say that there are strictly fewer elements of the set that set of all editors that, uh, satisfy access greater than equal to 0, then satisfy extroverted particular. There's 1 more.
I like.
Okay, yeah, I got this. Exactly backward. So that was very embarrassing. Okay.
So, no, this is the stronger 1. why? Because this set includes 0 on this 1 that's not.
Right. So if we draw, we imagine the sets, right? Like, this is for all x x greater than 0 and this is for all ex, uh, expert in the equals. 0, right 0 is in this sense, but 0 is not an element of this stuff. Right? And so that means that this set is bigger.
Right, and what the reason that we use it from from this week here, is that a stronger Oracle makes the set smaller, right? It forces us to purchase the program to behave.
In a more limited way, right? There's fewer things they do. So concrete that particular output is a very strong model. Right? It says that, uh, the, uh, the output must be exactly this out right?
Um, but I know we're calling. This is a little bit weaker, right? So strong oracles are often good, but you don't want them to be too strong.
Right like the gooey testing example that was given earlier is another like a good example of why you don't want to go work with you too strong. Right? So, strong is not necessarily always good.
Right. Too strong can be a problem but, uh, 2, we can definitely be a problem too. Right? Implicit oracles. On the other hand are typically quite weak.
All right, because they're not actually filtering out that meeting.
Okay, sorry for the very poor explanation of the library of that is anyone confused? Yes, I just saw your question. I don't know what makes sense. All right.
Yes, yeah. Or maybe we can take that as a strictly quality information. The quality of just normal people does not come back the data team.
So, then that's related, but, uh, uh.
I would say that that's more like a like a.
What kind of Oracle right? Rather than something that pushed related to it directly to the definitions of stronger a weaker.
Right, but that is a like, those 2, the the 2 checks you're talking about 1 of the days from Virginia.
All right any other questions there. Okay.
All right, so then moving on.
So, what let's, let's talk about Metamorphic relations now a little bit. Alright. So what what is the Metamorphic relation that's happening in differential testing right? And as a hint here think about how the outputs, for example, to see, compilers should be related.
So, I don't know if your relationship for a call is a relation so it's gonna be a function of some kind of.
So, what what is the Metamorphic relation here?
Equality right. Or, um, maybe, uh, put more, uh.
Slightly math here um, it's it's the identity function, right? In particular, right? We're checking of 2 programs. Have exactly the same output right? So, the, the relation between the output of C compiler number 1 and see compiler number 2 is that they're the same.
Right. That's the identity. All right so they shouldn't they shouldn't differ at all. Right?
And this is by far the most common Metamorphic relation when you look at the homework for today, you'll see that. You're asked to generate some Metamorphic relations and there's a limit on how many times you can use the identity function. You're allowed to use at once. But after that, you have to come up with something a little bit clever. Right? Um, and so.
For differential testing this is by far the most common, minimal population, but it's also the most common minimal for the relation. Just in general that it's, uh, the case that.
Uh, we want these outlets to be the same, right? That's by far the easiest 1. but there are others that we can come up with.
So, what other kinds of Metamorphic relations do you think we could use for differential testing? Can you think of an example where there's a non identity relationship between 2 programs? Outputs.
Targeting different platform.
It would have some yes.
Yeah, so you could imagine defining, like, say a mapping over how.
Different architectures are related to each other. Right? Like, say, like, this instruction means the same thing as these 2 instructions right? So, like, this 1 arm instruction means the same thing as these 2 x86 instructions or something like this.
That would be a different Metamorphic relationship. Yes. Um, that would be a.
It's not the identity function, but it is a transformer inequality.
All right yeah, so I don't actually have that as 1 of my examples that's about to show up. Um, which are what, strictly non quality functions.
Uh, or at least, you know, like something, something a bit different than equality, but that's a very good example.
Any other thoughts what about.
As an example here, uh, the relationship.
In, uh, zip and I know it does depend on.
Yeah, they're universes, right? So it's an unzip, right? So zip up. Some file creates a file unzip. It.
Um, this is that file and gets it gives you back the original bio.
So, if you do, I'm sorry.
Of any acts, right?
All acts that should be asked.
And so this is 1 example. Mm. Hmm. Um.
Hopefully I did this correctly. Oh, no, I did not. Okay. Well, well, we'll just see the other example now. So this is the 1st inversion right is the technical term for this, but, uh, yeah, so for all X, uh, zip, uh, zip of extra.
All right, another example is, I am potency. I know the word.
Almost no, 1. okay. This is an important math word that I will definitely use it some other point in the future. So I'd like you to learn it.
So, item potent is a word that applies to functions a function is item potent if I, if applying it more than once, uh, doesn't have any effect.
So that is to say if f. F. X.
It's always equal to for all apps.
That's item potency. All right. And this is a very, very useful.
Property to have of functions, and a surprisingly large numbers and have it.
All right, that are actually relevant in the real world. Um, the best example here is sorting right for all X sort of sort of X should be sort of X.
All right, certainly it should not be the case that sorting something a 2nd time ever changes it.
If that was the case, that would be very bad right? In particular, though most programs have properties like this. Right? Because most things, like, the outputs of most programs are deterministic. So really anything where who's domain range are the same right? Like, sorting takes an array versus another right is usually item potent if it's deterministic.
Okay, and this, this may also be the case. I like.
You may get converted because not item pregnancy like, uh, if you apply, uh, some function, like an arbitrary number of times, and then apply, like, 1 more time after that, then it should have eventually conversely and there's some minimum limit. That's also pretty common.
Do these 2 properties make sense? We see how these are Metamorphic relationships that we could use to test programs.
I don't see how we could find as a sort program or a zip an unzip program using this technique.
Good okay. I'm as an aside here. Um.
This is a also a points to bring up about program design. Right? So this is more of a software engineering rather than testing question or a discussion. But I want I want to bring it up here because, um, often is the case that, uh.
You are going to be asked to write a program and that program is doing some kind of data transformation. Right? Transferring data from 1 format to another.
When you were writing, such a program, like, if you're implementing a function with a similar domain and range in particular.
Uh, you can often get good tests for free by composing the program with itself, right? Like, we showed with the zip zip and zip example right? But for many, many programs, uh, this is the case.
It's often the case that you can run the program twice and you get some good or in in different modes and the results should.
Converge in some way, right there should be some Metamorphic relationship that you can get relatively cheaply. Okay. This is great. If you can design your program. So that this is the case.
Uh, then you're unlocking the potential Metamorphic testing right often, though you have to make some intentional design choices to allow this, like so, for example, you, you need to require that you have both and unzip. Your zip program doesn't go in only 1 direction.
But if you're able to get that design property, then you're gonna have a much easier time, writing tests. And so it's often worth it. Like, for example, if you need to implement it, it might be worth it to implement unzip as well. Just. So that you can do this.
Because it will allow you to test way better. All right so if possible you should design your program so that, uh.
So that this is possible want to do this again. Any questions about this, does this make sense for everybody?
As, like general advice, if you see if you see a situation, you feel you're in a situation where this might be possible, you should explore it. Right? Because, uh, if you can do this kind of composition.
You're going to get way better testing than you would get by writing test by hand, right?
All right. Okay so any questions on differential testing or minimal for testing with programs before we move on.
Check, how are we doing on time? What not so bad.
Okay, so then we're now going to move on and talk about Metamorphic testing with respect to inputs of the same program that have a relationship as opposed to programs that have really who's outputs, have a relationship to each other with the same input. And I think before I go too deeply into this, it is also going like it is possible to do both of these kinds of.
Testing at the same time, but it's often I mean, that's that's just going to be like a little bit confusing. So you'll probably want to write down what you're doing.
Okay, so the simple case here is that we have related inputs with identical output outcomes that, but just to say, we don't need to know the expected output for a given input in order to check if it's correct because we know that these 2 inputs should be the same and if they're ever different, then we have a problem. Okay.
So, a good example of this is like, absolute value.
Right. So if you have 1 absolute value function, then for all, actually should be the case that, uh, absolute value events is equal to absolute value. Megabytes.
Right, but they're actually quite a few programs, uh, with properties like this.
And this is the sort of case that we're going to be dealing with here. Right? Of course, this is this is a very simple relationship. Right? But we can do much more complicated ones, and I'll show some examples in a moment, but 1st, I want to give a sort of mathematical definition of what we're doing here or, like, a generalization of this, this idea right?
So my goal with this formalism is to generalize this idea of related inputs and outputs and formalize this notion of minimum. Fantastic. So, 1st, what we need is, we need some input i1 yields some unknown. We don't care what 1 is and in particular, we don't know. Work here with our own. 1 is correct.
We're not going to check that this is called the new formula, right? And then we're going to need a relationship, uh, or relation and our eye relates our i1 to some other input. I too right?
And this is what's called the follow up input relation right? And this is what allows us to generate another input from the 1st, 1, uh.
You know, we're able to get a, uh, a relationship about right? So, for example, in this, for all X, absolute value of X equals absolute value of negative X.
Our initial input here with X right? And then our follow up input function or relation here is, uh.
Right. We're multiplying by negative 1 and that gives us the ball of info. Okay.
And then we need the quote, unquote, necessary condition relation, right? Which is the relationship between and right. And so, for the absolute value example, this is just the identity function.
Right it's the case that, uh, the output should be the same. It should be equal. Right? But, uh, like these, these, from both the arbitrary relations.
And so, um, as we'll see in some of the real world examples that I'll show interest a bit, like, these are going to be these can be sort of arbitrarily complicated.
Right and you can write very complicated ones, um, that.
And then enforce them and you'll get surprisingly good results. Okay. So is this mathematical formulas and make sense to everybody? It's not that complex. Sorry it's not that complicated of mathematical formulas. And I hope.
Actually, let me rephrase that. Any. Does anyone have any questions.
All right, so then, uh, let's see an example, right? So, this is an example of a quote, unquote online service.
You may not use this example if you choose variant fee of homework 5 part 2.
All right, but you can use other online services. Right? Um, and in this case, uh, what, what online services is.
Yeah, Google search. All right so we're, we're searching for testing and Metamorphic testing.
And metomorphic testing and testing are related inputs. All right, they have a relationship that we understand in particular. We believe that testing is more general than Metamorphic testing and therefore we should expect that the outputs are related in particular. The testing is going to produce more. Certain queries are more for approvals and probably take longer than Metamorphic testing.
Right, so the, the generalization here is that what we're doing here is if we add in a like, a more specific version of the company start with testing.
That are full of input relation is, like, add on the word metadata, right? Or add on another we're related we're testing that we know is gonna be a smaller staff. Right? And then you look at how many results are it was the case, the Metamorphic testing would produce more results.
Then we'd be surprised by that that would be a red flag for Google search and they'd be concerned.
I want to understand this example. Okay.
Or something.
Yeah, so it's a, it's a, it will vary, depending on the program right? This requires us to have specific knowledge of the program. We need to know that, like, searching for a more specific term should result in fewer fewer results. That is a fact. That is true. About Google search, or about search in general, but that is not true about other programs.
Right if in fact, if instead of when you typed in, uh, like testing into this bar, you instead got like.
Uh, and outputs that was proportional to the number of characters that you checked in.
All right, then this revelation would be totally wrong. Right? We have to shoot a totally different relation for that.
All right.
So, this is an online service example I'm now going to give another example that is much more complicated. So, I'm very sorry in advance, but I think this is a good 1 because it allows me to demonstrate the sort of Metamorphic relationships that are actually very exciting in the real world. Right? All right. So, does anyone know to discrete discrete wavelet transform.
In particular in the context of image processing.
Battery law. Okay, that's fine. Um, for the purposes of this example, you don't really need to. Right but at a very high level, um, what this is doing is this isn't a demonizing technique for images. Right? So, what we're going to do here is we're gonna try to take an an input image and break it up to remove a noise that is like.
I like the camera lines or something like this. Right?
So, for example, so we're going to focus on this part. So just a single application that I just, uh.
We will transform the team here right? And it's going to split up into 4 dimensions or 4 boxes. Right? Uh, which, uh, this 1 is going to be the most important. This one's gonna be the output image.
And then these are going to be the components that are noise. Right and the underlying mathematical technique here, as I said, does not matter. Right. Don't worry too much about it, but know that this is something that is like, in a real, uh, image processing system. Like, the 1 that is running on your phone when you take a photo, right? And then it before it actually shows you the photo, it's running something like this.
To remove, like, noise from, like, the camera being moved or something like this. Right? Um.
This is how it happens. So, then the output, or like, what what this actually looks like it's like, suppose we have this picture of this ship right? This part is the ship again right? And then these parts are noise.
Right. So it's splitting it up into these bits and then the next step is going to be that you're going to remove.
Uh, like those bits of noise, and instead of take this this output. Right? So this is, uh, uh.
A transformation that, you know, involves manipulating this image.
Okay, so, uh, can anyone think of a Metamorphic relationship that might need to be true about this, uh, this technique?
Sure, yeah, that's true. Um, that's that's quite reasonable. Like, in particular, you'd expect it to be more stuff in the noise parts.
All right, so, if you know that 1 photo is worse than another, or needs to be noise more has more noise, uh, then you could check to see that. The noise actually is more in that photo, right?
And, yeah, if you had a way to add noise, that could be your follow up input.
Relationship right. I don't think that's 1 of the examples I have later on here, but.
It's okay. That's a good 1. yeah.
Yes, yeah, uh, it should be, um, so if if if this is the, in fact, in order to completely, you should definitely I agree. I agree.
I think that's also not in my list to follow up once. So there are many right? And this is actually, a good takeaway here is that if you just sit down and think about it for a minute, you're going to start to come up with something. But actually, there are a lot there are many things. That would be true. The ones that I have here are based on not my personal knowledge of discreet wave transforms, but of the person who originally made this example, and who I stole it from. All right.
Who is an expert, right? But like, for any program, at which you're an expert, if you start, like, really digging into it, you're going to be able to come up with more and more Metamorphic relations.
Okay, so the actual program that we would, we would see something like this end. It's something like a J peg. Right? I know that, uh, encoder is or an image encoder.
Few people yeah, somebody want to explain to the class.
It looks sort of like this, this is, uh, an abstraction of of what we're talking about here, but.
I just wanted to say is given to different record. Yeah. Yeah. That's exactly right.
All right, so, and then part of this is going to be the wave of transformation, right? So the program we're going to be looking at here, or the example here is a program that does this sort of encoding. Right? So, it's, it's actually converting something into a. J. peg.
Um, okay, so Metamorphic testing has 3 requirements right? What were the 3 requirements.
We talked about them a few minutes ago. What 3 things do we need here?
Initial input and then there's 2 relations, right? So we need the set of initial inputs right? We need a relation. I can generate followup inputs and we need a relation r0 that gives us a necessary correctness condition right?
Okay, so how are we going to get a set of initial inputs to a program that may be less images?
You all have done this homework, assignments, AFL, or something similar right? Like homework too or homework free. Alright. Um, well are there are many techniques that we can use not too hard. And then what about our, our eye and our oh, well, we're gonna have a ton of options for this. We, you all mentioned too, right?
Uh, but there are many others, so, as 1 example, suppose that are I, uh, turns the image by 90 degrees.
Pass it to the, uh, some kind of this, uh, some other text.
It can be answered output should look like yes, it should also be turned by 90 degrees in particular. The top left corner should also be turned around any degrees. Right? The noise might change.
But that's fine. All right so, uh, in particular, we want the output to look like that. Right? So that's 1 example. Right? Uh, that's a, a sort of transpose operation, right? Transpose the input image. The output components muscle should be transposed right?
Um, but there are many others that we can come up with right here is an incomplete list, right?
Uh, we can add a constant tool called and this is like a fact about inputting, like encoding. Right? You don't need to understand that any of these.
But the point I'm trying to make here is that these are, like, you can get quite a deep in here about what exactly these things should do. Right? And this is just a mostly arbitrary example.
All right for any program, uh.
You're probably gonna be able to come up with some kind of Metamorphic relationships here. Right? So, for example, like our, uh, the 3rd 1 here, like, converting the color values should also cause the output to be inverted. Um, like, only the BC. I don't even know what this means. Right? It doesn't matter. Right? The point though, is that these things are.
Sort of arbitrarily complex and require real domain expertise. Right? And so if you're working on a program with which you have domain expertise, you understand what it's supposed to be doing deeply as you might if for example, it's your full time job to work on that program. Right.
Metamorphic relationship. Alright. So our Metamorphic testing might be a good idea.
Okay, so some notes about this, like, as I said, these are very programs with it I said, I don't expect you to understand. It's like, if you're, you know, studying for the exam, like.
Don't study these study in this box, so the, the minority relationship relations are very program specific. They require domain knowledge with to write and to really to understand, right? Like, I had to go and read, like, read the Wikipedia page. I'm just freed. wavelet transforms just to give this lecture, even though I'm not definitely not an expert right? But it's a good example.
And that's why I'm using it here and I didn't want to write my own. But but you need real domain knowledge in order to do this effectively. I couldn't have come up with us. Right that's why I told them from someone else's slides. Because, like, my expertise is in other things, and I've already used the compiler several times as an example today. So I didn't want to do that again. Right?
It's also the case that some of these Metamorphic relations have interesting properties, like, interesting mathematical problems.
For example, minority relation 1 here, transpose, uh, is, uh, community, right?
Uh, I remember what community means running the class.
Like can do it with A's and B's and c's.
Finds C. B. .
See that that's associated. Oh, my goodness.
Say it again? Yeah, I may have said the wrong 1, uh, or may have written the wrong 1 on the board say it again close uh, we've got this E .
Oh, uh, parentheses seamlessly. Wait, is it like 8000+see? Like outside of France?
Is it just, like, change the order? Let's just change the works yeah. Well.
Don't worry about it. All right. Thank you for taking the hit there.
When do you have your need? I would like that. Oh, you don't right? You don't.
Um, but well, I guess to understand this lecture, but you don't in real life. Right? But community has, uh, has some useful properties right? This is a standard relation that we know about right? And you can go look it up. And if you recognize it here, then you might be able to use this in more interesting ways. And in particular, uh, you might be able to combine it with.
Other right, so in practice, it is the case, the composition of Metamorphic relations, uh, tend to be even more effective than the Metamorphic relations themselves.
So, you have multiple of these, and you can use them all at the same time.
Right. Especially if you know that, for example, there community. Okay.
Um, and this is like this, this is quite nice. Alright, what this means is that if you do Metamorphic testing, if you're able to understand the domain of the application deeply enough and write out what the Metamorphic relations actually are and build generators and so on and automate all of this, you're going to get a testing infrastructure that's very, very effective.
Right. Um, and, uh, like.
The literature suggests that I.
This is 1 of the best ways to do testing right? In terms of like, getting good results in terms of finding bugs. Um, but it is quite expensive. Right? So, in terms of practicality yeah.
Sorry uh, because I okay, so.
Let me think about it for a 2nd.
All right, so M1 is transposed. Right. Uh, so it's.
This sort of trend, uh, this sort of like operation, right?
Um, creativity is reordering. Right so, um, let us to say that if you, uh, like.
Due to reordering. All right and you switch the order of them, uh, 2 rotation 2 rotations, for example, and you switch the order in which you apply them. You should get the same result.
Sorry, I had to think about that for a minute. Um, yeah, so this, this stuff is is tricky, as I said, it is, it is requiring us to to actually use the math part of our brands, which is always a little bit tricky.
All right, any other questions here before I move on and talk about, uh.
The practicality of this technique in practice. Okay.
So, um, do we think this is practical? What do you all think.
It's going to be hard, right? Like, I think so. The right way to be the practicality here is that this is, uh, this is the sort of technique that you should use if you like, very much care about correctness, right? It's, um.
It's 1 of the most effective ways to test real systems. Right? And it appears very nicely with fuzzy. Right? Which is known to be very effective and only costs machine time.
All right, so if you do the work to set it up, it's extremely capable and you can just let, like, throw machines at it for as long as you would like. Right. And so it will just keep finding right?
Um, however, it is hard, right? That is the key problem, right? It's difficult to apply designing. Metamorphic relations generally requires you to have domain expertise right? And it's easier for some kind of systems and others. Right? So, for some sorts of systems, that might be quite easy. Like, it was very easy for us to come up with a Google search example.
Right. That 1 was easy to understand by contrast the discreet wavelet transform 1 that we just went through required me to think about it every time that I said something because it was like, it's quite a complicated thing.
All right. Um, so for some kinds of systems, it'll be a little easier.
But in general, we're going to need some amount of domain expertise and I'll talk about the homework at the end, but I just want to mention 1 thing here, which is that this homework gives you you all a lot of freedom on exactly. Like, what system you're going to target there's no prescribed like subject program and this is the main reason.
Right, but you need your expertise to do a good job with this. And I don't know what domain domains you all are experts at.
Right. So when you're choosing subject programs to write Metamorphic relation relations for when you're doing homework, 5, try to choose ones where you have domain expertise, choose something that you understand. Well.
All right and that's true for both parts. Really but.
In general you the reason we're giving you freedom in this assignment is because you will have a much easier time, and you will learn more if you choose programs that you deeply understand. Right?
All right, so my advice here is always be on the lookout for opportunities to carry this out, right? It is a great value in terms of increasing your confidence versus the effort you need to put in. If it is relatively cheap if you understand the relations already. Right? If you need to develop domain expertise in order to do it, maybe don't.
It's probably going to be too expensive, but if you already have domain expertise, this sort of technique is quite effective.
Okay, any questions on any of this.
All right, so that is all I'm going to say about Metamorphic testing.
Before we move on, does anyone have any questions about minimum for testing broadly construed, or property based testing, or any of this kind of stuff? Because we're not going to move into something quite different.
All right, in that case, I'll let's start talking about dynamic and variant detection. Right? So the intuition behind dynamic and variance detection is that programs usually behave correctly.
Most execution trace says do not lead to failure right? Especially if you have a test suite and a human wrote that test suite.
All right, most of those tests are probably correct. And so, uh, if I have a human written test suite with, uh, some number of tasks.
And you have some facts, like, the index is always people who randomly from -1, uh, and every single 1 of those tests on every, every execution. Then it might just be the case that, uh, the Oracle here, is that all the time.
Index equals red line -1 all the time and this is the key intuition behind, uh, dynamic and very introduction in general. We're gonna run some tests. We're gonna observe what facts are. True.
And then we're going to, like, fuzz against those facts and see if they remain through an arbitrary.
Does that intuition make sense to folks? All right.
So, and before I go further, I want to give some definitions here in particular. I know the word in variant.
Few of you right? You've seen it before. So, new variant is a predicate or program expressions that have thrown every, uh, every execution right? That is to say it's a fact, like, or a credit card right? Back over from variable.
All right, that is always the case, right?
So, when you're writing doing property, based testing, usually what you're doing is writing down in variance.
Right, if you're, uh, writing or a partial Oracle as part of property based testing and variance is what you're writing.
Almost always. Okay, so the key idea here is that if we have high quality and variants within variants that are actually supposed to be true, right? Then they can serve as test oracles. And so the idea is, if we think back to property based testing effectively, what we're doing then is we're as a human writing down some variance and we're going to use.
Them as oracle's. Right the idea in dynamic and variance detection is that we're going to try to automate this process.
All right before we explain how this works, I want to do a little bit of background here about program analysis. This will also come in handy later in the class when we talk about static analysis. And in particular, I just want to give some definitions and build some intuition here. Right? So there are 2 ways for us to reason about what a program does in general, right?
Quote, unquote forward or backward reasoning forward reasoning works by knowing the fact that it's true before the program runs. Right? And then reasoning about what needs to be true after the program is done executing.
All right, and this is called, like, this is so given a precondition what post conditions are for.
In other words, precondition and post condition no, 1. good. So this next line is appropriate. So what are Pre and post conditions? This is part of the thing. We need to know these words in order to understand, like, how works right? So precondition to a function. Uh, is a condition that must be true when entering the function, right? So, it might, uh.
Include expectations about the arguments, right? But in general, this is like some requirement that must be the case before the function executes. Right? So, for example.
Suppose that I have function like this.
The C programs with Fu, we've got a voice.
And we do a p dot something we call the meta.
What must be true here okay.
Yeah, so he should be defined. Yeah, well, in particular, when we call the school function.
Has to be a notch no pointer. He is a no pointer. Then this is going to stop call.
All right, so, uh, the precondition here is that, uh, he is no, not.
And that's just not right it's a fact that lead on the recently department. Correct?
You're used to doing this almost like a.
Taking this way, all this of what your program, which probably having to formalize this intuition.
Another example, this is, uh.
Let's take, um, what's that? Just imagine like a, it's app. This is a silly a function.
Takes 2 minutes, X, X and Y, and then returns X plus Y.
What are the premium? Is this function? Yeah.
Yeah, they're written right here. Right? The static types, static types of parameters for preconditioners.
In particular, they're saying that, like, if X is not an event, like, this is going to go, go forward, right? If you push, uh.
Like, a float value here, we're gonna get sort of nonsensical results.
Right even worse if you put a pointer here, or even like worse than that, you've put some other thing in there.
Again, the results will not be appropriate and so you can think of Pre and post conditions here as the promises that are made, uh, the requirements of the function before it's called. It's a precondition. And then the promises of the function makes about the results. Right?
So, in particular, the record type here, as opposed condition.
Right this function is promising that if you give it to end. So we'll get an impact.
Alright, you give it to other things. It's not promising it.
Of course.
Like, as I said here, I.
In this definition, you don't these do not need to be about the arguments, but they often are right the the preconditions that you're usually thinking about for functions are, like, requirements over the arguments, but that's not necessarily the case, like, imagine that you have a method that references a field without checking if it's not.
That that's a precondition that involves the field and not an argument, right?
You can also have things like preconditions over, like, the properties of the file system. Like this file needs to exist for this program is not going to work, right or something like this. Okay.
And post conditions, as I said, are implied, I guess there is some, some credit that are going to be true when you're leaving the function, right? It's the promise that the function makes right?
Uh, or, uh, put another way it's, uh, what what.
Uh, what will be true after the functioning finished executing.
And again, it often does, but may not have, doesn't have to include expectations about the return value. So, the simplest examples of these, the ones that you've almost certainly seen our argument or sorry parameter types and return types.
Right. Uh, respectively for these 2, but there are many many authors that you can imagine. Right?
All right any questions about these definitions, do these 2 concepts make sense?
I feel like, I've probably reason using Pre and post conditions before without realizing it.
Yeah, almost everybody. Good. Yeah, so the, the goal here is to formalize the notion that you hopefully already have.
Okay, so, um, coming back to this notion of forward and backwards reasoning right? So, uh, for reasoning is saying, like, okay, we know a precondition.
And we're trying to figure out like, what post condition is true based on the precondition and the program. Right? So, if, you know, for example, I raise this.
We're 25 things here, you know, that X, isn't it? Do you know why isn't it.
You know what? Plus does, can you tell me what the are from Texas?
Right. It's easy enough to right here. Right? And that's an example of for reasoning for taking these 3 conditions and you're taking on the program does and you're figuring out that there should be.
Backwards reasoning is exactly the opposite. Right? The idea of backwards reasoning is that if, you know a fact, that must be true after execution right then you can reason about 1 must be true beforehand. Right? So, if, you know, for example, again, taking this, well, we know.
We have 2 arbitrary X and Y, here, like, we don't know anything about those.
We know that this for X, plus Y, and we know that we were from the audience.
What must be true about X and Y, also have to be in right? Otherwise the program doesn't make sense.
And so I'm back, the reasoning is just going the other direction, right? It's reasoning from what actually has occurred and what, you know, afterwards, and figuring out what the inputs must have been. Right? Yes.
It depends on the semantics of the language and, I mean, I'm imagining like, a seat program here or something. Right? I think program this must be true what I said, but I.
Yes, I'm I'm definitely abstracting here.
And this is also kind of a bad example, I probably should have put a more complicated example in the slides. That was exactly correct.
All right, and so, in other words, backwards reasoning is given a post condition. What preconditions must be true.
Right. And so these 2 things are effectively allowing us give it for reasoning and back our reason each allow us to reason about a program.
And get from either the preconditioners to the post conditions, or from the post conditions to the conditions.
So, if you know, uh, a precondition, um, and, you know, the program.
And you have some kind of analysis, you can usually compute a post condition and vice versa, right?
Um, by using 1 of these 2 kinds of reasoning.
Okay, so what do you think what what are the pros and cons of these 2 ways of reasoning about programs.
Why don't we take say 3 minutes or so you can turn to someone next to you and discuss this.
And then we'll come back together.
Hello.
We're ready.
So.
Unfortunately, you need to understand the program is probably a kind of both.
That's a more of a kind of reasoning about programs in general, but I do see your point about backward reasoning.
Being a little harder maybe right um, in, uh, like, when you, when you discuss this for, we're like, when we run studies.
It is the case that people find most people find easier to do for reasoning backward reason. It.
Yeah, uh, I guess a pro for that, for the reason, it is.
When you're writing the program, usually have like.
Some kind of desired output.
In mine yeah, especially with, like, specific ones.
Yeah, so you do you have desired outputs in mind you know, what the behavior should be and so if you're doing backwards reasons, you can figure out what you need to do in order to achieve that.
Yeah, other ideas.
Uh, my boss, it's like.
Uh, without it sure. Um.
So, that's definitely the case for both kinds of reasoning right? For large programs is going to be hard. Right? That's just a fact about program like, raising about programs in general.
I'm not sure I understand your point about.
Being particularly like this being a particular problem in backward, reasonable.
It's certainly the case that, for reasoning has a tendency to produce, like, overly large sets of facts.
Um, we just to say, like a.
When you do for reasoning, you often end up, like we're using about lots of things that are actually relevancy that we care about. Right the nice thing of 1. nice thing about backwards reasoning is that people do it in a relatively targeted way.
Usually using backwards reasoning it's possible to say I suggested over here that, uh, you know, we have a particular output that we care about.
And we need to figure out what we need in order to do that. Right?
The reason we have to accumulate all the true things and then eventually figure out, like, do all these 2 things at what I care about.
So, for really kind of the sales force.
But again, this is analysis dependent.
We have now look at.
Isn't that? So.
I think the options that.
It will be stage, so it might be.
Overview is this applicable.
So, when you're doing for reasoning, uh.
I I think at least my intuition is to be sort of the opposite way, right?
That for reasoning is more likely to be hard to, uh, to figure out exactly what's going on again. This large sets of facts problem.
Alright, board reasoning when you're when you're listening forward, you're finding on all the proposed conditions that are true. Given a set of Pre conditions.
You're going backwards racing, you're home. Like, you can start with just the most initiative you care about.
And then you only find the preconditions that are relevant to it.
And so backwards reasoning allows you to find like, a more relevant set of facts.
For a score reasoning, allows you to find like, all the facts that exist like, where all the facts that are that.
Uh, so what I was kind of saying is, like, some option.
Uh, 1 request, so that it can be really on.
Yeah, I'm not sure. Yes so like, what, what do you think we can.
And same on the phone with, uh, maybe.
Feedback for reasoning, it's, uh, it's the same maybe to, uh.
But it's hard to, uh.
Okay, I understand your point now. Yeah, so I.
Certainly, it is a case that for specific kinds of functions and many to 1 functions are much more common 1:2:main functions. So, this is a great point.
Um, Ford reason is going to be more efficient. Right? I think you're going to think that's what you're saying, right then backwards backward reason it's going to have to consider many things and they split all this apart.
Right so the, the, the run time of analysis, let's say that's running a backup reasoning is likely to be longer than a board analysis and the examination.
I have a list here as well, and I want to just put those on the screen and let's let's go through them together. Right?
So, um, the key top line takeaways on, uh.
Once you all to remember about this, is that for reasoning is usually more intuitive. Like, when your reasoning about a program, you're usually doing for a reason implicitly.
When you're thinking about, you know, what this program does, but backward reasoning is often more helpful right? In particular, uh, usually, uh, if you want to understand a particular, like, why a particular outcome happens, backward reasoning is going to be how you run to achieve that.
All right, so, um, yeah, so so for everything, it's like, understanding what will happen and backward reasoning is, like, understanding what did happen right and, uh, or what should happen, or, you know.
Like, what the desire like the desire is right it's better for thinking through, uh.
Like, what the spec should be as opposed to imagining how the implementation will actually be here.
Right. And thinking about the spec is often much more useful, especially when doing something like property, based testing or thinking about in variance. Right? We usually care more about what the specification is supposed to be, just to say what the program is supposed to do, as opposed to what it actually does it. Right? Um.
So, uh, foreigners in these regional impacts, maybe are all relevant to the goal and the side effects may get large. This is this point. I was, I made several times right? Um, and that the reasoning is better uh, ideally specific. All right.
Um, it also is the case that it's, uh, much, it's often hard to realize the court reasoning and not going to work. Right? You can spend a long time simulating what the code is doing before you realize that oh, actually, this path is not going to lead to where we need to go.
So, hopefully not by contrast, um.
Back the reasoning is great if you know, where you, you know, where the problem is and you're trying to figure out why it occurs.
Okay, does anyone have any questions about this.
You can think of it that way. Yes. Yeah. And we're going to come back to this notion a number of times as we discuss various analysis. All right.
Especially as we get into the sort of like.
I start discussing dynamic analyses that rely on abstract reasoning, which is icon as an example of what we're gonna talk about more as we go forward. And as we get into this tech analysis, part of this, this class.
Okay, so why are we talking about all of this? So, going back to dynamic and variant detection the key insight here is that if we have some program location, and we can infer and invariant for that location, that is to say, we can figure out what what facts might need to be true.
And we could produce post conditions and Pre conditions.
Right. Which are the things that we're actually going to need in order to reason about the program.
The key the really nice thing about dynamic in varying detection is it? It's not just a test Oracle production technique.
But it's a technique that's going to produce function Pre and post conditions.
And, uh, as we'll maybe discuss slightly loop in variance as well. All right and this means it's going to be very useful for static analysis as well. Like, these are the sorts of things that a static analysis tool is going to be able to take advantage of right because these are facts about what the program is supposed to do.
And as I said, Luke, Luke and variants here, I've heard of loop and variants.
I'm guessing almost. No. 1, right? That's fine. All right. Um, this is not something you need to know yet. Uh, in particular, this will not appear on the midterm, but we will discuss it in the 2nd, half of the class.
So, libertarianism, invariant, it used to hold up at the start and end of each iteration of a loop.
And it's extremely important for, uh, a number of verification techniques.
Uh, or, uh, doing proofs about programs, right?
In general, if you want to do it for for better program for each function, you need a precondition in a post condition and for each loop you need to loop and variant. And if you have all 3 of those things, then you're usually able to produce a proof.
Entirely mechanically, these are the 3 things that typically a human, uh, to give you before you can invoke an automated for checking. Right? And so, 1 of the really nice things about something like icon is, is going to allow us to automatically produce these things.
Now, they might not be correct as we'll see. Uh, but, um, you know, in theory if we had a great test suite, uh, Dotcom would allow us to, uh, you know, sort of scale that test suite into verification.
Okay, so, as I said, we'll come back to this concept later in the semester, but for now, don't worry too much about it, but I want to introduce it here because again, these things right?
Okay, so the key question is, can we do this automatically right?
And there are 2 insights that allow us to do. So right. 2 insights that lead to, that are kind of algorithm. The 1st, is that, uh, invariant by definition always holds on every execution.
Surely right it has to. And 2nd, that we can easily detect spurious false in variance. Have.
How can we why is it easy to detect false environments? Yeah just run the program. Yeah, if you run the program and the variance is ever false, then it's definitely the case that it's not an investment. Right? The quote unquote investment.
So, if you, if you run the program and you ever observe that, the predicate that you think might be in the variant is false.
Then it must not be by definition, like, in variance, as we said, always hold on every execution. And so if you ever observe it being false.
Then it must not be an environment, and this is the key insight that leads to icon. Right? So, if we, if we require the program comes equipped with inputs, like an addictive workload or high coverage test cases, which, as we know, is very easy to produce.
Then, since the invariant would hold on every execution by definition, any candidate invariant that is false, even once can be rejected.
All right, and so this is the key, as I said, this is the side.
This is how works so, the plan then, is we generate many candidate in variance.
Pretty so large set of them and then we filter out the false ones by running the tests. This is the basic idea of dynamic and variant detection.
All right, so let's consider them a naive approach to this. Right? So here is a program while.
Right. It's we're bullying condition and see some command then we can instrument it to 3 times and very warm and very intuitive. Right? Uh, at the beginning of the loop.
And then we can run the tests, and we can see whether, uh, very 1 variant 2, very important so on our true or false. Right? And we just remove all the ones that are false.
And then we do the same thing for each task. Right? This seems fine.
Or does it all right so we run the test and filter out those that are false. So what is wrong with this plan?
I've left out something very important. This is a, the slide title says naive approach for reason.
What is wrong? Here does a hint where do the variance come from?
Sorry, what is true.
The variance, or B or something.
What is the condition results? Yeah, that's true. Um, but we're interested in. Let's let's suppose that we're only interested in the case for expert.
This is still not going to work right? And the reason we haven't defined where these variants come from. Right? So, let's, let's think through what are all the possible candidate variances in particular. How many of them are there?
But another way how many Boolean expressions might might be true?
Which ones are we checking? We haven't said all right and implicitly here, uh, like, what what we're suggesting is that we're going to try all the candidate in variance.
All right, we're trying variant 1 and variant 2, et cetera until we're done. Right? But it turns out that there are infinitely many candidates, right? Oh, yeah. Well, hint. Um, there are infinitely many candidates.
In general, if we're looking at Boolean expressions, right we can keep making more complicated ones forever. Right? And so a naive approach like this, if we don't do something clever in order to define the space of invariants that we care about, obviously cannot work.
All right, and so, um, this is the, uh, this was the key problem that needed to be solved, um, in a, uh.
A useful way in order to make dynamic in varying detection practical.
Alright, so icons key contribution is not so much the idea of dynamic and variance detection, which people have thought of before, but had never figured out how to do.
All right, the icons primary contribution was the idea of templates.
All right, the key idea here is to choose a set of finite useful in variance. Right? And that will actually turn out to be good oracles. Right? And, you know, do some science to figure out which ones actually are good.
So, uh, the set I, so, for example, given some program variables, X, Y, Z, you might consider these 7 sets of variance, right?
Uh, with these names, and these are directly from icon right? So, for example, um, dot com, we'll consider whether, uh, a variable is constant, right?
But, uh, uh, Eric was not 0, right? Or that it's free of variables in the concept uh, bad right?
Or there's no linear combination with 1 other variable or 2 other areas.
All right, but it's not going to consider, like, a quadratic relation. The icon would never try.
Something like, uh, is that equals a square? Sorry? A X squared.
Uh, plus B, Y, plus a quadratic equation is out entirely out, right?
The reason for this is that this kind of.
Equation is rarely useful in practice in real programs.
The reason that these, uh, specific is this specific set of, uh, templates is chosen by dot com, because they actually tend to work well in practice and tend to be useful.
Right. And so in order to actually do practical dynamic in variance detection, you have to have a good set of templates and which those templates is good does very amplify program. Right? So comes with a pretty good, like, sort of generic set, but, like, the most effective use of icon requires you to go and think hard about what, uh, what sorts of templates might be useful.
Your kind of program and so then it kind of turns into property based testing. Yes.
Yeah, so great point. Right? So, in that kind of code, you probably do need a different step. Right? So, that's a, that's a great example of a case for, like, this default set, which is for, like.
Data structures, it's probably not a good choice, right? Um, yeah, so.
Base set of templates is based on, like, the idea that you're writing, like, some kind of data manipulation, like simple data manipulation program, like, of the kind that you would see in, like.
An undergraduate like a high level undergraduate software engineering class, right? You're doing something yeah.
Data munging of some time moving moving objects around.
But for, you know, physics simulation, you definitely care about other kinds of things, right? Certainly, you probably do want X square or something, right? For machine learning like.
Something something went around, I don't know how to do machine learning, so, but you need different kinds of various templates and 1 of the things that is nice about icon as a system is that, like, it builds in support for changing the set of templates. Very easily. Right. It's pretty easy to do.
And, yeah, so the idea here, the other important idea is that dot com is strictly limited to 3 variables and therefore there's a finite number of these and variance.
So the number of ingredients we need to check is where we're kind of what we have at times, the number of ways, we can instantiate them. Right? But or with 3 variables, which is a finite number, it is no longer infinite.
Right and in in particular, like, the number 3 here, 3 variables was chosen because that's a good balance between.
Uh, being too expensive and not producing enough in variance that are useful.
So, there's our Parrot, that's the, that's a icon. So then the actual detection algorithm for, is it a.
It's a radish. Okay. Yeah. Wow. That's all right. Sorry. I don't actually know what I'm talking about when it comes to vegetables it turns out. So my apologies.
But you're not here for vegetable class. So, um, yeah, so the various detection algorithm that icon uses is the following so, for every program location for all the triples of indigo variables, instantiate all the templates to obtain a set of candidate in variance and then instrument the program to print to, to check them. Basically right, and then for every test case, run the instrumented program.
And filter out any falsify candidate in variance right? So anything that was false removed right? And in particular, um, also, I believe, uh, reduce the instrumentation.
Repeatedly in order to remove false candidates.
Uh, to make it faster, and then at the end report, the surviving variance to the user.
Yep. All right so any questions about this high level algorithm.
Alright, so then, uh, what's this running time?
I'm looking for answer in, like, big on a patient.
What does it depend on? What's.
Okay, so, uh, where did my racer go? There? It is.
So, the suggestion is build and where it is, the number of variables.
So, why do we think it's big of and in the number of variables? What operations? Linear that's the most expensive 1.
Maybe put another, what is the most expensive operation here?
That's true.
So, uh, okay, let's just just looking at this, right?
Um, this is like all of this work is linear in the number of test cases for sure.
All right, this work will be this about the same regardless of how many.
Test cases we have right but it will get larger depending on how many candidates variance there are.
Okay, this work, right? Choosing candidate a candidate templates.
Is going to be not linear in number of program variables and because we're looking at all triples of program variables.
How many triples of program variables? Are there any program variables? How many triples of them are there?
Yes, and cute. Right? So, in fact, icon is N, t0 N cube in the number of program variables. Right? And that is definitely the dominant term, right? So, it's, uh.
Cubic and scope variables and then linear anticipating program size.
Right. But, uh, this cubic of variables is brutal. Right? Icon is very slow. There are many in scope variables.
Which is weird. It's a very weird behavior because often most programs have only a few variables in scope in most places.
But do you have, like, a method that has many many variables in it? Really slow?
All right, um, and this is a warning to you about homework 5.
You get to pick the program you run the icon on. This is a track.
All right, if you choose a program that has, like, 17 in scope variables don't be surprised when icon is very slow.
All right, choose a program that has, you know, it's reasonably well written in terms of the number of variables at a time is fairly low. Right?
All right, and this is a choice, right? This all triples was definitely a choice. We go back and we look at this example, right? The icon made the choice to do it most 3 variables right?
And that reason was because, like, the, the number, like this number 3, here is the reason that it's.
If we did, and most 2 variables would only be end square. Uh.
Uh, and the number of installed variables, right? If we chose at least add most 4, it'd be end to the 4. right? And so, like, this choice of 3 is the dominant term in the running time of the algorithm.
Okay, any questions here does this make sense?
Yeah, so we'll check every combination of the 3. yeah, that so if we're looking at these in variance, right?
Like, if we look at, like, this kind of linear environment, you know, produce 1 here for each, like.
Each possibility of com, combination of 3 any other questions.
Right so moving on, we're going to do a quick in class exercise here. I want you to again, like, talk to someone nearby and in 1st unlikely invariants for a relatively simple program. Right? So, what I'm asking you to do here is to reason as if you're like, as if your icon, right or reason in the way, the same way that icon would write about it.
Simple program here right? And I've given you a precondition and is always greater than 0.
All right, and that program hopefully will be quite easy for you to reason about, I hope. Right and then here is a set of candidate in variance right? And your goal is evaluate all all of these invariants.
Uh, at, uh, all the locations that.
Say, I would imagine this is important.
At the function entry function, things it and at the beginning of the end of the.
So that sit here and here, and, uh, then we will, uh, cross all the ones that, uh.
All right, so, uh, say 3 or 4 minutes.
Okay.
Right.
Okay.
We feel like we have, uh, basically solved this problem.
Approximately 1 group, so let's take another minute or 2.
Sure.
Security is being paid for the looking variable. Sorry?
For this, so the working date is that.
Yeah, so I mean, what I'm asking for you to do here is just, we're just mentioned there are 4 places what we're going to change.
I'm not asking you to set up properly, so that's it.
You could Bruce yeah.
I was just checking.
That's the 2nd thing I can share this with some.
All right, who is feeling ready to go through this.
Who would like to keep working? Okay no, 1 is voting. So that's, uh, I think it's, it's time for us to move.
All right, so our goal here is for each of these possible and.
I would like to decide at each of these 4 locations, I should say entry the, the function.
Entry to the loop exit from the loop and exit from the function.
Whether this candidate in variance is true or false. All right, let's start with, uh, entry to the phone.
We can start with 0, right? Is it the case that is 0, mentioned to the phone?
Yes, all right. Wait. Are we talking about? What what are the functioning starts?
As I isn't even said, well, we can't falsify though.
Oh, it is uh, I mean, that's that's a fair point, right? In fact, the icon is not gonna print anything about I, as a function of recognition, because it's not in scope.
Right but let's imagine that in that case, uh, well, we don't know what it said to you before it's going to be sent to 0 anyway. Right? So all the ones about, I are kind of irrelevant.
All right at precondition time and so we can, we can basically that's the correct thing to do.
It's ultimately with all the ones that end so is any goes to 0.
No, false how do you know it does? So we can do that any greater than 0, uh, uh, confidence right? So, uh, this is not a free condition.
Oh, how should I mark that? Let's just say.
Uh, 3.
All right what about? And less than 0 same thing. Okay and less than or equal to 0.
Not a free condition and greater than equal to 0.
Oh, no, it's true. That is true. It's definitely greater than or equals. Yes. Yeah. It's greater than so.
That is exactly the correct explanation. Right? Yes. So so this 1 is a Pre condition.
Right because, uh, and very 0 at 5.
All right, and grid 0, which we have, we know is already as a precondition. We don't need to worry about that. 1. right? High profile involves with ignore it and the same for directions. Right? So, uh, the correct answer for preconditions is these 2 are free conditions and the rest of them are not right or I is not considered. And so the islands, you can imagine all.
All right let's do loop entry.
I'm going to abbreviate as all. Right? All right. So, at loop entry is equal. 0.
Yes, it is.
I submit to you any, I or sorry for any, any greater greater than 2. I believe this is definitely not the case.
So, in particular, right? Yes, I is 0, the 1st time you entered the loop.
But it's not this the 2nd. Oh, yeah. Right. Yeah. So, entry here is, is the beginning of the loop body.
Right. So it's not before like, this, this spot doesn't matter. So it's not privileged. Right? So, if any, every time we go around this is where, the spot that we're interested in.
Why do we know that? I'm sorry? Why did they ignore? I think it's out of scope.
I as a local variable here, so it's not, uh, like these, these, uh, like, I agree with 0 is meaningless because I hasn't been defined.
So, the precondition is is visible outside of the function.
All right, so it only can talk about. And because end is the only problem.
I is in prod, so.
It has been something sorry it has to be the thing.
So, when we were discussing preconditions in particular.
We can't discuss, uh, hi, because I hasn't doesn't have a value for you yet.
Say the function, but inside the function it does so at the loop entry point, right here I have been set to the value, right? It's either or 0 or it was set to buy this assignment. Right?
And so, at this point, like, we're, we're in the function so is, is installed.
Okay, okay so this is not.
A loop entry and because, uh.
If you go around the loop a 2nd time, then it will not be for it.
Right, yeah.
What happened? Yes. Oh, wait no. Yeah, definitely not. Right I believe that's actually false on all all executions, right? And the same for less than or equal to 0, right? Mm. Hmm. Because these 2 are balls so it's definitely false.
I agree with 0 loss, but not in subsequent.
Well, it's true in every iteration after the sort of stuff. Yes. Yeah. So the 1st iteration is false.
And so we're going to just right, because it was ever a point. There was ever point. It was false. We get started.
Alright, alright what are the? I agree with the right.
That 1 is actually a little bit a loop entry environment. Yes.
0, well, all the endpoints stay the same, except the ones that involved on this end was compensated.
Yep, that's a great observation. Right? Um, it is worth noting, though, the icon is going to, uh, evaluate the multiple right uh, every time, all the free ones. Right? And this is part of our dot com is expensive, because it's not able to do a brief like that.
You're implicitly using a fact about end here, which is that it's constant. The icon doesn't have access.
As a human conversion about that, but, yeah, the analysis not right so, yes, the icon will thrown out these ones that are not preconditions about, em, but it's going to evaluate both of these at every venture, right? And they're going to still be true because, as you said, and it's constant, right? And now we get to the excited ones.
Where we compare on it right? Okay. So equals equals that is that for the country?
No, and in fact that it's definitely false it's fault on every, uh.
Of execution, right? What do I less than that?
Yes, it's a requirement for a.
Sit there basically why there's a, basically in there, right?
Well, because it has to be greater than 0.
And and it can be like a half. Right? So we get over a few.
We are assuming these are intervals or right they're interval values and so, yes, this is true. Right? So, I mean, group of it is relatively complex.
By entering zoom don't you're not including like the evaluation of high it's not equal to it. Yeah. Yeah. So that's why I've drawn a little mark right here where we are actively with the guard has been evaluated. That is true. Right?
Because this is where I'm going to check, right right after we evaluate pick on.
And in particular, like, this, I, less than end is an excellent example of why dot com is effective right? Because the icon is going to is going to find that this is a loop entry in variance.
All right, because it is true, right? But the proof is relatively complicated in the sense that we need to use several facts in order to get to this.
Right. Actually doing a proof of the fact that I is less than in here requires us to reason about the fact that end is non 0 or not negative. I should say, right it's positive actually. Right and it requires us to reason about the fact that I is monotonically increasing.
And it requires the reason about the fact that I not able to, uh, N, and the fact this is monotonically increasing and the fact that I starts with something less than end means that we're not going to be able to get into a situation where that falls.
Right like that group can be done right? Like, that's the sort of proofing might've had given, like, say, just re, map boss and your undergraduate education.
But a program analysis systems, going after work relatively hard enough to do that for them, right? Like, it's not obvious.
I'll do it, but the icon is just going to observe the fact that on ever execution happened to be true. And then assume must be fair.
Okay, what am I less than realtime? Yeah, I mean, it's the same thing.
Yeah, so exactly the same reasoning applies. And in particular here we can take advantage of the fact that we have. We know that I lost in industry and this implies.
Yes all right. And so if this is true, this must be true as well.
All right, I agree with that fair stop and then last time we did.
Oh, no, wait no, why don't you don't enter the loop if, um.
Times greater than well. Yeah, but when I spoke to 1 that we increase the number, I.
And then it's not so, that's that's the case, uh, here, right that look like that, but not at the entrance. There's no new entry. Yes. So it's not a loop entry. Marianne.
That's right. The easy way to say to show. It's not here is to find an input for which it's false at this point.
Right and that's the right way to do it, because that's what the icon stuff right? So, I think if we just select, uh.
1, then this will work, right? Because then the 1st, time for the loop, I will be 0, and will be, and 0 is not created in 1. and so we're done.
And we found 1 time, for example, so it's definitely the case it's not a little better and better.
Okay, and then the same applies for the other ones. Okay. So now we've checked all the does anyone have any questions? We have a little better environments.
Right now we've gone through these 2.
Who thinks that they have all the right answers to the, the variants.
Yeah, why don't we take 1 minute to go.
Go through your answers that you wrote down to the database and the variance and double check them and then we'll come back to that.
Hello.
Okay.
Hey.
All right, so I would like to go through both, uh, the loop exit and the function I did at the same time, right? I'm going to call these, uh, this 1 is gonna be and this one's going to be, uh, key, which is the traditional post conditional notation. Because key is the precondition that he was in that slot, right?
So, uh, let's go through all of these, uh, like, candidate experience and check them in both of these locations right?
So, let's start with I, is 0 exit is like, do we have a counter example for.
Or, like.
An example of where it's not happening.
Yeah, so the, the thing that icon is doing here recall is it's fussing it and looking to see if there's anything that violates the property.
Right. So we can find an input for which this property cost equals more. Yeah, sure. So this is important. This one's definitely not true.
What about, uh, functionality? Okay, same thing.
Okay, I less than 0.
Yeah, also yeah, exactly. Yeah, definitely not true. Same same reason.
Less the wrinkles here. No.
All right I greater than 0.
So, I, I agree with for here, what do I.
Uh, well, it doesn't change, so mm. Hmm.
Well, does the loop have to have to have been executed?
By in general are allowed not to be executed. Right? So, conservatively, we have to consider at least the possibility that this loop will only go will not go around even once.
So, do we have a proof that we'll go around once.
Uh, yes, because and is, yes, because I is always 0 at the beginning, and it's always greater than 0 and therefore the, uh, this 1st test always is true. So, yes, I heard and 0 is the vaccine variant and it as opposed to initially.
Alright, greater than equal to 0. Oh, it's the same thing easily employed. Yes.
We're going to skip the ends again, right? Because then it's constant.
All right I equals equals that. True.
Exactly yeah. Okay. So at loop exit.
Is it true? Uh, I mean.
We're only working in space, right? We're only working in space and I think it's important to note here that what I mean, by loop exit is immediately after the last part of the body of the loop is executed. Right? And that means just like the entry 1, it's going to be evaluated for every time around loop.
All go into 1 thing. Yeah, there's only for 1 time. That's exactly right.
Only true when the Lucas about. Oh, right. Okay. So, uh, this 1 is not a loop exit in variance, but it is a post conditional function. Okay. Alright because, uh, it's implied by.
Yeah.
But it's not through every time, every exit if we only evaluate it for an equals 1.
All right, if we're only test case was equals 1 icon would report this will do bugs and variant incorrectly because then it would have been through every time because it was only about our at once. Okay.
But for if you have any other app, I will notice that this is as effective.
All right what about, uh, yes, you know.
All right, so it is a loop exit. It's not it's not going to go in the last 1. yeah so on the last 1 spots.
All right, but on the others, it is true, right? But there's 1 word false so not live exit and it's definitely not a best condition, right?
But I less than or equal time to.
Why is it true? I assume you mean for Rebecca.
Because the only time it's.
Not less than this. 1. it's low, so Yep. That's exactly right.
Yeah, so this is, in fact, a loop exit variant and it's a loop entry in variance, which, when we come back to the loop and variants later, you're going to see this and be like, oh, man, that's a loop in variant. That is the 1 loop in variant it's going to let us prove this program is correct right? So, this is the variable. This is the really nice thing about the icon is that it it is going to get this right answer.
Right. It might come up with multiple candidate loop and variance, but at least anything that proposal at least plausible.
And if it only finds 1, it's almost certainly the right.
Okay, is it a condition? Yes. Yeah. Yeah. It's a, it's a loop is the last thing, so I yeah, yeah.
Okay, what are the library for that?
On no, does that ever heard? No, no. So that's easy.
What about I greater than or equal then? Yes.
Equals okay, so, uh, yes, for the exit or yes, for those condition or yes, for post condition is definitely correct.
Around the fact, sometimes it's only the last 1. yeah.
All right, and there we go. Alright, anyone have any questions.
Yeah.
Yeah, so, uh, this 1 so is less than or equal and here is interesting because it is both a loop entry and variance and the loop exiting variance, which means it is actually, this is this is actually a would've been married.
And as I said, we're going to come back to those environments and we need to talk about site analysis later. Right? But, uh, this is 1 of the nice properties of icon is that it tends to fine with the variance with high probability.
They tend to be useful.
I.
Yeah, it is. That's also true. Yeah, that's a good point. Right? The reason I highlighted this 1, is that, uh, this is the 1 you actually in order to make interesting.
Um, and this 1 is not, uh, that's 1 of the things that icon is doing here is yeah, it, it has kind of both of these right? And that's why a human needs to interpret this.
So, good point, I didn't even think about that. Totally.
So those are the 2 little parents again only 1 of them is actually useful, but there are both.
All right, any other questions here last time, going to erase everything on this part of the white board.
If you have any questions about any of this, we're going to do this, then we're going to pass the slides.
All right, so moving forward, then let's talk a little bit about some limitations of of this technique. Right? So we've already, uh, hinted at some. All right, but there are the, the real question here is, does this produce.
Only true in variance, like, actually true ones or can it produce either false positives? Or can it miss real true in variance? That is to say it has false negatives.
So, the real quick the question I, I'm asking you all is, uh, icon, false positives, false negatives or both.
Or not oh, okay.
What kind of false positives would it produce? Well.
If that last 1, if the loophole yes. So ones.
It'll actually integration is sustainable. Passive conditions. Yep.
Yeah, so that's that's a great example. So it can definitely have spurious, false positive in variance because you have a insufficient number of tests, right? Yes, that's a that's a good example. Right?
Can it have false negatives that is to say, are there true and variants that it might not produce?
So, if it considered every possible variant, right then it would never miss it.
However, recall what we said about sort of a naive approach that considers every possible in variant there would be infinitely many of them.
All right, and so the icons certainly cannot consider all of them. It's only going to consider in variance that are in it set of templates.
If you run the icon without a particular template, that isn't the environment that you were looking for it will never find it. Right? And that is actually the most serious limitation of dot com, right? In particular. If your variance is not in a template, the icon will not find it. So, for example, the variant on the screen, there is the 1 that you need in order to prove binary searches. Correct? Which is, I think the example from the Tony.
Paper I mentioned at the beginning of the class, and you will com will never produce this. There is no chance of coming up with this. It's not in the standard templates and so it would not come up with it. Unless you added a template specifically for this, and in that case, you knew what you were looking for right?
Nothing prevents the icon, like algorithm from finding these in theory right? The problem is having the right template on hand, right? If you don't know the actual invariants that are, like, are important a priority, then how would, you know, which templates to run icon with right? You need to have some domain knowledge about the program in order to do this correctly. Right? And this is the really.
Running icon in the standard configuration comes with some, as I said, it comes to some template in variants that are pretty good for the sorts of programs that you might write in class.
All right, but for a physics simulation or machine learning code.
Maybe not, you need to choose some different ones right? And so you need to be aware of what you're doing in order to choose good templates in order to get to produce useful results.
So that said, templates are absolutely necessary in order to promote on the scale. The reason that you can run the icon on real programs, is that the template, like, the, the set of templates is actually quite small right? But that's also the reason that it misses. So many things.
All right, and this is a trade off, right? There's a direct trade off here between the, the kinds of environments and confined and the runtime.
That's why we've talked about how the run the run time is determined by this number of templates.
And by their complexity, right? And so is a direct trade off there.
So, if you have infinite resources, yes, you can find arbitrary and variance with icon.
But, in practice, you've got, and then as mentioned, there are also false positives. So there are 2:2:primary causes that are going to cause you problems in practice. 1 is limited input, which I think we mentioned right? So, if you only test your sorting program on, for example, the input 403, I kind of learn it helpful. If there is like the 1st element of the output.
Now, is that true? No, not for an arbitrary sorting program, but if you only use 1 input, then you would find it. This is like the example that we saw on, uh.
The 1 was on the whiteboard where, um.
If you only run the loop, once you get stories in variants.
Okay, and fortunately, as we've learned making high coverage, high adequacy test suites is easy. So surely you will be able to provide 1 to and therefore get.
Uh, good results. Of course, that's part of the whole thing that we were talking about in this entire class. Right? So, this is this is actually quite hard. Right? And so, therefore, like, the right balance here of when to actually start running the icon and enforcing its and variance and when to just, like, be writing new tests yourself. Uh, so that you can frame.
Basically, it's quite difficult and so it takes some practice maybe or skill in order to decide, like, is the current test was good enough in order to, like, start using icon and enforcing the variance is found or not.
I don't really have any particular advice for you here. Unfortunately, it varies a lot how much input like, how much coverage for example, you need in order to have icon produce good results across different programs, right?
Um, the general experience that I have heard from people who used icon is that.
Once you've done this a bunch of times, you kind of get a feel for it, but you all will run the icon as part of an extra work assignment. And so, you know, maybe on a hopefully small program but 1 of your choice.
And hopefully you will, you will start to get a feeling or an intuition for this as well. Okay.
The other thing is that you will get false positives from coincidence, basically, or facts that just happen to be true but are not about your program.
So, for example, you might has a tendency to learn facts like, X is less than or equal to Max and.
Uh, or, uh, point her at some point, or mod 4 is always 0.
Why is 4:00:their addresses, right? Yeah. Like, you can't address into the middle of a bite so.
Oh, yeah, they're all divisible by far they're all the visible that 8, right? Yeah, so they're all, they're all memory addresses. So they all end in 0 resonates and things like this and then they'll end in 1.
Back on, we'll learn this if you run an program, right? I think the icon has filters to filter out the obvious ones of these. But, you know, it's very easy to, uh.
Encounter more. Yeah so these are not false. They're true. Right? But there's not useful.
These will never be false in particular. They're always true for all programs and so they're boring. Right?
And icon, unfortunately, there's no generic way to filter these out.
Any questions about either of these false positive causes.
Okay. All right so this, this is all I'm going to say about icon. Does anyone have any other questions about dynamic detection?
Wrap up yeah.
Unfortunately, I already erased them. I don't have them written down on the slides.
Yeah, sorry you will have to redirect them. Can you go back.
Yeah, we can look again here there. So, uh, the most interesting ones are going to be, uh, this 1 and this 1.
Which are the loop in variance right? But I.
Yeah, I will go through the slides maybe tomorrow and update a copy that has the, uh, the answers on there you can speak to. Mm. Hmm.
It's fine. I'll just re, derive it in. Right? Like, write another slide here. It's not very hard.
I should have done that anyway, but I was being lazy. Yeah.
Is anything special about the variants that are true?
Well, the situation does, it was like, not particularly now. Uh.
Yeah, so the interesting thing about these 2 is that they're blue entry in blue pegs in variance, and therefore are going to be useful for proofs.
Essentially right, but the others, uh, they're independent.
Yeah, so these slides will go up. I'll put them up tonight, but, uh, they will be without the answers here and then I'll, I'll update them some time tomorrow.
Good question any other questions.
Take me to to.
Yeah, so the the basic plan here is that we're at some point going to start fussing against the variance. But the real question is when to treat the variances Oracles and when to treat them as well, like, when a violation of that means they shouldn't be in variance anymore. Right?
The traditional way that we do this for something like icon, is that we use a human written test suite to decide which invariant a candidate in variance are true true in variance. Right? And then we use a father and any invariant that is violated by the father is assumed to be a buck.
As opposed to a false invariant right now of course, this is a balance. Right? And this is what I was saying this is the point I was making about.
Uh, like, I think it was on the.
This this point, right? The, the balance here is very difficult. It requires skill to decide when you have a test suite. That's good enough that you can start running a, with a father.
All right, if you have a 2 week of a test suite here, then you will start to the buzzer will start reporting bugs that are actually in variants that were false and should have been treated as false dot com.
And, of course, you could add those to the test suite that you run icon with and then you could generate a new set of variance. And that would be fine. But.
Uh, you know, it's, uh, that that that is an unpleasant process in practice.
So, if you already have a good test suite, Dotcom is very useful. If you don't have 1 yet it's quite poor.
So, it's unfortunately a bit of a win more testing technique.
Other questions that's a good 1. all right.
In that case, I'm going to give a few notes about homework 5 and then we'll spend the rest of the time, like, pausing and I will hang around until 9 to help out with any homework 5 set of problems. So, uh, I'm more 5 has 2 parts. The 1st part is to run the icon on a data structure of your choice right? We strongly suggest that you select a data structure that you personally.
Right.
Perhaps for a previous class, you will, you will enjoy this more, and you will have a better time and it will be easier for you if you select the data structure that, you know, you did the implementation and therefore you understand it very well. Right. And then you have to design some Metamorphic relations for a real software system of your choice also. Right? The reason for the, your choice.
Here overall, is that, like, these things require you to have deep insight into how these programs should work right? And so you should make choices intentionally so that you already understand the programs you're analyzing.
All right, and so therefore you're going to be able to make good decisions with respect to the stuff we discussed today, right?
All of the both of these techniques require domain knowledge. So choose programs for which you have domain knowledge. Also I want to note if you get stuck because you're having a difficulty with some system that you've picked like, it's hard to run the icon in the system. Because it keeps crashing or something like this, pick a different system. Please.
Don't come to the course staff and say icon is crashing, because it will well, we can try to help you right? But it will take a long time and it will be unpleasant for everyone. You're better off just choosing a different system. Right? And that's the case too if you're really struggling with the Metamorphic relations for, uh.
Like, like, you download some system and you're trying to, like, instrumented to do Metamorphic testing and it's really hard because the built system is insane. And, like, it's crazy. Just pick a different 1.
All right, that's also part of the reason that there are 2 variants of, uh, part, uh, the Metamorphic relation part, right? There's a version for online systems like Google search, and there's a version for you downloaded the program. The Google search version are the online system version, requires you to come up with more and complicate more complicated Metamorphic relations because you don't have to actually run the program.
Right but for the, the other 1, like, you have to do a little bit more work, but you can come up with fewer relations. So, 1 of them requires more math. The other 1 requires more programming.
Okay, um, any questions about this about the homework.
All right in that case, I am done, like, sharing for today. Um, as I said, pausing and I will hang around and we're happy to give.