Is 6 o'clock so we will go ahead and get started. Um.
As usual, we'll start with the reading quiz here is today's reading quiz.
2 minutes on the clock pencils down.
All right question 1 true? Yes. True is exactly correct. You said that.
All right question too as well. Okay so question 1, this is a direct quote from the, uh.
Uh, from the paper, um, but since we're definitely true.
This is 1 of the important things that Ches enables it allows you to single step. The debugger.
I know what single stuff in debugger means is anyone.
Yes.
Yeah, so if you're thinking about something like a gdb, right? It has a command to skip forward 1 step or 1 1.
Instruction right that's what that's what the single stepping needs.
All right question too. True. False. The authors describe an infinite loop that just detected in a research operating system. They talked about this singularity system, they applied it to.
Did they find an infinite loop? Is it true?
unfortunately it is false so they did find a loop that wasn't doing anything but it was not infinite right so in particular this question is asking you about what you understood what this thing is
A spin loop is a loop that spins it's like doing something, or it's just going around and around, but it does eventually act. Right? So, in particular.
They had a thread that was looping infinitely, but then was killed right? So they were able to detect that. So it was not actually an infinite loop. It was in fact, the operating system did eventually deal with it. It was just a performance for a bug, right?
So, it was not, in fact, infinite this 1 was intended to be a little bit tricky to make sure that to see if you'd understood the, the actual thing that was happening that bug right?
Any questions about that all right so then with that, then let's dive into today's topic, which is dynamic analysis, other than testing. Right? And so, um, what we're gonna do is we're going to start by talking about, uh, like, sort of a high level idea of what dynamic analysis is. Right?
And then we'll get into some of the nitty gritty details of how 1 does instrumentation, which I think I promised you in a previous lecture. Then we'll talk about some of the sort of common property properties, that dynamic analysis share. And then I'll go through a series of real example analysis at the end.
my hope is that this will all be done by about eight o'clock so you'll have some time to ask us any questions that you have about homework six which you should hopefully have made some progress on although when i last checked the auto greater which was at about two zero pm today no one had successfully submitted something that got an zero score
So, I'm a little bit worried about how you all are doing given that. So please do use the time at the end to talk to us about how it's going.
All right, so then let's dive into dynamic analysis. So, a generic analysis is.
Uh, any analysis that runs an instrument program in a controlled manner to collect information, which you can then later analyze to learn about a property of interest, the important thing here and the sort of key bit here, is that a dynamic analysis runs the program right? This is in contrast to a static analysis, which is going to be the focus of the rest of the class.
Break after the midterm, which does not run the program. So, like, a static analysis involves looking at the program and trying to figure out what it does a dynamic analysis involves running the program and actually observing what it does. Okay.
So, we've discussed a number of a number of dynamic analysis already this semester. Can anyone think of 1.
What things have we discussed this semester that that meets the definition.
There are a large number most of them yes, that's a good answer because it's correct right?
Yeah, almost everything we've discussed in particular, uh, testing is the best, uh, best answer here, because it's the thing we talked about the most right? I think it's all into the mimic analysis.
Running any test, uh, involves executing a program in a controlled environment and checking whether it did the right thing.
Doing things, like, completely covered for, uh, detecting the variance like, uh, the tool that we discussed are also examples and almost everything else that we've discussed is a dynamic analysis. So far. Right? So put another way. Today's topic is, uh, kind of zooming out a little bit. And talking about dynamic analysis more broadly than testing, right?
Thing is like 1 example of a dynamic analysis. And so for this semester we've been focused in on testing itself.
And sort of the tools and techniques that we have to support it.
Right without sort of thinking about it in a broader context. And so the goal today is to sort of zoom out a little bit and think about other dynamic analyses, besides testing. And also what brings dynamic analysis together what are the commonalities between dynamic analysis? What kind of properties do they all.
Okay, so the key questions for today, uh, basically what.
I just said, what are the common features of dynamic analysis and what else can we do with the dynamic analysis besides just straightforward testing or things that are directly in support of testing? Can we do other sorts of things find other sorts of bucks.
All right, so Here's an example. Um, I have heard of the 25 bug.
I may have mentioned it at the beginning of the course. Yeah. So the 3rd, 25 was this radiation machine for treating cancer basically, it shoots a bunch of radiation at the cancer and kills it off. Right? So it's supposed to fire a very small beam of radiation. That's targeted directly the part of a person that has cancer in it. Right and the idea is that kills the cancer.
There were at least 6 serious accidents between 19 8,587 in which patients were given massive overdoses of radiation by this particular machine. Obviously that's very bad. And in particular these were caused by concurrent programming errors, let's just say race conditions. Um, and so this, this was really bad. There were a few people who died.
And a number of serious injuries resulting from this, right? And the root cause was risk conditions.
So, what is a race condition? Does anyone know what a race condition is?
Yeah, so it's when there are 2 threads that are attempting to access the same resource in some kind and the interleaving of the, the threads execution order actually does matter. Right?
So, in general risk condition is that any behavior of the system of any system where the output is dependent on the sequence, or timing of other uncontrollable event? So, in general, for software, this is sorry and more specifically for software, as opposed to more generally, which is the high level definition here.
All right, um, this is going to usually happen with threads when there's not a mutual exclusion, like locking. Right um, and 1 of them does, in fact, write to the state right? So usually, the way this is gonna occur is that, uh, you have, uh, at least 1, right? Uh, from 1 thread and, uh, at least 1 other thread that's.
To access the variable at the same time. So, for example, um, you could have a, uh, quote, unquote, uh, write, read read, right?
A race condition where there's 1 thread that's writing and 1 bread that's reading, right?
Can imagine that a thread 1 uh, they're trying to execute the right goes. 1st, then thread 2 will read the version it's written right? But if Fred 1 goes 2nd, then.
The version before the right occurs will be read by Fred too. Okay.
Similarly, if you have a double, right? Right both rights are trying to write. Then I.
Which threat goes 1st, we'll decide, uh, which version of, like, which threads.
Version of the the value is actually the 1 that's persisted. If red 1 goes 1st then thread. two's value is persistent and vice versa right?
These kind of bugs are generally very inconvenient. All right there are very difficult to track down and as we saw, they can have very serious consequences if they happen in the wrong place. Right?
Right. So how would we detect it? A race condition? Right? How would we test for 1?
I don't know how many ideas about how you might write a test to detect your resignation.
Yeah.
I'm sorry. Okay. Apparently my blood sugar is low. It seems to always happen around this time because I always eat dinner in an hour before. So maybe I should change that.
All right, I'm going to eat candy again as we continue this. Sorry can you repeat what you said? Well, I was distracted by my medical computer.
Alfred, uh, comparing threads access to C squared.
Yeah, you could right and I think that's basically boils down to the suggestion here. Right?
But you need to keep track of what's what's happening with the scheduler, right? What the schedule is.
Uh, you want to actually track for.
As such a bug, when you need to have the controller is the schedule.
Ideally, well, I already raised it, but you would try both schedules book right? Reads and both right? So where the right happens version where the read happens 1st.
In other words, that's for our buck. Like, that's what you need to do.
You need to check all the internal leavings of various access to.
Share variables, which is basically what chess does, right?
So, the paper were that were red for today the tool it's proposing, Charles.
Which we'll talk about in more detail towards the end of the lecture.
It's trying to solve this problem. Okay. Did I give you a candy?
Right. Oh, okay. That was excellent. Bowling by me. Um, sorry about that.
Okay, so this is sort of the key idea here is that what we want to do to detect a race condition is we're going to try to control how the schedule it works. We're going to control the environment and this is the sort of core idea in dynamic analysis right? Required and dynamic analysis. What we're going to try to be doing is run the.
And controlled conditions somehow, right as well allow us to answer a number of difficult questions. Not just. Is there a race condition.
But all sorts of questions about the program, we can ask questions like.
Uh, you know, does the program run fast enough? How much memory is the program using? Uh, is this predicated in variance with this program?
What does that? Correct? Icon? Right? Does the test we cover all the program that's coverage right?
Advertisers input control the variable, et cetera, all these sorts of things we could just run the program in a controlled environment and check a 3rd crew. Right?
This is the core idea of dynamic analysis, right?
I'm gonna stop here for a minute and actually finish eating this candy. Right? If you have any questions, please ask them.
Sorry, there is input what causes, but.
Good question. So the question is, uh, what is what is an adversary's input, right?
This is this sort of question that you might hear in a security class, or in a security context right?
What's the budget for this? There's no, from an adversary it's typically used in what's called a threat model, which is if you're not thinking about the security of a system, he's a threat model to decide whether you believe the system is safe.
Within the threat model, usually define a set of adversaries.
Adversaries for the adversaries are a potential attackers against the system that is to say, potentially malicious.
Agents within the system so, for example.
An adversary might be a user of your website, who is going to provide you with some input that you shouldn't trust. Right? So, for example, it might be like a disgruntled, uh, imagine, like grade scope for example, where, where we run the auto greater right? An adversary in that case might be like a student who's trying to.
Take over the auto rare, right? By intentionally submitting a copy of the assignment that is evil. Right? Instead of trying to actually do the assignment, the adversary student might instead tried to say, print out the source code of the interpreter.
I have tried this before you don't have to admit it. Some people have, you shouldn't do that.
And, in fact, a great scope in theory should protect my auto grid or against this in practice. It's believe that great scope is not particularly secure, but.
You know, uh, sad, that's the 1 that we were told to use in by and it is also by far the easiest for everybody.
Does that answer the question? Yeah. Okay.
This is a notion that would come out of, like, a out of the security literature.
Any other questions here, these difficult questions in general that we might try to answer with the dynamic analysis makes sense.
Alright, so let's let's see, uh, let's see an analogy of this right? I'm I'm familiar with a cardiac stress test. I've run on a treadmill or something like attached to me.
I actually have a few of you have okay, so suppose that you need like you, what there's some question about whether your heart is doing well, right? Um, we want to find out some information about it.
We don't want to actually like, uh, just.
We're worried that your heart might fail in some sort of real world scenario, but we can't like, have a doctor follow you around all the time. That's, uh, very expensive. Right? So instead what we might do.
Okay, so stepping back, just looking at, you just having you step into the doctor's office might not be sufficient. If the doctor just looks at you under, like, sort of normal exam room conditions. That might be not that helpful because your heart isn't really in danger there. And there's no sort of obvious marker for example, it's suppose that there's.
Obvious marker. Okay.
Um, instead what a doctor would do is, they would have you, like, run on a treadmill on, like, under our control condition, right? What they would do is they use, like, a blood pressure cop and, uh, sort of electrodes, like, in this very silly diagram, right? To keep track of how you do when you're asked to, like, run on a treadmill, right?
And this is the same kind of thing that we're doing in dynamic analysis, but in this analogy, in a dynamic analysis, what we do is we run the program right? Rather than running on the treadmill, we run the program in a systemic matter. Right?
That is to say on some control inputs for, uh, you know, some regulatory and input, but.
In an environment that we control right? And we monitor the internal state of the program at run time. Right? So what we're doing is we're instrumenting the program like.
Just like, uh, in the analogy to the cardiac stress test, we've attached some electrodes or attach to blood pressure cuff. We're attacked.
We've W, we somehow modify the programs that we can capture about, uh, data about how it's doing. Right?
And then we analyze the results. Right? And so, in the analogy here, the software is the person.
So, the program is being tested is like, the person, the controlled input environment is like, the treadmill. This is not like asking, like, we're not taking the person out into the woods and seeing how they do. If we ask them to run a.
That's not a controlled environment. Anything can happen in 5. K. a bear might come up and attack. We don't want that. Obviously that's unlikely to happen but we don't want any questions. This is a tightly controlled environment. We've removed the calendars, right?
The instrumentation in this analogy would be the blood pressure, coffee, electrodes, et cetera, right?
And the analysis is actually quite important too right? We have an EKG monitor or something over here. And there's you can see there's a technician over here. Who's actually figuring out like, is this going? Well, right someone's taking care of this.
All of these parts are the same kinds of parts that we're going to need to use to build a dynamic analysis. So all of our dynamic analyses are going to have all of these ingredients. So, does this analogy make sense to everybody?
Some water on my notes. All right. Um, okay. All right so, then moving on. All right.
Instrumentation is 1 of the core pieces of how we do dynamic analysis. Right? We've mentioned instrumentation before in this class. Right? I remember when we mentioned instrumentation in this class.
Yeah, sorry you do do you remember when.
Yes, it was closer to the beginning. It was 1 of our 1st, 3 lectures.
Not quite, um, but it was the lecture before that the main place where I mentioned this before was in the, uh, the context of coverage, right?
In that case, I told you don't really worry about instrumentation, right? For coverage. It's a solve problem. I think those were my actual words, right in the slides. It probably says something exactly like that.
Today, we're not going to treat it as a a sole problem today. 1 of the things we're going to talk about is.
How does instrumentation actually happen? I'm sorry about the need to just eat candy in the middle of lecture. I know. It's very annoying for you. All as listeners.
Alright, I apologize. Okay. Um, so, but that's better than me just passing out up here by the way. If that does happen, please actually call 9:1:1. that would be that.
Okay, so insurance program is going to be any process that we're going to use to modify, rewrite the program uh, either the source code to the binary to change this behavior. And usually, in the context of dynamic analysis, we're going to be doing this to record some additional information. That is usually our goal. So, if you think back to the coverage.
Example, what we're doing in coverage usually, is we're recording, did this statement to get executed, for example, by printing out that that statement was reached right?
However, we can do much more complex things, right?
So this is how coverage instrumentation works this is exactly the example that I just mentioned, right by printing we've reached line X on each line X, but we can do something more exciting. Right? So we can do it in a compile time. This is how, like, Jacob works, right? So why did we need to recompile with? Right?
It's so that we had coverage instrumentation, built into the binary.
All right, but you could also do it by running in a specialized VM. Right? I've heard of.
Nobody okay. Very useful tool. We'll come back to it and discuss it. It's a little bit more.
And at least a little bit more detail later in lecture, but at a very high level outline is a tool for monitoring the memory consumption of a process.
Right the core reason that you'd want to use now grind when analyzing something like a C program is to try to detect whether you're leaking memory.
That is to say what you've forgotten to call free on some point or did those words make sense?
I know what a memory leak is, I have written a C program.
Hopefully most of you, right? If you have not, you should make sure that you take a course that forces you to do. So it is a very useful.
Learning experience, let's say, for understanding, like, how programs actually like, how their computer actually works right?
The process of actually learning how to allocate and free memory, I think is something that all students should definitely do. All right.
okay the key idea there though is that you don't want to actually lose any of the memory that you allocate because it will be wasted right and so you as the programmer would be responsible for actually keeping track of all of that right it's way harder and you should never write see programs in like real life unless you have to right for performance reasons because you're likely to make a mistake
Right and, uh, you know, allocating and free memory. But, um, Brian, basically, it's like a, a later on your operating system that's going to tell you, like, as it's executing this program how much memory is being used and so you can keep an eye on it. You run it for a long time. The memory just goes up and up and up and never comes back down. That could be bad.
Similarly, you could run your Java program and a special right uh, that keeps track of, like, uh, for example, you know, how many like.
How much again how much memory it's allocating right? If you want to figure out why your Java program's performance is so bad and why it's constantly calling the garbage collector you might use a specialized gvm for that. I know what a garbage collector is.
Relatively a few of you. Um, so, let me briefly explain garbage collector. Um.
Is solving basically the same problem as the memory like.
Um, uh, the, the memory example, I just, uh, described for right and see, you have to allocate your memory by hand and then freight yourself right in our language like Java, the reason that you don't have to free your memory. Uh, yourself, is that, uh, uh, uh.
A process called the garbage collector is running, right? And the garbage collector in general is a dynamic analysis. We're not going to cover it in much detail today, but it is an example of a dynamic analysis. And it's the sort of thing that, uh.
Actually, as an example, this is the sort of thing that I might introduce on an exam is give you a bunch of information about it dynamic analysis. You haven't seen.
All right, and ask you to relate it to the, the sort of core ideas that we talked about today, right? That's the sort of test question I might like to write for this lecture.
And a garbage cluster would be 1 of the sorts of things I would consider there, because it's not something we mentioned in this lecture. Okay. So.
With that assigned, what is the garbage collector? Right? Basically, what it's doing is it's keeping track of whether there are still pointers to a piece of a location of memory that are alive in the program. Right? And when the last, the last of those pointers goes out of scope, the goal of the garbage collectors to free that memory.
All right, if you're, for example, running a Java program, that is using most of the memory that's available in the system, it's going to call the garbage collector a lot.
Because it's like, the garbage collector basically gets called anytime that there isn't enough memory available.
Right and the garbage collector is basically keeping a list of like, here's all the things I can.
And then it actually executes framework. Right? And so updating that list can be quite expensive if you have a lot of objects. So, a common performance hiccups that you'll see in languages that have garbage collection, like Java or go or some others right? Uh, is that, uh, you'll see big spikes in latency as the language stops, executing your program and instead runs the garbage collector.
And if it's running the garbage collector, like every 2nd or so, like, you will see, uh.
Like, most of your execution time being taken up by garbage collection, right? And really the problem is that you need to provide more memory. Like, you need to, like, download more.
That's not actually possible. You need to use a bigger machine, right?
But, uh, so the, the garbage collectors job basically is supposed to do that.
Did that explain that? Well enough? I feel more confused than I was before.
I mentioned this, I'm afraid that I'm like.
Rambling a little bit. He's the low blood sugar so I'm going to eat the rest of this other piece of candy before I go on. If anyone has any questions about garbage collectors. Please direct them to causee. Sorry to put you on the spot.
Okay, sorry about that. So, um, going back to the topic of instrumentation.
Let's move forward here in the slides, so a common student pitfall here.
It's confusing compile time and run time and I want to just dispel that now and make sure that we're all on the same page relative to this, because it's, it's easy to make this mistake. Right?
So, at compile time, we instrument the program by preparing it to collect information. Right? That is to say, for example, uh, we look at the example here on coverage, right?
Adding these Sprint statements is a compiled time operation, right? We do it. Like, suppose we're adding.
Presenting it to the program, right? We're changing the program before it starts executing. Right? And it's just saying we do a static.
Just 1 time, similarly, if we were adding the Python, right? We're going to do that to the source, though. Mm. Hmm.
However, the print statement actually executes at 1 time, right? It actually records the information at run time.
And so you inserted the program before, running it, and then the instrumentation actually runs at runtime just like the rest of the program. Right? And so, um, keeping this distinction.
In mind, it is important as we go forward here. Right? And it's an easy thing to get confused with that. So I just want to make sure that we're all clear on.
On this and that what I, what I mean by instrumentation, like, instrumentation unfortunately is 1 of those words that's a little bit ambiguous and can kind of mean both. Right? So, usually, when I say, instrumenting the program, I'm referring to the compile time activity of adding instrumentation but when I say instrumentation usually, I mean, the actual print statements that are going to be executed at 1 time.
I have a question anyone to have a question about that. Yes.
Fair enough. Yeah. So, um, the key thing to keep in mind here is that, um, the process of instrumenting a program to record information and dynamic analysis has to happen before we start the program.
All right, and so there is a, uh, like a step before execution that you have to do in dynamic analysis, or you won't be able to learn anything.
Right but is to say, you need to modify the program in some way. So that when it actually does start running, you'll be able to tell what it's doing.
Okay, that's a compile time process. That's called instrumenting the program.
Okay, but the thing you're actually putting in the program is called instrumentation and that actually executes with the program as it's running. That's what makes the analysis dynamic.
So, again, I think of the example of coverage is useful, right? When you compute statement coverage.
How do you do that? Well, 1st, what you have to do, like, say, live from the homework, right? 1st, what you have to do is compile it in a special way. You have to like, call the compiler a set of special flags. Right? Why do you have to do that? Well, it's to insert all those print statements.
Right. Then you have to actually run the program and in a special way right that actually collects all the information you want and then you use it like g cups to actually look at the results. All right.
And so, uh, the the point I want to make here is, like, that you have to keep in mind there's a distinction here. They're not this, like, yes, it's a dynamic analysis is a, uh, an analysis that involves running the program, but it's not exclusively running the program.
It's allowed to do other things too and and in fact.
Useful dynamic analysis, almost always must do some work at compile time or before the program starts executing in order to be able to collect the information they need.
Does that clear it up any other questions before we move on?
All right. Okay so let's let's go through an example here. All right. Um.
So, consider I.
Half coverage, right? Computing path coverage. So this is something we talked about briefly when we cover coverage, but I want to go through actually how we would do it. Right? So you wanted to determine how many times each basic path, and a method is executed on a given test input.
How would you change the program to record this information?
All right, so, uh, let's do an in class exercise here, so I can get more candy. Um.
For this example, right I turn to your neighbor and write down the instrumentation that you would add in order to collect this information.
Okay, so your goal is assume that I want to say assume that this is like, in a loop somewhere right? It's, it's going to probably be executed many times with different values of a. B. C and D.
Right, and the question is how many times has you've packed it yet?
Hmm.
So.
Hello.
Yes.
Is.
Uh.
Hello.
So.
I think I know what instrumentation needs to be added.
Give it another 30 seconds or so and then we'll go through it together.
All right does anyone have a suggestion for where we will need to add instrumentation?
Sure, yeah, that sounds reasonable. So, uh, say, then at the beginning of the blocks, right?
Okay uh, so in these locations, right?
What instrumentation should be at counter okay. Do you want 1 counter or more than 1?
Sure, so let's, uh, let's make, em, Ray, right we'll say that the count of, uh.
This is counter 0, this is top of 1. this is 2 and that's kind of free. Right? So, Council.
Is a more valued correct, right? Like this right? And then this would be account.
0+plus right and so.
Okay, this whole thing for everything, right? Does anyone I.
Did anyone have an answer that was not this? There are other reasonable things to do it.
But, okay all right great. So, this is what's on the next slide so, this is great, right?
And let's look into it in detail, so I'm just I'm going to raise that so that I can pull up the next slide. Right? The next line is this.
All right, um, this is basically what we had, right? Except that I labeled the agents.
All right, so you'll see here that I've labeled, uh.
P. Q. R. S. T. A. you right for these different places in the program right? And then I'm using this a, like.
Half name our indexing team here and different.
Hi, everyone agree that this is the same solution.
All right, so how, how will this actually go in practice? All right so suppose that we have, uh.
Peter Q. is 2 P2 R's for and right so we've, we've run this, right? And these are the results we got.
How many times does P to Q2 take a moment and turn back to, uh, whoever you were talking to about the previous 1, right? And I think this throat.
I think I know the answer. All right. What do you think.
0, why.
Steve.
Okay, looking at the program again I I think we do know right because.
Uh, quite honestly after this call, right we're going to go after this.
All right, so it's going to be 1 of the ads for sure.
Right.
So, I mean, this is straight line code, right? So you yeah, so so you're, you're changing your answer to 4.
Uh, so.
I I don't want to put you on, like, too much on the spot here because, uh, actually, this is under specified. There are multiple correct answers. Right? The problem with this approach. This, 900 implementation approach would be put up. Right? Which you all came up with it. When I asked you to do it, right? And this is immediately the thing that everyone would think of. Right? There's nothing wrong with.
We're thinking of this, the point of doing this exercise is really to show that this is a little bit trickier than it's right so we'll look at some examples right?
I'm putting it to you that there are multiple possible ways to get this number, but these numbers.
With different values for this in particular, consider this schedule.
Right, but I think is the 1 you are thinking of 2 times, right? In this case we've got, uh.
1st, we do, um, uh, and D right and, uh, right, right. And then again, and then we switch the other way, right? And then this last 1 is different. Yes.
But this does produce exactly the schedule take a moment and convince yourself of it.
Okay.
We all agree. Nod your head when you're, you're convinced right?
Right. So then instead consider this other schedule.
That only positive to go around 1 or it only causes, uh, to to be taken 1.
Do you think we could achieve? I, I'm actually not sure. I haven't actually thought about it too hard. I think it's probably possible to get other numbers here. Right? So the 4 is probably also. Correct? There's probably a schedule that leads to 4. right? The point here is that, um, there are many possible ways to achieve, like, different values of these, uh, these individual, uh, edges.
Right so this sort of this style of it is not sufficient to actually determine the thing that we care about. Right and the point I'm trying to make here at the higher level point.
Is that this is 1 of the hard things about dynamic analysis, right? In the statement coverage case, it's not it's easy to negatively print that you're at a statement, and then just read those logs and you get the right answer. Right. The math involved is not too hard.
But when attempting to reason about anything, even slightly more complex, like, Path coverage, you often end up needing to do some reasoning, like, often, complicated reasoning in order to make sure that the information you're collecting actually allows you to reconstruct the information. You care about the information. We care about here is.
How many times does each of these sequences? Each of these sequences appear, right? Cause we care about paths, which are of arbitrary life right?
But the information we pointed were was not enough to drive it.
Right. It's ambiguous. Yes, we'll send this thing with the video.
Yeah, so these are these are the values of the variables on a particular iteration right? So we're imagining that this is like, inside a loop and we're running it in this case. Uh, what is this? 1:2:3 4 or 5:6:times right? And on each iteration, the loop, these are the values. The variables to the ones in blue are this path.
Okay, so, um, in this case, we've got, uh, 2, in this case we have 1, right? But then the numbers here, I'll make these, this, this, this stuff work out.
Okay, okay. Does this make sense to everybody?
All right, so what might be doing step? I don't have an idea.
Can we just not compute path coverage?
But another way here, what I'm really asking is, what went wrong right? Why is it the case that just knowing these numbers what we've put in.
Our implementation here with this, this count is not sufficient. Why is it not sufficient?
What could we have done differently?
The next Tuesday, uh, take a look at that.
Sorry, so exclusively, so.
Ones that are taken so, if we just take a prospect is then we.
Also, consider this where C. D. screen.
So, what you're saying is that we need to, like, somehow keep track of which.
Which path we took the last time through. Mm. Hmm.
So, I think we could do it for this, but I'm worried that this will create a state explosion problem in the amount of instrumentation we need to add right? For each if statement, what we'll need is we'll need to have a check, the previous hit statement. So we know which thing to.
To automatically yes and so that means that, like, in this to a statement program, we'll need to merely add 1 new statement next to each of these. Right?
But imagine a program with tenant statements in a row. How many students do we need to need to add in the 10th? 1.
During the 10, right? It would get out of control very quickly. Um, we could do this, I think, but.
We'd be in danger of exponential increase in code size, which.
I mean, that's not a good thing. Dangerous, but it's a good idea. It would solve the problem. I think.
All right, so, um, here is a real example of this problem is from the 90 s but, uh, this is a real paper that was published in the 90. S, right and it's the same problem that they're trying to solve, right? This is another, uh, an example. Uh, similar to the 1 that we just looked at where the edge count, they've collected.
Right and they show that the 2 profiles are different than just to say, like, as you pass through this program, uh, if you just collect the edges, the number of countries ethicacy issues.
You will be confused about which of the.
And in particular in this paper, uh, they this is the paper where they posted, it basically boils down to the correct answer. So I want to be clear that this is not obvious, right? If you control the 90 s for people to figure this out. So the fact that you're, like, you all are struggling to come up with the right answer, like, sitting here in class it's not surprising. So, don't feel bad.
Okay, so this is the bigger from their paper, right?
So, what are they going to do differently?
So, what they're going to do is they're going to create encodings for the different paths right? And the encodings.
Will, uh, tell us which cap we're on right? And that will allow us to, uh, keep track of, uh.
Uh, which has were actually taken with relatively little space. Okay.
So, I think you just shared this figure for a moment and think about it, right? And then we'll talk about it.
The poor thing that they've done differently here right? Cause that they're not keeping track of account for each ad. Right? Instead what they're doing is they're, uh, setting a certain fit. Like, they're setting a variable to, uh.
It's a different values that language it's taken right? I notice this, like, our plus equals 1 that happens, right?
This art plus, what does that do it?
Well, what it's doing is, it is, uh.
Allowing us to differentiate between different paths that were taking in the past.
All right, so, um, think about like, how, uh, uh, goes rather than a right uh, D. E. F.
Uh, is encoded, uh, at 5 here, right? An app is included as 4 and so then whether it's 4 or 5, right? Um.
Or will be like, regardless this always goes, I feel like this tech crunch has gone off the rest.
Let me start up because I think that was confusing.
Once again, they are using the account system, right? So they have an array, right?
And the array is indexed by accounts, right? So.
It's 0 entry is the path, right?
So this is the council it's 1 entry is a.
All right et cetera, right? And that's what this happened in quoting thing means over here.
All right, and the key trick that they've used is that they figured out a way to make it the case that whether or not you take this, uh.
Is included into.
The arrangement of the pads in the.
All right, so, um, regardless of how you got to be.
Alright, there are like, okay, like, basically each place where there's a D, right?
The D, can we followed by either an f or an? Right? And in each case, it's always a D and then.
the sorry and then the yeah right yeah the uh the
And because they've arranged the array like this, regardless of how you got here.
Adding 1, when you have to repeat, will always take you to the, uh, the correct other path.
Does that make sense? Not seeing anyone shape like nod their head? Yes. So that makes me.
So, let's imagine, let's imagine going through this right? Um, let's just choose some, uh.
Well, let's just put some guards on this. Right? And then started.
Choosing some values, right? So as opposed that, this is a greater than be and this is a less than equal for some little a will be right?
And then, what are their choices? This is a statement, right? So we can say that this is secret D and this is, uh, see less than are able to date. Right? And then we have this other 1. right? So we can make this 1 easier than app. And this 1 be less than that. Does everyone see what I'm saying? Here?
Each of these, uh, places where there are 2 outgoing address, and it gets in that.
Yes, and so I'm just putting some guards on this somebody gives me values for a. B. C. D. E. F.
Yes.
one two three four five six okay yes
Now, let's step through what will happen so he's.
right so we're going to go down this path right so we have our so we know we're gonna hit a of the passwords a day and then we're gonna hit c right and now what we have is already
Right. Okay. And then after C there's no choice we always go to D next. Right? Because there's only 1 error app.
All right, and then we have to check E, rather than f is Eva. Good enough.
No, right so then we go to f, right? So the path is a C. D. F.
And look at that are, is 0, why? Because the only part, uh, our assignment that we got was the article 00:1:we missed all of the others.
right effectively what they've done here is they've designed and encoding so that whichever have you take
The last assignments to R will be the 1 that corresponds to that.
All right does that make sense?
So each part the value of power will be different for. Exactly. Right. And so, um, the, the key contribution of this paper was that they, uh.
They showed a way to do this, right? They showed a way to automatically figure out from the, from the graph how to choose the values of the right but it's not accurate that complicated. We're not going to go through the output.
Uh, because there's not really time, right? But I want you all to understand, like.
Well, hey, this is actually relatively hard to come up with these kind of algorithms and B, to understand like, what's the difference between this technique and the naive approach that we can.
I'm going to start erasing on the whiteboard.
Unless anybody has a question for the last time, we had the question of how many times that bot was actually.
Yeah, so we're solving the same question here, right? So, I mean, effectively what we're doing here, when I say we got the info, like, we got an article 0, what that means is that we're going to go and we're going to augment like, we're storing accounts over here for how many times is and this count of our + plus is going to take that from 0 to 1 right? And then if we go.
On the loop again, we will, uh, and then change the values right? Well, once again gets a counter with our plus boss every time with indirect value bar.
Okay.
Just to give you some intuition for, like, why this works. Right? Their core trick basically, is that they're a.
There's 1 assignment like that is that equals right? Along each possible path here right?
And then when you have a branch like this, it's always a.
And it might not always be plus equals 1. right? So, if we had like, a more complicated structure here with, like, more possible branches, there might be somewhere. It's like, plus equals 3 in somewhere. It's plus equals 1 or something like that.
Yeah.
It's very similar to be talking about it. Yes. That's a great connection. Right? This is the same kind of technique. Right? It is. This is fundamentally in the code strategy, but doesn't include a strategy.
Rather than data, but this actually gets at an important point that we're going to come back to later in this lecture, which is that, uh, code is actually also just data.
Alright, and so the same kind of tricks that tends to work on data also tend to work on code.
Okay, so Huffman coding is a way to encode, uh, you know, data and a way that compresses it.
Basically, what we've done here is we've compressed the amount of information here.
All right. Yeah.
Is there any other questions.
Let me grab my eraser.
Yes.
To move on, right? So, um, I also want to point out. I.
This is true. All the best research is just my opinion, but I've heard a lot of other people who are better research than me say, as well. Right? The best research makes sense in hindsight, but it's very hard to come up with.
All right, this is an example of that where this work, like.
Do doing some coding, but once you realize that this is the answer, it it does make sense.
All right, it's a very sensible way to approach the problem, but almost everyone not usually attempts the count of counter each edge + plus strategy that everyone came up with at the start. I want to be clear. Like, I didn't come up with this the 1st, time. I looked at this problem either. Right? I was tricked exactly the same way that you all were the very 1st time. I saw these.
Right. Or solve this problem and that's standard right? Almost everyone is touched by it the 1st time so don't don't feel bad. All right. Um.
The cool thing about this kind of work is that, like, once you, once you, once you understand it, like, it's so obvious that this is the right approach right?
Okay, so, um, do you think we could do better than this?
Like, I'm not asking you to come up with an answer in the sense of.
Uh, come up with a way to do better. I'm asking. Do you think anyone has come up with a way to do better.
And if you are thinking of the analogy to Huffman coding, that's actually a good way to answer this question.
Huffman coding is the thing that we teach in data structures, right?
Do you think that there are more efficient in coating algorithms than Hoffman coding?
For compression yeah not just probably definitely. Right. They're definitely more efficient inputting than I have been coding for, uh, for compression. You can do some much clever things. All right.
They don't all work on every circumstance. Right? Encoding and decoding is easy in the sense of like.
You can learn it in 1 lecture, right? If you go and look at the source code for Jesus, it's much more complicated than just hop encoding. Right?
So, with that in mind, do you think we could do better than this better than this?
Seeing some nods and the answer is yes, right of course, this immediately becomes a research problem. Right? As soon as somebody sees this, they're like, Ah, I'm sure I can do a better encoding strategy. Right? And so there's been lots of lots of follow up work right? So, instead you could do something like this. So, here, what we have is, uh.
We've, we've got a bunch of different variables, right?
Uh, augmented it every time to go over an edge and arrange them in such a way that we can derive the, the answers, uh, to each of the paths right? From, uh, some linear combination of these variables.
All right, we're not going to go through the details here. Right? But this, these kind of techniques can be about 3 X, faster, blah, blah, blah, right. Lots of research about this. But the sort of core contribution here is realizing that.
Uh, there's ambiguity in doing edge counts, and that you can do something better, right?
Okay, this was an extended example, but, uh, like.
I think we're about the yeah, I think we're ready to move on, right? Unless anyone has any more questions.
I hope this gives you an idea of the flavor of the kind of stuff we're talking about today. Right? The point of the sort of overall point of this lecture is to show that statement coverage, for example, like, it's very easy to see how to do the instrumentation and how to compute statement coverage from all those print statements. But for most interesting problems.
A little bit trickier.
There are no more questions, let's think of another example right? So, we asked about evil adversaries before right now. Let me know what evil adversaries are. Right? Uh, let's talk about information for the primary and the key question information flow tracking is 10 data that's controlled by some evil adversary. Like, say, a user accessing our website over the.
Influence sensitive computations. So natively how do you think uh, 1 might, uh.
Check if this is happening.
So, if I asked you to develop a website.
All right, that we knew some, some people would access who are going to enter some data into a form that is not to be trusted right? Because it might secretly try to do like a cross site scripting attack or something.
How would you defend against that? I know what any of those words mean.
Some of you yeah. Can somebody define cross eyed scripting for me? Yep.
Yeah, yeah, yeah. So the idea is that the user is able to input some data into a form and if you accidentally allow the user to, like.
Uh, insert say, like an end quote, or something like this, they might be able to actually modify the, the text of your program right? If your, if your code is done incorrectly.
Javascript makes us for some reason way too easy right?
Okay, so how how, how does 1 defend against such an attack.
At a very high level. How would you approach it? Yeah.
Yeah, sanitize your input, right but how do you know what to sanitize?
Yes, no, that's exactly right. Right. You restrict to think what you can do with it. Right? So, 1:1:way to do this is you, you're like, you could attach a tag to data supplied by user right? You put it into an object that isn't treated the same as every other object, right?
But there are a lot of ways that we could do this, so, you know, we're trying to answer questions, like, kind of user password, ever be displayed clear or Ken network data, or ever enter it. These are just other examples besides cross side scripting. So there are 2 important definitions we need in order to formalize this notion. The 1st is a quote unquote source, right?
Versus a place where sensitive information comes from right? So, for example, input from the network, uh, user passwords, at the time of the day, I think this could all the sources it depends on the problem, right? Which of these things, we actually care which sources we actually care about right?
And then we also need syncs. Syncs are untrusted. Uh, things are places we don't want, uh.
We don't want the data to go. Right? So, um.
They can either be sensitive computations. Like, we're going to execute a sequel command and we need to sanitize it. 1st, right. Or we're going to run some code and we need to make sure that it's not malicious. 1st. Right?
Or they could be like, text display in the clear like, we're going to print this out to the screen and it, it must not be a password.
All right, or we're going to store this to death, and it needs to not be a clear tax password, something like this. All right.
And, uh, for any of these things, what we're going to do is we're going to keep track of what is a source.
All right, we're going to instrument the programs that we know when something's the source.
And then we're going to keep track of what's a sync, and we're just going to check if at any point anything that's supposed to be a source.
Uh, ever passes intuition. All right. Does this plan make sense?
How far do we think this is can you imagine implementing a system that would track sources and see if they hit syncs.
So, for.
Let's see.
Data 1.
Yeah, so we can definitely do that, right? We can pass special data into the, into all the things we think, our sources, and then ever like, check all of our syncs and see if they ever contain that special data. Right that's an extremely effective technique. Very good idea. All right.
Oh, there it is. I mean, the other thing, I think I already mentioned a bit, which is like, we need to, but we can, um.
Do like, add a special standardization routine or spectral versions of the various input channels. All of a sudden separate versions of the sources that instead of producing, just like a string produce a special string that has a like, a.
Like, a marker in it somehow that we can recover right? Like, for example, we could have our.
Our network input from the network, uh, always attach a special header, right? Uh, while we're running in our testing mode, right? And then we, we, uh, we can then check all of our things for that header.
All right does that technique make sense? All right.
Yeah.
Sorry like, uh, Jason I checked them. Yes. Yeah. That's another way that we can get. Yes, it's very similar.
Similar experience does everyone know to check? Some is.
Okay, it seems like pretty much everyone. Does anyone not know what to checks on it?
Seems like now. All right, we'll move on. Um, okay.
And so let's, let's look at an example of this, consider this program.
All right, so this is, uh, fairly, fairly straightforward, uh.
A bit of code, right? So what here do we think would be the sources.
Yeah, 1st, 2 lines. Yes. Why are those 2.
Number 2, that username and password, right? These 2 things are getting something that's probably important.
All right, and then, uh, what what do we think are the things.
Yeah, last line yes, totally correct. Why the last line.
Yeah, I mean, um, and it's also kind of spaces that are evil dotcom.
Um, yeah, so I, I got to the point here, this, like, this is a fairly straightforward to do as a human right? And you could imagine writing an analysis, but also does this.
All right, so any questions about this, like high level example.
I'm going to go much faster through these last couple because we spent so long on the previous 1. the 1st 1. okay. So Here's another another example of something we could do with dynamic analysis.
We could try to determine how much time a program is spending in each of its components like each method, for example.
Why might we want to do that? Yes, yes. We want to. We might want to do performance bottleneck. We want to know where the program is.
Right if we are concerned that our program is taking too long, we might want to know that. Like, we're actually spending 90% of our time and some computation. We don't really care about and we don't want to go and optimize that.
All right, so conceptually, what we would do here is, 1st, we'd record the time at the entry and exit of each method, right? Just put those in a side file and then we would subtract 1 from the other. All right, and then we would update some global table right? Does this make sense as a high level plan for how we would figure out how much time we're spending in each method?
Can anyone think of anything that might possibly go wrong with this?
We've talked about at least 2 things today that could cause this to be wrong.
Yeah, race conditions for sure. So this is going to be asking asking to create a 1 to erase conditions, right? If we're not careful right? We have multiple threads running with 1, global table. That could be 1 problem, but also, just in general here.
Scheduling is going to matter a lot, right? If we are preempted in the middle of executing a method, are we going to accurately estimate how much time we spent on that method? No, we're going to wildly inaccurate. We can make it a wildly inaccurate estimate. So, we need like, a per thread o'clock or not? No no per process clock. So, like, this gets way more.
Complicated I want to be clear. Another thing that could happen here, is that the garbage collector could run. Right? That's the other thing that we definitely talked about today. That could go wrong here, but there are so many others and this is actually extraordinarily complicated topic. Right? In practice building a profile, or is like.
Quite far, um, and so it's way complicated enough to merit probably at least 1 lecture possibly more than 1. so we're not going to go into the details here. Right?
But, um, yeah, we don't, we do not have time to cover, uh, any of this in detail, but again feel free to ask me about it at some point in the future. But my goal here is just to give you a sense for the kinds of things that we can do, uh, with the, like, these kind of dynamic analysis right?
So then, the real question is, what do they have in common? Right? So, we've talked about edge coverage, Path, coverage, information, flow, tracking execution, time profiling, and, you know, a bunch of others, like dynamic variant detection or computing statement coverage.
What do these things have in common.
Yes, yeah, you always have to instrument the code. 1st right? What else do they have in common.
And we based on the title of this lecture on the front page dynamic analysis there right? We know that they all involve running the program.
All right are there any other things that we, we've noticed that they have in common.
Certainly, they all involve, like, doing some kind of analysis afterwards. Right? Um, they also often have this sort of like, possible way that things can go wrong, like, related to things like data races or preemption, or have something like, we have to be thinking part about how the operating system is working. Right?
Um, yeah, so there's a lot of things that they haven't caught. Okay. And so, um, my goal then yeah, so they all involve recording a subset of all the information about the program's execution as well. Right? So, all of this, uh.
All these things allow us to, like, sort of.
Give a pattern for how dynamic analyses are built before I show that that pattern I want to talk about this, this notion of recording a subset. Right? And why am I mentioning it? A subset here?
None of these things are recording every.
About the program, right? They don't allow us to recover the entire state of the program.
And I want to just briefly answer the question of why not just do that.
Why not just save what the program? Everything the program is doing all the time.
The answer basically is like.
It cost right? Like, in theory you could do that if you had a very big computer, but in practice you cannot. Right so suppose that we have a 4 gigahertz computer, this is a sort of standard processor speed these days. Right? Suppose our program, we have a program that runs for 1 minute right? And suppose further.
Suppose that we, we record just 1 bite for every instruction. That's wrong.
All right, like for example, like which instruction we're running, right? How much data are we recording?
Paper and, like, do the math.
You know, raise your hand when you think, you know, I have a close estimate at least.
All right, what do we think about 40? Yeah. A lot. Right? So this is this is just based for properties about 246. I think that's right.
Yeah, 140 gigs for a minute, right? Um.
That's all not right. It's 4,000. megabytes. Every 2nd. Right? Um.
If we just consider, like, how fast are modern SSDs we're almost hitting the limit of them just with this right? Does anyone know how fast monitor instances are.
No reason for you to the, the fastest.
Oh, we know that's why right? Yeah. So on a hard drive, we couldn't do this.
Right like a traditional hardest can't write this fast. Right? So we need to use.
Um, modern necessity. So, uh, this said as of January 22, but I also checked again. That's what I'm these slides for me, but I checked this morning and the answer was still about 7,000. it hasn't significantly changed in the last couple of years. So, about 7,000 megabytes per 2nd, is the best on the market.
Uh, there are actually a number of them now that can hit this, uh, Benchmark, but, like, we're, we're basically, like, pushing up against the limit just with 1 bite for instruction right?
So, we record everything the program is doing, like, physically impossible for us to record all that information, right? We just won't be able to. Right? And this is going to fill up your hard drive stupid fast. Right? I mean, 240 gigabytes is a lot.
All right, like this is basically all copying data right? And then doesn't leave it before.
Uh, right in 4,000 megabytes per. 2nd here it doesn't leave a whole lot of resources for the program. We're actually executing. Does it.
All right, this is an extra would be extraordinarily expensive.
So, the point here is that, like.
It's critically important in designing a dynamic analysis to decide what we are going to actually record and, like, think carefully about that if we record everything. Well, we can't.
Okay, so we cannot record it all. So that means that we have to do some kind of implementation, right?
And so, with with massive compression, if we're very good, we can maybe record about half a megabyte for mega instruction there, which is the.
Uh, like a, that's a made up unit, but, like, this is about the feasible limit.
That we found in practice right? But of course, there's also instrumentation overhead involved here. Right? So the more instrumentation overhead we have, the more we're going to affect the running of the program itself, right?
And so, the relevant information here depends on the problem that we're trying to solve. Right? And so we, when we are designing a dynamic analysis, or thinking about how to use it and analysis how to, like, how to deployed analysis.
We need to make sure that we've chosen to record the right information.
And so 1 of the core questions, uh, designing, this kind of tool is always like, what are you going to record.
If you need to compute information, you're trying to do information flow tracking, versus you're trying to compute path coverage. You need very different kinds of information. Right? The 2 basically don't share information at all. And so you need to do different sorts of instrumentation ahead of time, in order to actually record that information, right? There's no sort of like.
Bullet for instrumentation, it's a per problem kind of thing that you have to solve you to decide.
What do I need to record in order to figure out the answer to the question about the program that I want to know the answer to.
All right does that does that make sense to everybody? Good.
So basically, what this means is that we have to focus in our particular problem, right? We must be focused. Right? We must be like, we have to think about it. I think that's the core thing. I want to get across here so that you have to think about what you're doing when you when you're dealing with this kind of analysis, it is not just run the program and hope for the best. Right?
So, yeah, I, I always love this, uh.
This this quote of most problems in computer science can be solved by adding either a layer of abstraction or a.
Basically, what we're doing here is we need to we need to figure out an abstract property that we care about. Right this is a add a layer of abstraction kind of situation.
Okay, so this puts us in a position then to talk about all the components that we actually would need in order to build in a dynamic analysis. So, um.
What I'm going to do here is sort of explain all of these parts and then we'll go through some examples.
So, 1st, we need the property of interest we need to decide what we're trying to learn about and why right? And so we have to we have to make a decision here. We can't learn about every property.
Then we need to define what information related to the property of interest we care about. Right? How are we going to actually learn about it? Okay.
What is the information we want to collect? Right? Mm. Hmm.
Then we need a mechanism for collecting that information. Right? Which is to say, how does our instrumentation actually work right?
Then we need actual input data, right? We need to decide how we're what input we're going to use to drive the program. Right? And then finally we need a mechanism for learning about the property of interest from the information that actually gets collected. Right? And so.
Like, thinking back to the question of past coverage, right? This was where, uh, we went wrong in our initial attempt at path coverage right? We recorded some information right? Um, we had a mechanism for collecting that information.
But we weren't able to get from that information to the actual property of interest, which was half coverage because what we recorded was edge counts. Right? And edge accounts do not unambiguously, uh, differentiate, uh, paths.
So, we need to then we need to be able to answer to this last question as well.
All right, so does this high level high level structure makes sense to everybody.
For how dynamic analysis works. Okay. So then let's look at an example that we know. Well.
Branch coverage, right we talked about branch coverage when we did coverage. Right? So let's go through this. So, what's the property of interest here?
Yeah, it's branch coverage right? More specifically it's the, uh, whether or not each branch was taken right? So, uh, I mean, it's just yeah, it is just rent coverage, right?
Okay, um, what is the information related to the property of interest that we're going to going to collect?
All right, so well, the property is a percent, right? This is like the output here is going to be like, 95% branch.
But we need to collect some specific information from the program that we're going to use to compute. Right? So what what, what information do we think in.
This is actually quite yes. Sorry.
Yeah, it's it's, uh, like, uh, just taking yeah. Branches taking. Yeah, that's a good way to say it. Right? Like, which which branch was executed, when that's how I have it in the slides here. But, like, the question is like pretty Frank, did we go down it or not? Right we're gonna collect that information.
That will allow us, we hope to compute this 95%. That's fantastic. Then we need a mechanism for collecting that information. How will we instrument the program to collect? Which branch was executed? When.
Print statements yeah print statement should work right? Just add a print statement and each branch. Yep.
All right, then we need test input data. What testing? What are we going to? What's happening? What are we gonna use here?
Yeah, well it's, it's this right this test, right? Whatever tests we, we're trying to compute branch coverage for and so that's relatively easy but we could also do something like, uh, use test input data. We generate earlier in class. There's many options here.
Okay, and then we know mechanism for learning about the property of interest from the information we collected. So, how will we get to 95% from these logging statements?
We have a list of logins, uh, of print print outs from the program. Highly compute. The branch cover.
Exactly number of prints that happened divided by the total number of branches. There are right and, you know, maybe there's some like, we need to do uniqueness checking. Right in case we hit the same branch twice some things like this. Right? But in general, like, this is for brand coverage place, this isn't too hard, right?
So, we have to do some post processing, we have to discard the duplicates and then we divide observe number by serial number, which is what we said.
All right does this make sense? Do you feel like you could do this with for some other analysis if I asked you to.
Like say, I asked you to do this for icon do we feel confident? We could.
I would suggest that by the time of the exam next week, or sorry in 2 weeks, uh, you should feel confident to be able to do this sort of thing for any of the analysis we've discussed.
Right, because this is the sort of question I think it would be perfectly fair to ask on the on the exam. Yes.
I think you read from the data.
Yeah, so it depends right it depends on the analysis like, in this case, um, we're, we're, I guess we set up here, uh, branch coverage of the test suite. But, um, I guess I didn't well define in this particular example like, what we're getting the branch corporate job, right? If we want the branch coverage of any specific test suite and the answer here is like.
That specific test suite if the answer, the question is, we want to get branch coverage in general.
For some test suite, then you need to make a test case, right? So, you know, uh, it, it really depends on the context. Right? But, um.
Where this has input data, it's gonna come from matters a lot for most dynamic analysis because most dynamic analyses tell you something about the program with respect to this input.
All right, we'll come back to this. I have a few points later in the deck on, uh, input dependence right? And I'll talk about more of that.
Does that answer the question? Well, enough for now? Yeah. Yeah.
Any other questions here? Yes. Um, so this is a.
Like, for brand coverage, especially, um, like, if you got a bunch of print statements, you need to like, uh, like.
Get them out of wherever they are, and.
Separate them from the program's actual output, things like that. Right? So, for example.
Consider a program that also Princeton standard app if you print all your branch coverage information to standard out, right? Then you need to split that apart somehow. Right?
Instead you printed all of the branch coverage information to a particular file, then you need to go and open that file and make sure that they're not duplicates and stuff like this.
But in general, like, there's always going to be some kind of growth properties that in almost every, uh.
Like, actual practical implementation of some dynamic analysis. Yeah.
Other questions here before I move on. Okay.
All right, so that was the end of our overview. How are we on time.
Ooh, not so good. Okay so I now want to talk a bit about instrumentation, right? And then we'll jump into properties, but 1st, with respect to instrumentation here. Um, my goal here is to like, not.
To allow you to go build, go out and build your own instrumentation right now it's to give you a sense of what would be involved if you did. Right?
Okay, so the question here is, how would you actually instrument a program to collect information about a property of interest? We've talked about, like, adding print statements and so on. But, like, how would you add print statements to a program automatically?
Like, serious question like, how would you do that? There are some, some options, right? We could do a source of source transformation right? That is to say, we take the program source and we modify it somehow. Right? And we could do that by hand for example, right? Like, 1 way to compute coverage information for our program is that you could go into that.
Program and right. Print after every line.
Do you want to do that? I don't think so for even anything, anything, but the smallest program doing it by hand is probably not going to be feasible, but you've done print line debugging. Probably. I know what what is meant by front line debugging. Right. That's where, you know, you know, that there's a problem at a particular line and you like, print out and you just right.
French blah, blah. That is instrumentation, right? It's very specific tailored instrumentation at your too lazy to automate.
Most of the time, right but it is a form of instrumentation so the kind of instrumentation that you would do by hand, right?
For these kind of automated dynamic analyses, though, we're going to want to do something smarter than go by hand right?
Um, so, uh, yeah, so, like this, this 1, we did by hand, I made you to do this 1, right here.
Another thing you could try and a regular expressions. All right. I'm familiar with regular expressions.
Does anybody know what this means?
Or even maybe it means is wrong, because I don't know what is what is for.
Huh.
Um, not quite so this, like, s, slash slash something, Flash something slack g, uh, format um.
Format is said, which is a Linux utility, right? Oh.
um so what this is going to do is it's a search
It finds everything between this 1st, slide from this next slide, replaces it with everything in between these next Paris office and then this g means do it globally in the whole file, right? Don't just do it once.
What's happening here is that we're searching for, like, any line that has a semi colon at the end, right?
And then we are, uh, adding a, a.
uh they compute the time run that line and then print the time the current time minus what the time was in the start right so what this is doing is checking how long each uh each line takes six
All right. Um, okay, so what, what can go wrong with regular expressions?
Like, just in general here. Yeah. Okay. Uh, so, yes, this, this would probably be expensive, right? In the sense of, like, uh, actually applying it to a large code base might take some time. But, like, programs aren't that day to be honest.
Right like, regular friends are relatively fast to it. Why compared to some of the other prices we're going to discuss in just a minute? So that sounds like a key problem like this.
They tend to be slow, but this would probably be fast enough. Right?
But even on, like, a small program, where this will be fast enough, there's something that could go wrong.
My transcription is worse.
Yes, they can go. That is correct. Right? The key problem that we're going to start to run into here is.
uh this is looking for lines that have a semi bullet in the seventy four right
What, if there's something in the program that I like, it should not actually match this right?
So, for example, we have, uh, align, like, this is a comment.
Semi colon why is this on here? That line? I don't know, but we need to interrupt this line to check. I wanted to start skip.
No, probably not right so that would be a bad idea, right? Um, where else could like it and in general, like, uh, the, the key problem with right is that writing a Reg for code is going to be too hot right? That's not going to have any false positives or false negatives, right? In the sense that we're going to discuss in this 10 minutes, right? Cause it's getting right right, right.
They're going to be basically impossible in general. If someone suggest that you write a right next to, uh, anything more complicated than necessary. Certainly replace info. You probably shouldn't do that.
Probably about so what can we do instead? Right?
So, we're going to we probably want to do something else. Right? And this is the point where we're going to have to turn to, uh.
Like, something more complex than regular expressions, which usually means, uh, compiler.
We're probably going to have to either modify the compiler or modify the run.
So, how easy is modifying compiler? I have personally modify the compile before.
Surprising that's known in room except me. All right so it is hard, right? In general like, this is like, for example, the reason that you need, like, well, why do you think AFL needs to have its own version of.
Well, it's computing coverage for your program, right? And so AFL, its job is to insert insert the instrumentation that AFL needs.
Right that's the reason you have to compile like, when we did the AFL lab, right? While you had to compile with AFL, instead of just regular, if you can compiled it with regular GCC, it wouldn't work.
Okay um, so, uh, you know, if you actually want to write your own instrumentation for this.
You're probably gonna need to modify a compiler in some way. Right but, uh, you know, that that may not be as bad as you think. Okay, so here's the sort of general way that this might work, right? If you can do a source of sort of transformation, right? If you can write a regular expression or some other relatively straightforward way to do it, you can get Internet source, which you can then compile that's easy. Or you can just.
For our client, who is an instructor, compile it directly so in order to actually instrument something with a compiler, we're going to need to parse some code. Right? I know about parsing.
Hopefully most of you, right? All right. So parsing trends program text into some intermediate representation like an abstract syntax tree and pretty printing is the reverse of that. So if we have something like 3+items, 1, parsing would turn it into an like this 1, right? And pretty printing actually note that this is an, like, the ones you're using for homework, sex and.
Printing turns it back into, uh, this are you, we're, we're all familiar with these terms.
Okay, so, uh, in this representation modifications to the program are actually relatively straightforward. In fact, what you're doing in homework 6.
Writing mutation operators is really fundamentally not any different than what you need to do in order to, uh.
Add in print statements automatically. Right adding in print statements automatically would also be a modification to the that you then need to print back out. And so part of the reason that homework 6 is covering this week. In addition to being relatively hard. Is that.
what you're doing in homework six is not that different than what you need to do if you wanted to implement your own dynamic analysis of some kind right
Fundamentally homework is about a dynamic analysis. Okay.
So this is a pretty standard technology you usually do not you usually should not write your own parser as a general rule, right?
Did you write your own parser for homework? 6? No.
I hope you did not instead you're using the Python library, right? And it should be doing that for you right? Often Pre printers are written separately right?
There are standard ones for most languages, but, you know, it's actually not that hard to write a pretty printer. It's just recursive to send over the and print it out the way that you want. Right? And you can like.
For most practical languages so we think it's white visitors pattern measures that you.
I feel relatively easily modifying. Right? That's the sort of libraries that you're using for homework 6 right now right? Uh, for Python but for other languages, you'll you'll, you'll encounter the same kind of ecosystem. Right? Uh, they might be more annoying like, for example, Java, Java parser is note, like, notably terrible at actually representing the semantics of.
It has bugs everywhere, but, you know, sucks right?
That that's trivial. You don't need to know it. Right? The point is that, um.
In general, like, this is a standard tax that you should be able to get for any language. You're actually interested in most of the time if you're building your own language totally different but you probably will never be in that situation.
No, Java has a, uh, its own grammar that is not the best.
Yeah, um, so the parser in, uh, Java is fine right? So, uh, it's, I mean.
It basically is the language definition. The problem is that it's not easy to access. Like, it's intentionally like.
Jarvis is designed intentionally to make it hard for you to extract the, and the reason for this is a mystery that is only known to the people who originally wrote Java in the nineties. Right but the language itself makes that really hard for some reason. And so people have built some open source alternatives. The most famous of which is Java parser, which is a buggy mess.
You know, it's an open source project that people do in their spare time. Right? So that's why it's bugging us.
I'm being a bit harsh to the, the project. That's not really fair. I should be nicer. I've just been frustrated with it like, in the last 2 weeks.
All right, so you should basically never have to write a parser. Right? Even if you want to do this kind of dynamic instrumentation as you're seeing, and hopefully hopefully seeing in homework 6.
It's very tractable. You can definitely do it right?
And so, as I said, you are already doing it for homework 6, the basic concepts, you instrumentation is exactly the same, right? If you wanted to instrument the Python program to compute statement coverage, you could easily write that code. And I could have.
Quite frankly, I could have given you that as the assignment instead of mutation testing instead. I just think mutation testing is much more interesting.
Because you actually have to be a little bit creative to do well, whereas for statement coverage, you would just have to do the monitoring, right?
Mutation testing, there's a more to it than just a monitoring.
All right, any questions about this does it does the commonality between these 2 make sense.
Alright, great. Okay. So another possible option here, rather than instrumenting the compiler is to rewrite the binary, right? It's to say to go into the binary or class file or the, like, and make a change to it directly.
Um, this sounds insane, but it's actually more common than you think for a dynamic analysis. Right? So, for example, this is the standard technique for doing dynamic analysis in Java. Right? Instead, what you would do is you modify the, the, you modify the Java byte code that the is going to execute. I'm familiar with Java code.
In the sense of not like, I could interpreted Java by code program, but in the sense of I know it exists and, you know, it could give a definition.
So, basically, the core technology of Java, the thing that makes it interesting relative to other programming languages is that, uh, Java does not produce executables.
You don't get an executable out. You also need to have a quote, unquote job runtime environment, or in order to actually execute a Java program.
Why is that? It's because Java converts your program from Java into quote, unquote Java bytecode right? Which is this stack machine? Right that? Uh, it looks a lot like, I say, this assembly and, in fact, it's very easy to convert from Java by put into activities assembly. There's a reason for that to make the gbm fast.
Right. Um, and, uh, this job site code is portable, right? You have a different for each, uh, architecture right? So, you know, if.
You're running on 86. you you have a for equity 6 you're running on a phone you have a, for the phone.
All right, uh, it's an arm processor, and you can basically what this means is, you can compile your job program, wants to Java by code, and you can execute that program anywhere that you have a Java interpreter.
All right, I'm in this sort of 2 stage compilation process was like, the big invention of Java. It's like the thing that that made Java famous, or made it popular.
Um, so the idea then is that you can, uh, transform back and forth between John and right, right. It's not very hard.
There are libraries that will allow you to do this. And so what do you want to do?
If you were going to, like, do dynamic analysis of Java, like, say, you want to compute half coverage or something for a Java program if you're going to just take the Java bite code, convert it into an address orientation, and then convert it back.
All right, and this will allow you to relatively easily instrument, drop a company.
And so this is the sort of standard technique, and a lot of a lot of programming languages, right? It's just go straight through the binary.
Um, so, yeah, this is just briefly, this is like, what Java bite could actually looks like, uh.
Can anybody read this? Not surprising again I just want to be clear that this should look a lot like, assembly to, you.
Right, yeah, like, download by code is kind of it's like a special version of assembly that Java interpreters know how to read right? Or Java or how to read it. Right?
So, what does the like, how does the actually work? I just want to give a brief idea of how this, and make it clear. This is actually not that complicated.
So, like, there's a, there's a spec for this, right of how the Java code works right? It's fairly straightforward. Um, and you can go. See, you can go look at the byte code from Java process. It's it's quite simple. If you have Java, you also have Java, which if you run Java P on some cost file, it will print it all print out all the backups.
Look at it, it's not that hard to read. Right? So, if you want to do that, you totally can. There are lots of analysis and rewrite frameworks for these kinds of things. Right? And so, for example, there's this like, uh, there's the Apache B cell is what I would call it, but the by code engineering library, right?
It allows people a, allows you to relatively easily do this kind of work, right? And there are other tools as well. Right? So the again, the point here is just like.
the tooling here is standard right for every language the tooling is relatively straightforward like you're seeing what it looks like in python but i want to emphasize that this is not super hard right if you want to instrument drug programs like someone who's done all the hard work for you right you will just have to interact with it in a relatively straightforward way and so if you want to deploy a new dynamic analysis
It's often not that hard to build 1 if you can get them, like.
Define the correct information that you're collecting right right. The hard part is usually making sure that you collect the right information. Not like the part where you actually modified the job by code modifying. The Java by code is straightforward. Right? Like, yeah, you might have to spend the day learning how to use this library but like, once you do it, once, it will not be too hard.
Does anyone have any questions about that high level point?
Okay, moving on then. All right. Yeah. Okay. So yeah.
Again, I want to come back to this point about quote is data, right?
This all of this is just treating the code is another piece of data and modifying it. Right? The same way that you would modify like.
A graph, right? And if you're doing a graph library, right? Ummm.
You compile in real time, or just like other libraries you can write code against them, you can modify code and rerun it and it's modified form. It's not too hard. Right it's not all that fundamentally different than other kinds of programming. Right? And I think there's this sort of nathi reputation that like, compilers have in general for being really hard to work with. And that is somewhat deserved.
For, like GCC, right for most other languages, if not true right and these kind of libraries make it relatively easy to do at least this level of, of, uh, of change to programs in an automated way. And I know I complain about Java parser but, like, it's honestly not that bad. And the thing is, you don't usually need it because.
Apache cell is so much better, right? You just modify the by code and stuff.
All right, so, um.
Just as a like, just to give you a sense of, like, what, what is possible here right? The kinds of rewrites that you can do are all there's all sorts of stuff. Right? So you could, for example, it's pretty easy to rewrite. Uh.
Your program, so that at the beginning of each method, you check these parameters not at all.
Quite straightforward, right? It's a pretty simple track. Alright. Um, you could, uh, you know, performance your overflows you can do profiling, uh, you can, uh, like music, like, do some fancy stuff with the database. And now we're using database connections you can add encounters, uh, to keep track of path for branch coverage. Like, this is exactly how works.
Like, this stuff is not too hard.
That's again, the emphasis I want to make here. Okay so there are also some other techniques that we could use for inspiration, besides the ones I've talked about I just didn't talk about these relatively quickly, just to give you a sense of what other options are available. Right. The 1st is using a or emulator. Right this is a valid grind works.
You.
So, do you use the same standard, Eva, or you probably learned, uh, when you use, like, more of the program and see.
All right. Um, so the, the key idea here is basically, is that you like.
You rewrite the code as it's running right? Um, the, the like, the, the VM takes care of this for you right? This is how breakpoints actually work right? Um, if you've ever wondered how actually does this break points are sort of a special case, because, uh, they require operating system support. Like, the operating system specifically gives permission to, to.
Things that would be crimes for other programs that would get them help. Right but you can also turn that on for your program too. If you want it just requires you. Okay.
Um, so this is also how that works there are other tools that work like this.
Another option is meta programming. All right, so I've heard of monkey patching.
Okay, so basically, the idea of monkey patching is that you're running a Python program, right? You realize that you've made a mistake, but you're like, halfway through training machine, learning model and, you know, it's going to crash from the, the training phase.
You have 2 options 1 is you could just kill the program, fix the bug and then start over, but then you'd waste half the training, right?
So, instead Python permits you to change the text of the program as it's running. All right so, instead, what you do is you, like, you tell the interpreter actually, like, I meant this other thing, please replace this part of the file with this other text.
There are security implications for this.
All right, it means that, like, doing this is very dangerous if you don't trust the source of the new patch. Right? But if you are, like, in this situation, like, with the machine learning and all that, I think has been extremely useful, a few other, like, many other dynamic, dynamically type languages also support this like, Ruby also has this property. Right? Um, and it can be quite nice.
see macros are also like this right there are form of meta programming i know to c macros
Very few of you. Okay. Um, I've seen something like pound if.
At the start of a header file before.
Yeah, okay a couple of you. So basically what this is is it's a directive to this guy. Surprisingly do, uh, it's a director to the C. compiler telling it to go look up some variable, that's decided to compile time. Right? So, in particular this pound, if in depth here is used to not include the same header file multiple times.
So, it means that you can compile, like, many programs that are.
I have many many like libraries and only include 1 copy of each header file. They all depend on.
all right um so it's you know you if you write c basically have to do this if you're and interacting with other people's code right um
I kind of expected this to be something that you all relate to something that kind of move past it since it doesn't seem like anybody really knows.
But these also, unlike unlike monthly, Patrick, I would just say these get resolved at compile time instead of 1 time. Right? And so they, uh.
They allow you to, like, sort of modify the program when you compile it, but not as it's running right?
But you could imagine doing something like this in order to, uh.
Or using a macro, for example, to turn on and off coverage instrumentation by hiding it behind a macro guard. Right? So that it would be included. If you were running in coverage mode, and not included in the, the object file if you're not running a coverage.
Finally, they're also sort of generic instrumentation tools. I've heard of apps aspect oriented programming before.
Couple of people yeah. Okay.
So this was a, uh, this technique after Kareem programming is super popular for a while. Um, I would say, like, the thousands day.
The idea basically is that, uh, there are quote, unquote cut points in the program where you can, uh, sort of, you say, I want to do a thing of of type X. right? And then that type X is left undefined.
And then you define X later, right? And then everywhere you've said, I want to do a thing of type X, you insert the X. so, for example, uh, this are classic example of this is a login.
Right. So if you, uh, you say, I want you to log some information, but I'm not sure how much.
All right, and so you write down, like, here's the information that needs to get logged and, like, here's how how to decide whether to actually log it or not. Right and then what you can do is when you start the program, you can say, like, log at the info level or log into debug level or log at the non level. Right? Whatever.
And then, like, the aspect of running programing tool will automatically do that for, you.
All right. Um, so there are a number, as I said, there are a bunch of other ways that we can do this. Um, all of these, I would say are probably less popular if it maybe.
The 1st, 1 then, like, the, the techniques that we talked about before, right? But any of them are perfectly valid ways to implement your dynamic analysis. If you want to.
Any any questions about this before we move on actually any questions about instrumentation in general I think this is the last slide I have on this presentation.
Alright, moving on so I now want to talk about some properties that dynamic analysis sort of have in common right? What are the sorts of things that we can do with dynamic analysis? Or what are the sort of like how, how do they tend to behave right?
so first let's talk a little bit about cost and limitations first one of the big things that uh like big costs of dynamic analysis is that there's always going to be some performance overhead for recording right you're recording some extra information as the program runs right
And that means that the program will run slower right? Maybe it's acceptable for use in testing right? Like that the program is a bit slow, but is it acceptable in production? Right? That is to say 1 user.
It depends on exactly how much overhead, right? If it's a very negligible overhead maybe it is. Okay.
But usually, in most situations, you do care about performance and so, uh, dynamic analysis can be unpleasant for that reason. Right?
It's best deployed in a situation where, like, during testing, for example, where you care, but some dynamic analysis have very high overhead.
When we talked about a racer and a bit, like, keep that in mind.
You also there's going to be some computational effort for the analysis itself that's happening, right? Like.
After after you're done, and if you record some information, and then you want to compute something like computing, that thing might also be hard. Right?
As most of the examples we've looked at today have been relatively simple things that we've computed like.
Branch coverage involves doing some division, right that's easy but you could imagine a dynamic analysis that records, like a few small pieces of information and then, like, compute some complicated thing from it. So that can get large.
You also have transparency limits of your instrumentation. Right? And what I mean, by this is that instrumentation can change program behavior. Right? That is, I think about the observer effect in physics. I know what the observer effect is.
Few people right? What is it? Yeah, like the property changes when you use.
That's exactly right. So if you, if you, if you look at a, uh.
Where does this actually, uh, current physics like a cork or something to? Yeah too carefully shooting his cat. Sure.
Yeah, so the key question here is, uh, do you want to do you want to ship the thing in your timeframe?
Or are you willing to accept like that is to say, are you willing to pay the performance overhead for, uh.
Using your implementation for, in production, right? Or are you going to put yourself in danger? High devices? We'll cover in a minute but bugs are are.
At a high level bugs that are caused by instrumentation or caused by the lack of instrumentation. They're not observable with that.
And then there's also a question of accuracy, right? Do these techniques have false positives or false negatives? Right?
And I realized that I have not actually, formally defined the terms false positive and false negative for you all. So I'm going to just briefly as an aside go into that right?
So the idea of false positives and false negatives is based on an idea 10 X actually happened.
2 possibilities like this is the ground truth is excellent actually happen or not but for now.
And then there's another question, right? Another access, which is did a tool warn us about X or I did, did.
Uh, we think that X was going to happen for whatever reason, right? And usually, when we're talking about false positives, false negative reason to talk about, in terms of, like, a false positives or hospital.
So, false positive occurs when, or actually, I should say, like, what's in the yes, yes. Corner here.
True. Positive. Yes. That's right. So the reason it's true. Positive right? Is that, uh, it's true right? The yes. Is match and it's positive because the thing did actually happen. Right? So then what's in the? Yes, no corner in the top, right?
False positive. That's correct. Right because the tools that it would happen.
Right. So we, the positive here means it's a warning right of the tool warned, but it's false because it's not correct. The tool is wrong about it. Right? So then what goes into the other or the bottom left corner.
A false negative, right? Because the tool incorrectly said that nothing would happen and something that.
And then finally true negative goes in the bottom. Right right. And so a tool is doing well, if it's in the top left or bottom, right? And poorly in the other 2 categories, right?
All right, so this is a useful tool for thinking about anything that might want us about problems in general, right? And I'm going to use this a lot. I realized I used it a bunch in prior.
Like, prior lectures, but never defined it. So, does anyone have any questions about this? Does this all make sense.
I want to be clear about what I mean. Okay. So, um.
Before we can really think about how.
That analysis generally behave, I want to define 2 other terms related to false positives and false negatives. Right the 1st is soundness, right or sound analysis. A sound analysis is 1 that reports all defects, and therefore it has no false negatives. Right? So, typically, this will over approximate the possible behaviors of the program. Right? That is to say, it includes all possible program behaviors and.
Also some others, right? And that means that they're generally going to be quote, unquote, conservative with respect to safety. So, if if the answer is, I don't know.
Then a sound analysis is going to say it's not safe right?
by contrast uh we also have a a notion of completeness right so a complete analysis is one where every reported defect is an actual defect that is to say there are no false positives right and this will typically be done by under approximating possible behavior of the program right and so it will be conservative with respect to reporting right it will not report unless
Sure.
Do these 2 concepts make sense? No one's asking questions I'm going to move on, right? But, let me just, uh, sorry for the J, Peggy quality of this, uh, this, uh, figure, but I like to figure a lot and I couldn't find a better version of it. Right? So, uh.
I just want to give intuition for this. Right? If someone says you can trust me when I say that your radiation dosing software, like running your 3rd, 25 is safe.
Where where do you hope that they are? What kind of tool are they using?
yeah you want them to choose tool a right soundness it's under approximating program behavior right the idea here for over approximate right if if tool a uh soundly says that your your code is like that the code is safe it means it actually it right it means that there's no behavior that the program could possibly do there'll be unsafe because it
Approximately the behavior, so it checked all the actual behaviors and it checks some other ones, right?
So, it will only certified programs that never have an.
that is to say the programs in britain by contrast your tool b will certify some programs with errors and also some programs with no errors
Usually, I such a tool will do better right? In the sense of it will sort of buy more programs with no errors.
But at some cost, right? It will make a mistake occasionally. Okay.
By contrast a complete tool here, uh, would.
Certify every program with no error. Right? That is to say it never warns about a program. That is correct.
But it will sometimes warn about, uh.
But it will warn about as many programs with with an actual error as it can, it will miss some. Right? But in general, there's nothing over here that's saying, like, we'll find all the programs with errors.
Well, well, there's nothing over here that's sort of by all the programs that architecture.
Well, we'll find them all by rejecting them.
All right does that make sense? Let me just give a couple more things here. Right? So some sound analysis like a says, you want to say, it means that he want actually is safe.
Right, that's what it means. That's what soundness means, but I might be safe is safe there, but tool a, a says it's unsafe.
Right. So that's a false positive. So sound analysis almost always will have false positives. Right? By contrast is, in fact, safe.
But, uh, the complete analysis, there's like.
Uh, all right, I maybe I've missed drawn see here. Um, but, uh.
Oh, yeah now, hang on, if you want is actually safe and the complete analysis. See, does it say.
Right, but see may say that program 5, which is unsafe actually, is that.
It's going to have false names. It might miss things. Okay. I'm sorry for it. I hope this is, uh.
Not confusing, but that looks on your face. I suggest that it is.
Does the intuition here makes sense? No. Okay. Yeah. So, um, the core intuition is that soundness means only certified programs that are definitely 100. definitely safe. Right? So, if you say, like.
If you say that I have a sound tool for preventing no point or exceptions, then, uh, if that sound tool says that the code is safe. No, no pointer exception can actually occur.
By contrast a complete tool for no point or exceptions is promising that every time it does warn about an old pointer exception, it's possible for that D reference to actually be occur on a no value at run time. Like, there is some execution of the program that will actually cause an old pointer exception.
The sound tool is some of the time going to warn about programs.
That are actually safe because it's over approximating the possible things that could happen.
It thinks a pointer can possibly be an, all on some execution, but really that's infeasible. For some reason.
Right. By contrast the complete tool might miss a real problem because it's only going to warn if it can truly certify the thing. And if it can't find that.
a set of inputs that causes the thing to actually occur no point or exception say like for example because the the value that's actually null requires us to solve some formula that's very hard right then it won't it won't warn about that even though that thing might actually fail at runtime
So, complete tools, every time they warn you should take action because if they if they've warned, it means that there's something definitely wrong.
Sound tools if they don't warn, then you have a proof.
Right that is to say, you can trust them and under the hood, the way that you build a sound tool is you do approve right? You build up, you use a different broker.
The tools are useful for different things, right?
So here's a question is it possible to have a tool that can be both.
Why do you think that.
So so being both really requires us to get the right answer all the time.
All right, and so for some properties, this is doable right? Like, if I ask you, what's what's, uh, like.
please provide me with a tool that satisfies whether my certifies whether i'm doing addition correctly right you can do that you can completely and soundly tell me every time i'm doing edition wrong right over say
It's not very hard, right? However.
Uh, for programs in general.
We know that there are going to be some limitations here. Right? In particular like, the bad news is that every interesting analysis of programs.
Is either on sound or incomplete or both, and the reason is again rice's theorem, right? Which we saw in the last lecture, which says that any non trivial semantic program of a property is undesirable. That is to say anything that is semantic. Here means it's about the meeting, right? And not about the text.
And non trivial means that it's not it's true of some programs and false of others.
All right, so, like, is this a program is a trivial property over programs because it is always true.
Right. Um, and, uh, like, does this programs text contain the spring? Fu, is, it's, it's not a semantic property. It's a syntactic property. We can simply search the program for it.
Right, but if we ask there was any variable in the program ever contain the value quotes food.
That's a semantic property, because it's asking about what the program actually does.
It's something about the program's actual meaning or the program's actual run time behavior.
And so why is this a corollary devices? Let's just step through the proof, right?
So, our program analyses are decidable, right? Because they run on that computer.
You're going to execute them on a computer and therefore they must be decidable. That is a that's what desirability means. It means that there was an algorithm for them. They're going to execute in finite non exponential time.
So they must be approximate in some way, right?
Uh, because they cannot possibly.
Compute the value exactly right. Properties of programs because of the presence of loops, and are not, uh, able to be checked directly.
Alright, we can't get a, we can't always track them. Exactly. There's no algorithm for that. That's what rises there and tells us right?
And so if they're approximate, in some way, they have to be approximately in, at least 1 of the 2 directions right? Either they must over approximate or they must under approximate.
And so soundness requires that you only over approximate, you never under approximate and completeness requires that you only under approximate. You never over approximate.
And so, because because of that, you can't have both right for any any interesting program analysis.
So, the bad news is that every analysis you're going to encounter in real life is limited in 1 of these 2 ways, right?
Does this make sense? Okay. So then what should do do you think it's easier to achieve for dynamic analysis? Soundness are completeness.
Yeah, so, um.
Soundness is going to be easier to achieve right when we can reason about all the possible executions. All right.
The problem with dynamic analyses in general is that they're driven by some input data.
Right. And so that means that they're going to like any information we compute using the dynamic analysis.
It's going to be conditioned on the input data that we choose.
All right, so that's actually going to mean that soundness is going to be hard to achieve because we need to generalize from, uh.
The executions, we do see to all possible executions. Right? And this is like, dextra back again with testing can show the presence of bugs, but not their absence. Right? And so in in practice. Actually it's going to be completeness. That's easier to achieve for dynamic analysis. Because if the dynamic analysis actually demonstrates a problem.
Then it must have actually occurred, right? I mean, and of course there are, there are sources of approximation and dynamic analyses that can make it wrong. But, like, in general, like, if we observe a problem, then we're going to be able to, to.
Dynamic analysis allows us to observe the problem directly. So completeness is the answer, right? So generally dynamic analyses are under approximating because they're reasoning only about the program actually executions. They are actually absorbing right?
Um, it's hard to generalize from, like.
The number of tests that you have to all possible executions.
By contrast when we come back to static analysis, static analysis is traditionally at least, uh.
Designed to be sound right it is actually possible. It's been embarrassingly recently shown possible to build stack analyses that are also complete.
Um, and in fact, that means it's probably possible to build a sound dynamic analysis too, but I don't think anyone's ever shown us down a sound dynamic analysis in practice. So we're reaching the.
The edge of like, what's possible in the current research literature I mean.
The, the 1st, like, proved complete sounded a static analysis was like, 2019.
All right, so.
Okay, so we'll come back to that after spring break, but when thinking about dynamic analysis, you should be thinking, like, completeness if it's either of these. But in practice, like, most analysis are both.
Unsound and incomplete. All right, let's think about testing it's possible to get false positives from tests. Right? We talked about like, flaky tests that failed because of non determinism, for example, right that's definitely a false positive.
And so, even if you are trying to write test carefully, it's easy to accidentally get it wrong.
So, like, testing in general is not going to give us a proof of completeness. Some, some dynamic analysis.
Okay, another thing that's important about dynamic analysis and the dynamic analysis all share and this is related to the notion of why they're complete but not sound is that they're very dependent on the inputs that you give them. Right? Whichever inputs you choose to exercise the program is going to have a big effect on what actually you learned from your dynamic analysis. So that is to say.
Fullness of the analysis depends a lot on the quality of the test input data.
And as we've discussed in this class, how hard is it to make quality test input data.
Or, to tell whether testing, but data as a high quality, it can be difficult. Right? So.
This means that in sort of an arbitrary setting, if you were just asked, like, run this dynamic analysis on this program.
It's hard to tell whether you've got good answers, right? Or whether the answers are representative of what the program would look like under different conditions. Right? And so you must reason about the test suite as well. And that's part of why we're talking about this now. And why we didn't cover this on day. 1 right is because the test input data.
You already have to have a way to tell if it's good in order to evaluate whether the results of a generic analysis are actually actionable or a useful.
All right, so this is great if you have lots of tests and they're really good. Right? So in practice, the sort of received wisdom is that whole system tests are often better for, uh, dynamic analysis. Then, like, unit testing is, um, the reason is basically that, uh.
Many unit tests are not super indicative of the way that the system actually behaves in practice. Right? And in most dynamic analysis, you want to learn something about how the system actually behaves in practice.
Um, this does this, this is definitely like.
Advice for the general case, but can be wrong in specific cases though. Right? And like, when considering specific analysis or programs.
Another important question here is, are the tests indicative of normal use and is that what you want.
All right, often the case that we write tests in such a way that they are like, regression tests, for example, and therefore specific input to replicate known defects scenarios to say, like.
Your regression tests, they might specifically trying to, uh, trigger things like memory, leaks or race conditions right? Is that representative normal use.
No, almost certainly right most progression test suites are not representative of how the program is used in real life. Right?
And so, uh, the usefulness of the output of dynamic analysis on a particular set of inputs definitely also depends on like, Where'd you get the inputs.
If they're a regression test suite that tells you something about how the program behaves on the regression tests, right? In bad scenarios. But that might not be what you want to know. Right? If you're asking questions about, for example, execution time.
You're probably doing it wrong if you're doing it using the regression test suite because the regression testing is probably not representative of, like.
Where a user is actually are spending their time, you're trying to optimize, like, the speed of the program. You probably want something that's more indicative of, of how users actually use the program.
Yeah, what are the.
In the real static analysis.
People that multiple people often ladies it's definitely it's a good question. Yeah. So, um.
The 1st thing I would say is that the most common dynamic analysis is definitely just write tests.
And everyone uses that right?
Some of the other dynamic analyses we've discussed, like, fuzzy, for example, are quite common as well. Right?
Yeah, profiling yeah. Memory. Yeah. Collecting like, memory, profiling time profiling. All of these are pretty common. Yeah, I would say probably. Yeah so, for that reason yeah. Dynamic analysis is more common in terms of, like, looking for complex bugs right or like, trying to do.
Difficult automated reasoning the state of the, the state of the art is much stronger in the static analysis side. Right? That is to say, like.
More effort has been paid by the research community, I would say to the static analysis side and so if you want to do complicated reasoning, it's often like an automated way. You can often do more with static analysis.
But with dynamic analysis, like, it's often more actionable the results are often more actionable.
And this is probably a weakness of this research community generally right has been like, historically program analysis comes out of compilers, which.
A static analysis, and so much more attention has been paid to static analysis.
Good question. Okay. Um, moving on since we are, uh, already at 805 um.
Another thing that dynamic analysis has to contend with is the observer effect, slash highs and bugs right? Which I mentioned before. So, Heisenberg is a fault. There's some kind of problem. That occurs during execution. That only occurs either with or without some kind of instrumentation. Right? And what I mean, by with, or without here, is that what I mean, is it dependent on the presence or.
The instrumentation, right? If you have instrumentation and a bug doesn't occur, but if you take the instrumentation away and the bug occurs, then it's a hyphen bug, right? Or vice versa.
Does that make sense? So, I think I just said this out loud, so I'm not going to say it again.
This is caused by these artifacts, like, in physics right? Instrumenting and monitoring the program changes its behavior.
Right. The program might be slower the program might be using more memory, et cetera. Right. Um, and so, uh, you know, when you're.
Like, looking at a dynamic analysis, we're looking at the results of a dynamic analysis you often have to deal with these kinds of things, right?
And this is like, yeah, it's very, very inconvenient. Let's say when.
Trying to figure out what's going on if you encounter a problem. And then when you attach it, there's a problem in production.
And then you attach to B*** and the problem goes away that's going to make you very frustrating.
So there are sort of 2 key considerations that you keep in mind when you're designing instrumentation, or thinking about debugging a horizon bug. The 1st is Canon, showed you deploy the instrumentation live, right? If you can.
Uh, that makes your life way easier, right? If it's possible for you to run the with the instrumentation in production, then you have the sort of ship what you test that I mentioned before, right? A situation where if something goes wrong in production, you'll have all the same instrumentation that you would have in the test scenario and it will be easy to reproduce.
Right. So if you can do this, you often want to the downside of course, is that the performance overhead might be too expensive right?
So that's a difficult trade off. Like, the choice not to deploy with implementation is a choice to probably encounter highs and bug. Eventually.
The 2nd thing is consideration is will instrumentation meaningfully change the program's behavior right? Um, and this is, uh, this is a question more for you as an analysis designer, rather than it is for you as someone deploying the analysis.
But when you are trying to debug or Eisenberg, this is also a useful thing to think, like, because if you have a high bug, it means the instrumentation is having an effect. Right? And so, uh, the bug is probably in some way related to something that's similar to the instrumentation.
Right. So for example, if you have a bug, uh, that's, um.
Goes away like, when you try to run in.
Right then probably the bug is somehow related to memory.
Because what is the algorithm doing? It's instrumenting your memories.
Right. It's keeping track of what what what you're freeing and what you're allocating. And so if your bug goes away, when you change, how you free and allocate, that suggests that there's something wrong with something that you're doing, relate to free free and allocating. Right?
And so this is a useful sort of line of reasoning to follow if you're trying to debug something like this.
Any questions on the does it make sense?
So, uh, instrumented with so.
Yes, so usually, uh, it, it relates to instrumentation, but, like, it's, it's rarely caused directly by the instrumentation. In my experience. These kind of bugs are often caused by, like, a side effect of the instrumentation. Like, the program is dependent on timing, or the program is dependent on, like, scheduling or.
Program is dependent on, like, how much memory is available at a certain time, or something like this or is that.
All execution. Okay.
Yeah, so that's more like that's more like standard non determinism rather than the bug. Right? Um.
Various conditions can be highs and bugs though, like, if you observe the race condition in practice, but then when you try to, like, control the scheduler, the instrument, the bug goes away.
Your deterministic schedule does not exhibit that suggests that it's a scheduling book.
And that would be a hazard bug in that case, right?
The instrumentation to change the schedule and force the schedule to be deterministic, makes the by go away.
So, it's the other questions here.
All right moving on. So I now want to just go through a couple of examples, fairly quickly of a few, real example analysis. Right? That are practical analyses that have been deployed in practice and I think I've got 4 examples here. Yes. So, for examples these are all from companies that you either.
Or have died, but we're famous at a time. Right. D. C. used to be a big deal and.
Then I got bought by somebody, I want to say, maybe sun back in the day, right? Like.
So this is a company basically like, it's, uh.
Dot com, boom. All right, but Netflix in Microsoft you probably heard of it.
All right, so we're going to go through all 4 of these, and I'm just going to briefly explain what their key innovation was and how they actually worked. Right? And the goal here is again, to give you a flavor, right? Most of the well, actually, all of these are historical. All right but I know for a fact that at least chaos, monkey and driver verify are still in use. I'm fairly confident that chess is too.
I'm not actually sure. All right, so, let's start with eraser. So, erasers question. The question is trying to ask and this is.
Like, eraser is a joke, because it has the, like, it sounds like it has race in it. Right? Is there a race condition in the program? Like this.
Is our recommendation here? I think there is a risk condition.
I think there is not a race condition. We have.
2 votes to 0 There is not right and why isn't there a race condition here?
Yes, yes, because they're always locking before they unlock it. Right? So every access to the is definitely bargain. Right?
So this is, this is, in fact safe, we protected.
Right. Okay now let's consider this a slightly different variant of the program.
Right in this variant we have, uh, 2 different access to spread and each 1 is guarded by its own.
So, a race condition in this program.
I don't remember why I put it in the slide.
I remember why I put it in a slide. This example is taken directly from the paper.
And I have a screenshot from the paper and a few, a few slides.
And I didn't want it to be different than what's here.
In the paper, they make another separate points about why that we're just ignoring.
Yes, yes, the, the 2 logs are fair, right? So if if this the right 1 is holding a new 1 then, uh, thread 2 will will span until, uh, becomes available. Right?
In the block work once I.
Um, so.
That is a very tempting idea. All right. Um, but in fact, this program does have a lot a race condition. There's a way to schedule the threads in such a way that, uh.
There's a database on ve, right. The reason is consider what happens if red 1, Holt, and and then thread to holds and they both try to write to be at the same time. So.
Right so the key insight in eraser was that if.
There's ever a case where there are 2 different locks that can be used to acquire the same variable. Then that leads to a possible race condition right?
And so the, uh, the key idea then of their, uh, technique is what's called a lock set algorithm.
Right. So the locks that algorithm keeps track of the set of locks that possibly can guard something. Right?
So the way that this works, right? Sorry the, the idea is each are variable must be governed by exactly. 1 lock.
Right if it's if there's a shared variable discarded by 0 locks, I mean, okay, then databases are certainly happening there. That's easy. But, um, uh, shared variable guarded by 2 or more locks is also dangerous.
All right, we don't want that and so that's what eraser was able to determine. Yeah.
So, basically, the algorithm is start with all the locks that could possibly protect me. Right? So, you know, looking at this, like, there are 2 locks.
You want me to right? If you observed that M is not held when you access V you remove that lock right? From the set of locks that could possibly guard V.
And if they set of of locks that could possibly Garvey is ever empty.
The nuance and you definitely have a risk condition, right? Even if you didn't actually see the race in this particular execution. The nice thing about eraser is that it allows you to know their race is possible. Even if you didn't actually observe it with the way that your threads were scheduled, right?
So here's the bit from the paper that I was talking about, right? If we look at, I guess this new 1 is either. I don't know why, but why in there that was.
I might have been halfway through making the that that figure I changed my mind.
So, sorry, uh, I'll, I'll remove that before I post this lines.
cause it's confusing alright so um here's an example right this is exactly the same example but uh we saw before that this one doesn't have like um and you can see here that um the the locks being held uh goes down to zero
Right. And so then I.
Eraser would warn about this. I thought I think I saw a hand a minute ago, because I have a question.
Yeah, okay. I'm just imagining things. All right so then does this work, right?
So, uh, yes, but, uh, it it is not perfect, right? The overhead is by a factor or 10 to 30.
All right, that's quite a lot of overhead. Should you run a racer in production?
No, definitely not, but it can be a useful testing tool and it Henry false alarms. All right. Good. Sometimes pretty false alarms.
However, um, you know, they applied it to a bunch of, uh, you know, relatively, uh, well, at the time important software is ultimately the ultimately so important software now. No, but this was the 90. S, right. Um, and, uh, they found, uh, some serious data races, right? Um, so, uh, this technique was, at the time, it's quite effective for data.
Detection, um, you know, we can do better. Now, you all read chess.
But, uh, you know, this was, uh.
Fairly fairly successful for the time. And the nice thing about it is that the algorithm is extremely simple.
Any questions on eraser. All right let's look at another example I want to talk about Netflix is chaos. Monkey tool. Right? So what is chaos? Monkey right? So this is a tool that Netflix invented in the early 2000 tends to test the resilience of their their I. T, infrastructure that is to say of their data centers.
Right so this is a tool primarily designed for the purposes of dev Ops.
Uh, basically the, uh, the problem it's trying to solve is well.
Hard drives do fail sometimes. Right? How often have you personally experienced a hard drive failure?
Quite possibly never right but the, the average hard drive lasts for about 5:6:years.
Before it goes out, right? If it's in continuous use. So if you have a data center with 10,000 machines, right? How many hard drive failure should you expect per day?
Like, on the order of, like, 3 to 5, right?
And so what that means is that hardware failures in sort of like large data center situation are basically guaranteed, they're going to happen at some point. And so, if you don't test your service for resilience to going down to hard drive failures, you're gonna have a bad time, right? When the hard drive that sort of 1 of the machines and services all is running on does inevitably fail.
So, chaos monkey is a really a high level technique for trying to solve these kinds of problems like the hard drive example. I just gave right and trying to test for them. So, basically the, the reason they have, they named it. This is that is a quote from 1 of the, uh, the authors of it imagine a monkey.
Data Center, right the forms of servers that host all the critical function of all the online, like this data center, right?
A big old computer, lots of racks, right? The monkey randomly runs up to machines and rips out cables, uh, or destroys devices by, uh, hitting them with a hammer or, uh, you know, I just.
Just as much damage as possible. Right? The monkey's job is just wander around the data center and cause problems. Right?
The challenge then is designing a system that is.
Dealing with the fact that it works, even though there are monkeys.
Right, the way that the core philosophy of chaos monkey is, you need to be able to defend your system against such random failures, because they're going to happen because equipment, whereas at, or, uh, you know, like somebody trips on a cable and pulls it out of the back of the server or whatever right? Bad things do happen occasionally at scale and so all.
All of their services need to be able to withstand that happening. Sometimes.
So, that's the philosophy, right? Um, another way of looking at this is that the actual practice practicality here? What does it say? Oh, it randomly choose the server and disables it.
And that's basically all this. This is not a complicated Albert. I mean, there are various ways to do this better or worse, but basically all it's doing is doing and randomly doing bad stuff. Right?
But they argue that, like, yes, it sounds insane but actually, this is the best way to tell whether the system it's actually resilient.
All right, and so knowing that this happens frequently, it creates a strong incentive for engineers to actually build systems that are defended against it.
All right, and so this incentive alignment here is as part of their goals, right? They want to make sure that if, you know.
A hard drive does fail, or, you know, power goes out from a particular server or something like this.
You know, nobody has their, uh, they're moving interrupted or whatever it is that people do on Netflix and so this gives the engineers an incentive to pass the chaos monkey tests.
to actually to deal with these problems so yeah they claim that chaos monkey is one of the most effective tools to improve the actual user experience of services right and i believe them like this is a very effective testing technique right this is kind of like fuzzy in your data center
You can think of it that way. Yep. Any questions about this idea.
I think this is insane. I do.
Like, it sounds insane on on on its face, right? And they, they kind of lean into that with the name, but.
It does actually work, so.
Yes, this started.
Yes.
That was an example to to explain right? I'm about to show what they actually do. Yeah. So that that's not that's not a real example in the sense of, like, that's problem, not theirs, right? But it's also the case that they want to be, uh.
Defended against that and like at that time in, you basically always just got bare metal.
All right, so in 2011, say, if you rented an server, like, physically you were renting a server, right? And you weren't like, renting, like, 3 servers across 3 different availability zones that if 1 of them goes down, your data is already replicated and you don't have to worry about it. Right? Like, Amazon has kind of brought that into their own area. Right? But.
At this time was not did not have that kind of support. So let's look at some examples of the sort of like, these are specific, real examples of chaos monkey. Uh.
Um, uh, agents, I think, yeah. Um, they call them, like, so send me an army Simian just means Monkey, right? So, uh, latency monkey is 1 of them, it induces artificial delays, right? Into the communication layer. And so the idea here is that they're simulating that other services are under heavy load, right? By making the, uh, like.
It's like, slowed down delivered a minute later or something like this. Right?
So this is that's 1 thing that they do another is quote, unquote, conformity Monkey, right? And its job is basically go around and look for instances that fail some best practices check like, not be part of an auto scaling group and just turns it off.
Straightforward enough right? Um, another 1 of these is the quote unquote. Dr monkey right?
And so, what it does is, it looks at, like, external signs of health, like CPU load and if it sees a machine that's running at 99.9% CPU, it turns it off.
All right again, like, that's the whole idea of this is just like, random random, shutting, like, randomly shopping things down and making sure that you can deal with that. Right?
Um, and then, uh, localization, international monkey detects, uh, like.
Basically, it's like, uh, mixes up, uh, like languages and things like this.
Asking for things in the wrong character set, asking for things, like, it chooses at random, like a, a language to request, and then makes the request in a different, uh, country that usually doesn't have that that language associated with it and things like this. Right?
So these are some real examples any questions about these does that make sense?
There are also plenty of others. Right? The whole idea of chaos monkey really is that it's a philosophy really? Rather than a specific tool. These are specific tools.
These individual ones, but there are there are a bunch more, right?
So, continuing on our parade through analysis, let's talk a little bit about chess, which is the tool that you all read about for today. All right, so just the core intuition of chess is the coupling effect hypothesis, which we discussed before, because I want to remember what the coupling effect hypothesis was.
We had a long discussion about it again. This is program for the exam. I test suite that detects a simple fault.
Will also likely detect more complicated faults right? That's what the coupling effect hypothesis does. I think I made an argument that I didn't believe this with respect to mutation testing right?
However, for scheduling, there's much better evidence. Right? And in particular chess actually crushes it here. Right? So, the, I, I said this is the, the coupling of this again right? You're balancing code a free balancing well, there's some big percentage of of the bug. Yeah, he has that they're probably also a small tree.
Right when we talked about a couple of different type offices before we were talking about it in the context of mutation operators, right? I argued that mutation like small mutations are unlikely to boil up this sort of big bugs that actually happened in practice.
This example, though is the input data, and in the case of input data, it really depends on the program for. I totally believe this. Right? And similarly for chess works, because for schedules.
The coupling effect to hypothesis is true. Right? And so part of the reason I wanted to bring this up again was just to make clear that.
I was arguing specifically about mutation testing when we talked about this last time and not about coupling effect hypotheses in general, right? You can have them for lots of different things.
Okay, so the idea of Chaz, suppose you have some concurrency bug that you can show off with a complicated sequence of 16 threatened or leavings and perceptions. Is there also a sequence with just a few presumptions that also shows off the same bug? Probably.
All right, usually there's not really a reason for most scheduling problems to require a 16 thread and generally there aren't that many locks in programs. There aren't that many shared variables. Programmers are very scared of using them. Right? And so if there is a problem, it probably means that you can, it's, like, caused by a small number of interactions.
and so the theory of chess basically is that most concurrency bugs are caused by a small number of interactions the fact that just works suggested that that is in fact true right
So, chess is a tool for finding and reproducing highs and bugs you all read the chest paper. So I don't feel the need to read this back to, you.
Does everyone feel like they understand trust from the paper? Did anyone notice who, uh, like, uh, the authors were.
Purchase yeah, so 1 of the authors of is who is a professor in the sales department?
Here yeah, so.
I think he has taught this class in the past. Yeah. And he's primarily a program analysis person. So I think it was 65.
Yeah, he's currently teaching, I think, a PhD level program analysis. Course.
Yep all right anyway so he's 1 of the authors of trust, which is just kind of cool fun fact fact.
And so if you ever want to know more things about chess, I'm sure you really would love it. If you wander into his office and started asking specific questions about it. Genuinely, I think you would like that.
All right, um, does chess work, right? Um, so, chess is pretty lightweight, right? Um, it's oblivious to synchronization protocols, such as locking discipline. It doesn't care how you're doing locking, which is convenient. Unlike say eraser, which does need to reason about what is a lock and what is an unlock, right? It needs to know to do that in order to compete the locks. Right? And this is.
It's important for low level, carnal code.
To not have to worry about, like, what the exact locking discipline is, because for low level, colonel code, the logging discipline is often very, very complicated. Right? Because there's a lot of interacting. Sure, sure. It's runtime overhead. It randomly samples a small percentage. Right? So, that means that that allows it to.
Be relatively cheap, right? Um.
It already it reuses Breakpoint facilities that are already supported by the hardware. Right? So this is also, um, a useful thing to do right? In general. If you deeply understand how the hardware works, or what the operating system provides, you can often really reduce the overhead of dynamic analysis by taking advantage of things that exist, but are not currently being used. Right?
So, 1 of the key tricks for making a dynamic analysis, have negligible runtime overhead.
As you think about the kind of thing you're trying to do, and what kind of program it is, and then things are there, any free registers all the time.
Can we use that if you can do that? Right? If you can cleverly take advantage of the hardware that you have available right? For utilize cores that aren't currently being used or utilize as I said, registers that are being used for utilize the fact that there's break point facilities are already there. Right? And you can set hardware break points and most modern architectures.
All right, that allows you to do much more with less. Right? This is an analysis design question, though.
Um, and it was effective, right? It found a bunch of colonel. Sorry a bunch of data races in the Windows colonel.
And, you know, prevented them so it was, it was effective.
Any questions there on tests on, like, how, and why it worked.
All right, um, let's let's hurry through this last 1, right? The last 1 I want to talk about is the driver verifier. This is a rather old, but still supported Microsoft technique, right?
The key idea of the driver verifier is that, um, when you, uh, when you call, oh, instead of calling open, you make a special my open, right?
And, uh, with the salt probability, it raises some, and with some other small probability, it raises and operating system, right?
The idea here, basically, it's to do a chaos monkey thing for the operating system.
All right, this is making the operating system fails sometimes at random right?
Why do we care about this in the context of drivers? They don't know what a driver is.
I know drivers some of you. Yeah, so the, the key idea of, uh, uh, uh, uh, driver is basically, is that if you have, like, uh, an open, uh, computing ecosystem, like we do, right? Uh, it's often the case that you want to, like, go to the store and.
I like a mouse, for example, from a 3rd party vendor, right? Microsoft doesn't know about the 3rd party vendor or about the maps.
So, what software actually controls the maps.
Right. The driver yes and that's supplied by the mouse's, uh, vendor. Right? So, like, some person who works at the mouse company.
I realize now, sounds amusing. Makes it sound like I'm talking about Disney. I'm not whoever makes the mouse, right? Provide some code, uh, that run like.
Effectively is the software for the maps, right? That allows the malice to interact with the operating system. And, like, when you click the mouse and operating system, events gets generated that says, like, I'm asking a card right?
That code needs to be plugged directly into the carnal because, like, clicking the mouse needs to be a fast operation, right? It needs to be the case, like, because we're playing StarCraft or whatever then when we click the mouse that then happens very fast. Right? It can't be the case that that has to go through a bunch of layers of, like, special protection to make sure it's safe. Right?
The problem with this is that 3rd parties who are writing.
Uh, driver code, or not trustworthy, they often make mistakes, right? Because it's just code and it's easy to make it, uh, make a mistake in it. Right? And it's easy to, in particular ignore the possibility that things can go wrong right often when writing something like mouse, click code, you think, okay, I click the mouse and then it works right? But what if the operating system says like, no.
I'm busy right now, because I've been preempted by something else. Right? What are the operating system? Just stalls for a 2nd, like, it does happen right?
And in practice, what happens, then if your mouse code doesn't handle that case, is that, uh, you get the, the famous old Windows thing of blue screen of death.
The blue screen of death, basically on most of the time and Windows back in the day. Like, back in, like, 2000 ourselves happened all day, all the time extremely annoying. And 2, most of the time it was caused by 3rd party drivers.
Being just wrong, because the state of the world was that most drivers were incorrect, right? That's still kind of true. But driver verifier helped us move in the right direction. Right. Um, so, uh, it's a, you know, tool included in Microsoft Windows that replaces the default operating system. Subroutines.
Right and it monitors and stresses drivers. Right? So its job is just to pretend to fail. So, the driver is, uh.
Like that don't do the right thing get caught, not doing the right thing. Right? And so the key idea here basically is that Microsoft will go off and.
Like, run in the background of this thing on anything that you plug in when you're not using your computer. Right? So, if you, if you plugged like a mouse into your computer, like at 1st, it's just running untrusted. Right? It it hasn't been verified yet. But then when you like.
Turn your computer off for a moment, right? Or, you know, like, you leave it to sleep. Microsoft will be like, okay, the operating system spins this up and just test that mouse and checks that its driver is. Okay. And if it's not, it sends a report back to Microsoft that says, like, go call up this company. They suck.
Right. That's basically the technique, right? And this was deployed and I believe that, like, maybe around 9092001 something of this. Right? Um.
And it works, it worked very well, right? Yeah. So it simulates a bunch of this, these kinds of problems.
Yep, yep. Okay there we go. Uh, it's included in every version of Windows since Windows 2000. um, Microsoft claims still over 70% of blue screen of death are caused by, uh, 3rd party drivers, even in the modern world. Right? Um, rubric hazards. The number was higher. Uh, in general, um, Microsoft is actually pretty, uh.
Tight lipped about how often a blue screen of death occur. So, like, we only really have an anecdotal data here, but like.
Uh, I have heard from people who would know that.
Driver verifier is 1 of the reasons for this, there's 1 other important project, which is the quote, unquote, static driver verifier, which is a model Checker. Uh, that also was extremely instrumental that happened around the same time between those 2. Uh.
The number of blue screen of death apparently dropped really precipitously, uh, around the, like, between, like, 2001 and like, 2008.
Again, Microsoft doesn't publish numbers about this, so I can't.
Give you any, but I search.
That, uh, this technique and slam were really important, right?
Okay, any questions about this driver verify everything.
Still included in any copy of Windows that you have, because it works.
All right, so that's, uh, that's all the examples I wanted to go over. I'm just going to briefly summarize and then give a few announcements. I'm sorry that I have run.
Quite longer than I had expected to, but there's a lot of fun things to talk about in dynamic analysis. So, 1st dynamic analysis runs an instrument program and a control banner to collect information that we can analyze to learn about a property of interest. These 5 things are the core ingredients of dynamic analysis, right?
When thinking about dynamic analysis, you should be thinking about these 5 things, right?
Testing itself is a dynamic analysis, but so are most of the other things that we've discussed in this class so far? Right and so the stuff that we're talking about today is, like, generally applicable to most of the things that we have covered in class.
Instrumentation like, you can do it at the source code level or a binary writing, and there are other things as well. Right?
You should have at least a decent understanding of the details there. Right?
And dynamic analysis, limitations include overhead, right efficiency, but also they can have false positives and false negatives.
Uh, it is possible to create complete, dynamic analyses, but most are not.
And finally, lots of companies use dynamic analysis, especially for hard to test bugs, like concurrency problems or memory leaks or drivers.
Right places where it's difficult to just sit down and do the testing yourself dynamic analysis are often the next the next thing that people reach for, and before they reach for static analysis.
All right, any questions on dynamic analysis before I do a few brief announcements.
All right, brief announcements 1st recall that there is an exam during the next class there's no class next week at the spring break after that we will have an exam. So you'll recall you'll be permitted to bring 1 letter sized piece of paper with handwritten notes, double sided before next time that you're here, you should prepare that. Right? Um, recall also that you will have to turn that in.
Along with your exam, we will compare your handwriting and we might compare it against other people's notes sheets. So don't copy each others, make your own right? Um, for the exam itself. Um, so, uh, what, we're actually what we're going to do at 1st, I will do a lecture, right? There'll be.
I will actually time it ahead of time so that it does not go beyond 7, but it will probably under 145. that's what I'm going to aim for. Right?
Then we'll have a 45 minutes, a half hour review session, right before the exam starts, right? And then we'll have the exam for an hour and a half starting at exactly. At 7:30:right.
Why are we using this schedule? Um, the reason is that, uh.
Uh, the exam is during Ramadan and so between daylight savings and when, like, sunset, getting later since that's at about 7:10:on that day. So I want to delay the exam until after that, because some of, uh.
The people in the class are going to want to break the fast right?
Okay, if this exam schedule is a problem for you please let me know as soon as possible. So that we can try to find an alternative arrangement, I will design the exam to be done in about 60 minutes. But I'm going to give you the full 90 minutes, because I'm very bad at designing exams and it will probably be too long. So, sorry in advance.
All right, um, finally, um, for the rest of today's class, what little is left.
Um, and I will hang around to answer any questions that you have on homework. 6. all right. Um.
I am concerned that there have been very few submissions. Right? So, if you are not, like, making real progress by now, you are going to.
Probably struggle a bit because the 1st, time you submit a grade scope. It is unlikely. You'll get a high score. Right? Homework 6, especially requires you to be a bit creative with choosing good mutation operators and doing some experiments on your own to figure out, like, what mutations are actually useful and what aren't right.
And I am concerned that no, 1 is scoring badly yet, because it means that you may not have time to score. Well, all right. So please get started on that. As soon as possible. It is due end of day anywhere on earth on Wednesday, next week, right? Based on the class vote that was taken at the beginning of the semester.
All right any questions on any of this.
Yes, by I mean, like 8 and a half by 11 metrics.
Like, a princess? Yeah, Princess sized paper. Yeah.
Right.
All right, um, if that's it, uh, then, uh, I will see you all, uh, next week.
No, not next week the week after next yeah.
So, sorry, I can't hear you.
Okay, and.