This is the second of two taped telephone interviews conducted with Lott for my Mother Jones article. The first interview covered a wider range of general questions, many of them background for the story. This interview, conducted on August 19, 2003, focuses in on issues surrounding coding errors and clustering.
Chris Mooney: So, you know, I went back and looked at the tables, as you suggested that I do. And I have some questions. You know, your corrected table 3a, which I downloaded — your corrected version, and then there’s an Ayres and Donohue corrected version, have the same values for decreases in murder, rape, and robbery. I’ve got these in front of me. But your values are statistically significant and theirs are not. Why is that?
John Lott: Well, one nice thing about putting things up on the web, is, even if you can’t figure out, I know you don’t have a statistics background, academics who look at it can immediately see how things are calculated, cause in the do file that we have there, you know the regression list that we have there, you can see what statements we use. And we do virtually the same thing that Ayres and Donohue did, at least in the body of their work, their first paper. You know, I haven’t gone through in depth to see what they’ve done in this reply. But the only thing that we do differently from what they did in all their results was, we include a robust statement, which essentially deals, I won’t bore you with the details, but it essentially deals with how the variance can vary across data. And so, what that almost always does is, it causes the results to be somewhat less statistically significant actually than would otherwise be the case.
From what you’re telling me, and what I remember seeing, cause it’s been a while since I looked at their thing, I assume that they’ve done something different than what they had done in their original paper to calculate the standard errors. And, but our stuff, and you can go and you can, again, if you can get a third party academic to look at this, can go and verify whether or not the type of statistical test that we’re doing in those corrections is the same as what Ayres and Donohue claim that they’re doing in their original paper. Along with adding a robust statement, which makes it somewhat more — less likely to be statistically significant.
Now as I remember, they never did the Poisson regressions, right?
CM: I don’t know. But….
JL: And those results were also reported, and that has almost no difference. In fact, if anything, those results are, if you look at the coefficient [unintelligible] there, I think are almost somewhat stronger than they were before. But I won’t go into that in depth.
CM: But I have another follow up here. I mean, it seems that — I looked at the do file, and it seems that you changed the calculation, because in the response — well, you took your name off it — in the response you adjusted for clustering at the state level, but in the do file, that clustering isn’t there.
JL: It’s the same as what Ayres and Donohue did. Did they change what they did across the papers?
CM: I’m not…
JL: If they redid our results using their statistical method, they should have gotten something that should have been fairly close to what’s reported in those tables.
CM: Which tables?
JL: In the corrected tables. If they didn’t change what they did, then — if they didn’t change what they had done in their original paper, then they should get very similar results.
CM: But all they did — I’m talking about their reply. All they did was correct your coding errors, and then re-run it…..
JL: Yeah, my own guess is that they did more than that.
JL: Look, what I’m telling you, we have our data up there on the page, we have our do files up there on the page, and, anybody — in fact I was just talking to an academic from Washington University in St. Louis, who had also been trying to get a hold of me — you know, anybody can go and look at that.
CM: But I have looked at it. And listen, I mean, the table, your table 3A, it reads in the note, “Robust standard errors are shown in parentheses and clustering is assumed by state.” But then when you look at the do file, there’s no, the word cluster is nowhere to be found.
JL: As I remember, what was done there was to try to replicate it along the lines of what Ayres and Donohue had said that they should be doing for their regressions. And that’s what ‘s shown there.
CM: Right, but what I’m saying is, this table, this table reports….
JL: I’ve repeated it, I really don’t want to get into this in depth. You said it would be short, I’m giving you an answer, you either take it or you don’t take the answer, but…
CM: Well, it seems like this table is false then, because it says….
JL: It’s not false.
CM: [continuing]…there was clustering but the do file says there’s no clustering. How can that be? I mean, what’s…..
JL: Does the do file, what does the do file say? Does the do file….
CM: I just opened it and looked at it. I mean, I did a search….
JL: Does the do file — the tables, that are the corrected tables, what do they say on it?
CM: The corrected table, in the note?
JL: The corrected table on the website, what does it say?
CM: I’m reading the note to you. “Robust standard errors are shown in parentheses, and clustering is assumed by state.” That’s in the note. I’m saying, when I open the do file, there’s no clustering.
JL: Okay, well then, I don’t know, I’ll have to look at it, it shouldn’t say that in the table on the website. It should just say robust standard errors.
CM: Okay, but I mean, but why shouldn’t it, because in the response to Donohue and Ayres there was clustering, so why would the clustering leave?
JL: It was done to be the same as what they had done.
CM: But I’m telling you, in the response….
JL: Look, I’m really, I’m really….
CM: I’ve looked into this in a lot of detail. In the response….
CM: [continuing]…to Donohue and Ayres, you….
CM: [continuing]…it says that the data’s clustered. Now, you say the data’s still clustered, but your do file doesn’t say that. So it seems like you’ve taken the clustering out, and that might explain the discrepancy.
JL: Look, the statement should be, should be that robust standard errors were calculated. Okay? And that’s what I believe was in the do file, and it was set up so it would be the same as what Ayres and Donohue had done, except for the robust errors, cause they didn’t include that calculation in their paper.
CM: So if you didn’t cluster then, then no wonder you didn’t get — you got statistically…..
JL: No, I don’t think that’s right.
CM: Well, sure. That’s a big difference.
JL: No, I don’t think that’s right.
CM: I mean, isn’t it standard practice to cluster in this kind of analysis?
JL: Did they cluster in their paper? If you look at the law review paper
CM: In testing your results, yeah.
JL: No, no, no. In doing their original paper, the original paper they published in the Stanford Law Review….
CM: I’m sorry, I hate to stop you, but the original paper is completely not germane. I mean, we’re talking about…..
JL: It’s completely germane, cause they are running, they’re running panel data, and the question is, they’re running it with tests that they think are correct, and so what this has done is to try to mimic the types of tests that they have.
CM: I just don’t see why that’s germane. We’re talking about correcting the coding errors. You’ve told me there were minor coding errors in the response. And then you told me there’s a corrected table…..
JL: [Unintelligible] And I’ve just answered it. Okay, you said there was another question. Do you have another question?
CM: But you haven’t really answered it. Because I’m saying, why is the corrected table, why does it say that it’s clustered by state, but actually is not clustered by state?
JL: I’ll look at the web page when I get home, all right? I can’t look at it right now. But as I just told you, it shouldn’t have said that. And the, you know, and if it said that, then that was a mistake. But….
CM: Well, I’ve got it right here. I mean, it’s not a question.
JL: Okay. And, I’m trying to be honest with you right now….
JL: …in terms of — look.
CM: I know you’re trying to be, but I mean, this is a discrepancy that could explain why….
JL: I don’t think it does.
CM: Well, why not? It makes a big difference in the result.
JL: Look, Chris, I’m really, I’m not up to badgering right now. I’ve tried to answer your question.
CM: I think that you changed this. I think that you changed this, knowing that that would…..
JL: Alright, look, look….
CM: Tell me you didn’t.
JL: If you have another question, I’ll be happy to answer it. I’ve tried telling you that if it says that then that’s a typo there, but it’s not, it’s not — there’s, but we’ve tried to, I’ve tried to make it clear to you, I thought that the tables didn’t have that in there, and if it does, then…but you know, you’ve got the do file, people can go and run it again, and they should be similar to the types of statistical tests that Ayres and Donohue used in their paper. And it’s the same type of panel data, so if they think, if the regression results should be run with a certain type of statistical test, then if anything, these are more restrictive than what they have in their paper.
CM: Okay, I still just don’t understand this. I mean, it seems pretty clear that….
JL: Okay, I’ve tried….
CM: [continuing]…you’ve changed the test — you’ve changed the test, and that’s why you get statistically significant results and they don’t.
JL: And what I’m saying is that they would have used different tests — look, I’ve said it. You have it on tape, multiple times, me saying this. And — did you have another question?
CM: No. I mean, I think that this — I think that this goes to the heart of the whole debate over whether the coding errors matter or not.
JL: Alright, well, then I guess you can ask why they used certain tests in their first paper.
CM: But I’m asking why you used a cluster in your second paper and then not in your corrected. That’s my question.
JL: And what I’m telling you is if it does say that, and I’ll go and I’ll check when I get home…
JL: What did I tell you?
CM: You’ll go and check….
JL: No, no, no, before that. I’ve already said this to you before.
CM: It shouldn’t say that.
CM: Okay, well, you know, if it does say that — I mean, you can check for yourself, it’s right there — but if it does say that, then I guess you’re going to correct this?
JL: Well I mean, if you want to use the same types of tests that Ayres and Donohue did, which is what I believe, that’s what’s shown in those corrected tables. Then you can see the types of tests that they believe should be used, and what the results that you get from that are.
CM: Yeah, I mean….
JL: Let me ask you this, did they go — have you asked them if they go and use the types of tests that they advocate [inaudible] and use them on that data, what would it look like. Have you asked them that?
CM: On what data?
JL: On the so-called data with errors. Have you asked them that?
CM: Yeah. They say that by removing this clustering, you…
JL: No no, I’m asking you. They have regressions in their paper.
CM: Of course, lots.
JL: And you asked them if they used the types of statistical tests that they use in that paper, all right, on this data, would they get a statistically significant drop in violent crime?
CM: I haven’t asked them that particular question.
JL: Well, you should ask them that, because that seems like it’s the core of the issue.
CM: I think this is the core of the issue. I think….
JL: Alright, well look, you have an agenda on this. I’ve tried answering this question for you, okay, and I’ve tried….
CM: Well, just answer me this. Does this kind of analysis, do you need clustering in order to get the right statistical significance, or do you not need clustering?
JL: Ayres and Donohue didn’t use clustering.
CM: I’m asking if you need clustering. Do you need it?
JL: I’m saying, you should ask Ayres and Donohue why they didn’t use clustering.
CM: When testing your results, they did!
JL: No no, when they presented all their regressions, and the ones that they go and claim show an increase — and they make an error that they don’t address at all, in terms of not including both the dummy and the trends together at the same time in trying to figure out the net effect, when they’re just even recording just the effect from the dummies and their hybrid specifications — what did they use? Did they use clustering or not?
CM: I don’t see why that’s germane. I just want to know your opinion. Does this kind of test, in Table 3A, does it need to have clustering? You’re an expert. Should it have clustering?
JL: Right. I’ve answered the question.
CM: Is it a yes or a no?
JL: I’ve answered the question for you lots of times.
CM: You’ve never answered this one.
JL: Look, I’m not going to go into this any more. If you have another question, I’ll answer it.
CM: It’s very simple. I mean, do you not know whether they should have clustering? You’re the Ph.D. here, and here I am, I’ve learned this overnight, and you’re not answering my question.
JL: Well, apparently you haven’t learned it, you haven’t learned it as well as you might have liked to have learned it.
CM: I think I understand it well enough that either you use clustering or you don’t use clustering, because there’s fifty states, and there’s only fifty tests, and so if you don’t….
JL: Right, but here’s the deal. Here’s the — you want me to explain clustering to you?
CM: I want you to tell me whether you need to use clustering!
JL: I’m going to answer your question!
JL: Here’s the issue of whether you need to use clustering or not. The issue is how the errors are correlated across observations within a state. You’re changing the law in most cases, but not all, at the state level. And so, now, one way you can go — the debate, in the original literature on clustering, was using data that didn’t have fixed effects for each one of the local jurisdictions. So let’s say you have a state, and you have county level data. The original debate, and the types of adjustments that are done there, in the original papers that are talked about in this literature, were dealing with data that didn’t have these county fixed effects. The country fixed effects by themselves pick up the differences in this error across county observations. When you go and you go and do clustering over and above that, you’re in some sense double counting, because the fixed effects are already picking up this error that’s correlated across counties. And when you go and do clustering over and above that, you’re kind of double adjusting for it.
CM: So why did you do clustering in the original response, then?
JL: You can do it if you want to be kind of like extra super double careful in terms of doing it. And in any case, it’s biased towards finding something that’s there. And so the question is, what type of bias [unintelligible] do you want?
CM: Okay, and so then — but how is it fair to put it in and then take it out? When the whole point is, you’re trying to….
JL: I told you, I told you, it shouldn’t have been in there.
CM: It shouldn’t have been in your response, published in the Stanford law journal?
JL: It isn’t my response, first of all.
CM: Right, but you authored it, and you’re defending it.
JL: Yeah, I’m happy to defend it, and I’m happy to say that, if you look at the data, and anybody who goes and brings it down and looks at it, and does similar types of tests —
CM: I’m looking at the data.
JL: Okay, look, Chris, you’re not being objective on this. All I’m saying is, if you do the test….
CM: I found a pretty big discrepancy here!
JL: Alright, look, if you believe that, then that’s fine. Okay? I’m just telling you….
CM: You haven’t convinced me of any reason why I wouldn’t believe it.
JL: I just gave you an explanation for when you use clustering, okay….
CM: Right, but you also used it and didn’t use it, which doesn’t make any sense to me, and it can make a difference in the results. I have this on good authority from econometricians that it would make a difference in the results — and that in fact, it’s standard to use it in this kind of test.
JL: And it depends, and there’s an issue of whether — okay, well then why didn’t Ayres and Donohue use it in their paper?
CM: I don’t know. I know that they used it to test your results.
JL: Why don’t you ask them? Okay, look. All I’m saying is, if you’re making these types of categorical charges of what should or should not be used, and these people, Ayres and Donohue, don’t use it in any other regression that they report in their paper — if you can find one other regression where they do that, then I’ll be interested in hearing that. But they don’t use that test on any regressions that they have, they only do it, you’re telling me, on the replicated stuff.
And look, besides that, I’ll point to something else that’s in the paper, and that is the Poisson regression. One of the things that we tried to point out — as far as I know, they didn’t try to replicate these at all in their stuff, and there’s a reason why they didn’t try to do that — is that, when you go and use OLS, ordinary least squares, on this county level data, you bias the results towards not finding a result. And there’s a simple reason why you’re doing that, and that is, because you have, you essentially have this truncation, cause the values can’t go below zero. You really need treat that as what we call count data, and, I can’t remember whether it’s Table 6, or whatever, that you have there…..
CM: Table 3A is the only one I’m really interested in.
JL: And why is that?
CM: Because that’s the one that they corrected.
JL: Right, and why didn’t they correct any of the other ones that we said were the main result?
CM: This is an example. I mean….
JL: No no no no. Look, we go and say that the Poisson is the most believable estimate that you can have in the paper, okay? They don’t look at that, and besides that, you’re telling me that Ayres and Donohue are convinced that you need to do clustering, and yet none of their regressions….
CM: I didn’t tell you that they were convinced. I said that they’re testing you after you did clustering.
JL: And I’m just asking you, if clustering is necessary, then I’d like to see you mention in your story that not one of their regressions that they did on their own had clustering.
CM: I’ll have to look into that. But I guess it depends on the kind of test….
JL: Well, what happens if you find out that not one, not one single regression that they did in their paper had clustering? What would be your reaction?
CM: I would have to find out whether they needed them.
JL: Well why, it’s the same dataset, right?
CM: No, this is your new data…
JL: No no no, but I’m saying, the same type of data, it’s county level panel data.
CM: I guess I…..
JL: I will be interested to see whether or not you report that in your story, because if they go and are making a big deal about this, and haven’t used it in even one single regression table that they point out in their original paper, that they’ve done, then I’d like to hear the explanation.
CM: Okay. And maybe I can ask them that, but I think it’s more interesting to find out why you first used it and then didn’t use it.
JL: Well, it’s the same thing for them. They didn’t use it, and now they did.
CM: They’re testing you! They’re testing your model.
JL: Right, but if they’re testing me, and they want to use the regressions that they think, the statistical test that they think is the most valid test, then why don’t they use that on that?
CM: They’re not using the most valid test, they’re using the standard that you set.
JL: Okay. Right. And…
CM: That’s your test, that’s why they used it to test you. I mean, they corrected the errors and ran everything else the same. That’s the whole point. They corrected the coding errors, they ran everything else the same, including the clustering. Then they said your results were no longer significant. Then you went back, took out the clustering, ran the corrected results, and said they were significant. That’s what happened. Right?
JL: The Poisson regression results…
CM: That’s not what I’m talking about.
JL: And why?
CM: I’m talking about Table 3A. This is at the center of the whole discussion. Table 3A. You took out the clustering in Table 3A.
JL: We’ll talk more if you turn off the tape recorder. Otherwise, I’m not going to talk any more right now. I have to go in any case. You said it would be five minutes.
CM: You can go any time you want. I can’t force you to keep talking to me.
JL: Look, I’ve tried to be nice to you, and — you know, it’s just — anyway. It will be interesting to see their explanation for why they used it in all the other stuff. And I don’t think it makes a difference, people can go and look at the data….
CM: Which is exactly what I’ve done.
JL: Okay, and who’s looked at it?
CM: I have looked at it.
JL: No no no, but who? You have an academic. Some third party. Who’s looked at it?
CM: I have been going over this by e-mail with a number of people. I asked Whitley about it, I asked Plassmann about it by e-mail, and I asked Donohue about it by e-mail. And I’ve been sort of figuring out as I go along. It was actually Whitley who told me to look at the regression file.
JL: I told you yesterday to look at it.
CM: Well, he told me to look at it, and find out what kind of thing — and sure enough, I mean, I found out that there was clustering in it.
JL: All right….[inaudible] appreciate your time. If there’s something else you’d like to ask, I’ll try to answer it.
CM: Okay, thanks very much.