Vox Overstates its Own Reporting

You keep using that word, but I’m not sure you know what it means….

Vox dot com has this article they seem to be recycling once a week or more lately, in which it claims in its headline and tagline that after Australia banned and confiscated certain weapons in 1996, homicides “plummeted.”  It was a “smashing success,” we’re told.

But something seems awfully fishy about the evidence offered in support….  It just seemed strange to me, so I decided to look deeper into the article and see what was up.

What actually happened?

On this day in 1996, 28-year-old Martin Bryant begins a killing spree that ends in the deaths of 35 men, women and children in the quiet town of Port Arthur in Tasmania, Australia.

Bryant, who is believed to have an extremely low IQ and may be mentally handicapped, began the day by killing an elderly couple who were the owners of Port Arthur’s Seascape guesthouse. Some theorize that the killings were Bryant’s retaliation for the owners refusing to sell his father the guesthouse. Bryant’s father later committed suicide, an action Bryant is said to have blamed on his depression over not being able to buy the property.

After having lunch on the deck of the Broad Arrow Cafe, located at the site of the historic Port Arthur prison colony, a tourist destination, Bryant entered the restaurant, removed an automatic rifle from his bag, and began shooting. After killing 22 people in rapid succession, Bryant left the restaurant for the parking lot, where he continued his shooting spree, killing the drivers of two tour buses, some of their passengers and a mother and her two small children, among others.

History.com.

In response, according to Vox citing an op-ed he wrote, “I knew that I had to use the authority of my office to curb the possession and use of the type of weapons that killed 35 innocent people,” [then-Australian Prime Minister John] Howard wrote in a 2013 op-ed for the New York Times.

Good for his word, Howard soon built a coalition around a reform of Australia’s gun law so radical that it ended up banning many categories of firearms outright, and years later the number of Australian households reporting that they owned at least one weapon has fallen by 75%.  The Cut.

And indeed, Australia’s homicide rate is lower after the ban than before, and Australia hasn’t had a “mass shooting” since (which officially it defines as 5 or more killed in one event, not including the shooter).

To its credit, Vox disclaims that causation is difficult to attribute, but under such a hyperbolic headline, where it claims “[T]there’s good reason to believe the NFA, especially the buyback provisions, mattered a great deal in contributing to those declines,” it’s fairly obvious that Vox means to persuade us that the evidence is persuasive, if not compelling.

But is it?

What does “Plummeted” mean, anyway?

According to crime data made public by the Australian government, it’s hard to justify using this verb.  Full disclosure, I’m no statistician, but anyone with access to the internet can go to an online calculator and run a regression.  So let’s do it.

It’s alleged that the Australian gun confiscation caused homicides to “plummet.”  In the 8 years, inclusive, prior to the ban’s adoption, the average rate of change in the rate of reported homicides involving firearms was -1.57 per year.  In the 8 years inclusive, following, it was -2.37.  That looks pretty good!

(I’m going inclusive because I wanted to make sure to catch any “plummet” at the time of the ban.  I’m not writing for academic publication or trying to live up to any external standard; just trying to make sure I’m looking in the direction of reality.)

And the slope of the regression line across the years after is steeper than the slope across the years prior, -2.84 versus -.64 — and the slope across the whole set is -1.91, so the slope of the whole trend is more similar to the slope after the ban than the slope prior than the ban is.  If 1993 and 1995 are outliers, however, but not 2001 and 2002, the effect is much smaller — assigning the average for the period before in those places yields a slope before of -.84; substituting the average for the years after in 2001 and 2002 reduces the slope to -1.78 — smaller than the slope across the whole unmanipulated set.

And on the face of it, without the date line labeled, it’s hard to see the inception of the ban:

And if we graph the homicide rate by all means across the same range, the slope is even harder to see (though it is negative 0.02):

In fact, ’01-’02 saw as large an increase (1.6 to 1.9 per 100k) as as ’96-’97 saw a decrease (2 to 1.7).

And that’s the problem: the data are so few and the range is so small (which is to say that Australia’s firearms homicide rate was so low and steady throughout the period of interest) that these data can easily be made to say different things.

Indeed, the difference between the average rate for the whole set (63) and the period before the ban (70) is 7; the difference between the average rate for the whole set and the period after the ban is also 7 — consistent with a secular, more or less linear decline.  And don’t forget that this whole conversation revolves around an exceptionally lethal attack that by itself suffices to skew the statistics.

Hardly slam-dunk evidence of a “plummet.”  Indeed, according to Crime Statistics Australia (CSA), it doesn’t look like there is any inflection at all around 1996 for homicides regardless of weapon; possibly relaxed deterrence allowed for an increase in homicide by other means that offset, to some degree, any gains to be realized from the gun ban.

If you squint really, really hard … you still can’t see anything.

What does Vox say Vox’s reporting says?

Well, after assuming throughout much of the article that homicides “plummeted,” Vox, to its credit, does quote an actual study about whether or not the data support a conclusion that the Australian weapons ban and buy-back were effective: We can’t know.

Pinning down exactly how much the NFA contributed is harder. One study concluded that buying back 3,500 guns per 100,000 people correlated with up to a 50 percent drop in firearm homicides. But as my colleague Dylan Matthews points out, the results were not statistically significant because Australia has a pretty low number of murders already.  (From the Vox article).

But this appears to be the strongest evidence offered in favor of the ban having “worked,” and the only evidence expressly promoted as such, so there’s possibly a lot of post hoc, ergo propter hoc going on here.

Nevertheless, Vox insists that

There is also this: 1996 and 1997, the two years in which the NFA was actually implemented, saw the largest percentage declines in the homicide rate in any two-year period in Australia between 1915 and 2004.

But according to the official data as reported at CSA, the homicide rates per hundred thousand in the reporting years ended 1996-1999 were 1.6, 1.6, 1.6, and 1.7, so it’s not indisputable that that claim is justified.

Or did they mean the homicides-by-firearm rate in the two years following the ban?  67, 64, 63, according to CSA, or a 6% reduction.  1990 to 1992: 75, 68, 63, or 16%.  No contest.

Too, 1995 was an outlier year featuring a sad 80 homicides; the year prior had 60, and the year after had 64, so 1995 looks a lot like statistical noise in the CSA dataset.

Furthermore, in a model built by the Australian Bureau of Statistics normed to a standardized 1991 population, the rate of firearm-mediated homicide in 1995 was just 67 and the lowest of any prior three-year average, so again the deviation, if any, from the trend can be decided on the basis of which dataset you prefer.

But the trend was always negative going back to at least 1980, so the contribution, if any, of the gun ban on the total homicide rate or the firearms homicide rate is harder to detect than we are led to believe.

Why bother?

There is, however, unambiguously statistically significant evidence of a real benefit that can be decisively attributed to the gun ban — a sharp and sustained reduction in the suicide rate.  This would seem, by itself, to justify a policy response, so it isn’t very obvious to me why we have to go through all this statistical legerdemain to find out whether we’re being told a defensible truth — if anything had “plummeted,” I shouldn’t have been able to make any of the arguments above, because in that case there would have been a sharply discontinuous evolution in 1996.

And there just isn’t.

So the headline and tagline are deceptive.

And that makes me sad, because on the whole, I like Vox, and I think its heart’s in the right place.

Why do I care?

I care because these data are expressly being marketed in favor of a specified policy action but do not in fact prove what they purport.  Readers who don’t have the time or interest to deconstruct the argument as I have tried to do may come away thinking “Aha!  Let’s just ban all the guns.  Problem solved.”

But if, as I speculate, the simple criminalization of firearms may remove disincentives to violence by other means — that is, if the public possession of the means of self-defense actually exerts a deterrence effect — we are likely to be bewildered after any speculative gun ban when just as many people get dead, just not shot.  In order for the homicide rate to remain so close to its averages after the ban, logically there has to have been method substitution and/or a more liberal use of violence.  Or it’s just statistical noise — we are after all dealing with effects on order .1 per 100,000 population per year.  And meanwhile, it has in recent years cost a metric ton of political capital to fail to get anything done, so it’ll be doubly costly if we finally do and it doesn’t have the desired effect of keeping more people alive.  And indeed there is scant evidence that prior weapons bans have had detectable effects in America.

And notice:  I have advocated no specific policy, and I don’t pretend to know what the right number of firearms in American society is at this moment in history — but it’s probably not zero.

But these data so clearly don’t say what they are purported to say that Australia’s Ambassador to the United States wishes we’d stop saying that they did.

That said, I’m certainly not hostile to “common sense gun reforms” if they can lessen the worst excesses of mass-casualty violence, or tend to help suicides stay alive until they can get the help they need to live fulfilling lives; we just can’t know until after we run the experiment, and Australia’s data do not provide compelling evidence either way, despite Vox’s hyperboles.

Advertisement