Wednesday, November 25, 2009

CRU Climate Hack: Hackers Took Two Whacks At Data?

I've sent an email to Mr. Paul Hudson about this, but he hasn't responded yet. The reason I emailed him is he says:

The controversy surrounding the global warming scandal today deepened after a BBC correspondent admitted he was sent the leaked emails more than a month before they were made public.

In his BBC blog two days ago, Hudson said: 'I was forwarded the chain of emails on the 12th October, which are comments from some of the world's leading climate scientists written as a direct result of my article "Whatever Happened To Global Warming".

But here's Gavin Schmidt from Real Climate:

Nonetheless, these emails (a presumably careful selection of (possibly edited?) correspondence dating back to 1996 and as recently as Nov 12) are being widely circulated, and therefore require some comment. Some of them involve people here (and the archive includes the first RealClimate email we ever sent out to colleagues) and include discussions we’ve had with the CRU folk on topics related to the surface temperature record and some paleo-related issues, mainly to ensure that posting were accurate.

Since the dates are exactly one month apart, I've asked Mr. Hudson if we weren't dealing with a simple typo here. If not, it looks like our hackers were dicking around with CRU data for over a month without being detected, and circulated at least two packages of emails (or more precisely one package of emails and one honking big .zip file.). So why did they wait a month before the big reveal?

PLUS!!!! Exclusive E-Mail from Hugh Miller, the guy behind an elegant chaos, a searchable index of the CRU emails and files. I asked him: are you the guy behind [an elegant chaos]?

Yes, unfortunately for no better reason than I had the space to post it and an hour to generate a simple text search - there's no hidden agenda, and I just downloaded the ZIP from the same place as everyone else.

It was originally really just for me and a couple of others to use as a comparison for any stories that quoted from the messages, there was no way I was going to run through text files every time I seen a quote. But the internet has it's own way of doing things. Unlikely to be on my server beyond tonight, it's just an email account really and it wont cope with the bandwidth.

I've had the usual flurry of emails from people insisting I extend it and do more with it, but that's unlikely at the moment.

Checked your blog, as I'm sure you've guessed I fall into the skeptical camp, but you're obviously more than welcome to use the search to make sure that online quotes are accurate and also not cropped to hide phrasing. One of my biggest groans on any website is the continual point scoring and pickiness of the commenters below (not saying I'm not guilty of that myself).

So if someone says one of these emails says X, Y or Z, you can confirm that right away. Since these things were out in the wild from Thursday it makes sense to me that they should be available in a user-friendly format. As I've said to others, I have no intention of allowing comments below them or to make any comments on them myself.

There are a few others online now too, also containing the 'data' files that came with the zip, since they exist I have no plans to add them to that search, I don't really have the time and I'd imagine that someone, somewhere will create one or adapt one with lots of extra features.

Anyway, I hope this gives you a bit of an overview of the motivation behind it. But feel free to email me with your views (bear in mind I'm in the UK so there may be a delay between replies).

Cheers,
H

Tim Lambert thinks its a misunderstanding.

11 comments:

Mike514 said...

On Nov 20th, you wrote:

Probably the worst bit, from an optics point of view (certainly its the thing being waved around by the denialist crowd) is a 10 year old email from Phil Jones

You went on to say that the word "trick" was causing the biggest ruckus (the "worst bit" as you put it), when it really shouldn't have.

But today, you continue to write about the CRU climate hack, and in fact have spent the last week studying it and sending emails.

Are you having doubts since your Nov 20th post? Is a part of you thinking that there's more to this than thinking the word "trick" is the "worst bit?"

It's a genuine question. Just curious.

bigcitylib said...

No. Now I am trying to think of interesting things to write about on the topic, since I am basically reading everything. The "trick" thing is still the worst bit. But I am interested in just how the hack went down, just because as the faux outrage burns itself out I think that's where the story will go.

Ti-Guy said...

as the faux outrage burns itself

I've been calling it "wingnut bukkake."

Tim Lambert said...

The Daily Mail story is rubbish. Hudson was just saying that he was forwarded one of stolen emails a month ago, not that he got them all

Mark Richard Francis said...

"hide the decline" is the buzz now.

They are running through program code and related data sets, proclaiming the fix is in.

Problem is, the reasons for data manipulation have to be consulted within all of the extensive peer-reviewed literature on the topic in order to determine if the manipulations are appropriate. The "hide the decline" reasons were published in peer-review in 1998, apparently.

Do you think the denialists are going to bother to check?

Nope. Just wait They will have t-shirts soon enough.

We are going to be hearing about this one for years.

Frank said...

Climategate in New Zealand?- Temperature Records Manipulated

"Researchers find records adjusted to represent 'warming' when raw data show temperatures have been stable."

With a little help from me friends.-Joe Cocker

Ti-Guy said...

"...With a little help from me friends.-Joe Cocker..."

They talk like this in real life too.

sharonapple88 said...

Climategate in New Zealand?- Temperature Records Manipulated

No, not really.

"Look again at Treadgold’s graph. He makes no distinction between the blue and green lines — he just joins them up. Temps before the mid-20s were recorded at Thorndon, near sea level, but then the recording station moved to Kelburn at 125 m above sea level. It’s pretty basic meteorology that temperatures fall as you move above sea level, so the two stations are not directly comparable. Treadgold affects not to know this… But there’s no need to throw out all the old data, you can apply a correction."

A little bit more on the data.

With a little help from me friends.-Joe Cocker

"With A Little Help From My Friend" was written by John Lennon and Paul McCartney and was in Sgt. Pepper's Lonely Hearts Club Band. Gak, what are they teaching in schools nowadays. :P

Ti-Guy said...

Why are you bothering, Sharon? They're not interested in accuracy or anything resembling the truth.

I just safely and reliably assume that whatever they're posting in support of their position is false.

Of course, it would all be worthwhile if FrankD came back and said "Why thank you for that. I stand corrected." But we know that ain't gonna happen.

sharonapple88 said...

Of course, it would all be worthwhile if FrankD came back and said "Why thank you for that. I stand corrected." But we know that ain't gonna happen.

No that's not likely to happen. Just wanted to get the information out there. It's interesting to see how they get it wrong, or how the information is twisted.

Alex said...

http://it-networks.org/?p=2646
IT-Networks has been updated with more info on this hacking