Monday, December 20, 2010

Exposing the faithful's double standard

On the blog where they are attacking my claim that summers are getting cooler, some there are complaining that my TMax profiles, which shows a drop since 1900's, is nothing of the sort. Without "proper" statistical analysis to prove there is a drop there is no way I can make such a claim. So I decided to do a test on these guys and presented this graph:


I presented this as a nice example of a dropping temperature profile. Seems very clear what is happening here, the trend is down. But, no, not to these people. Without a vigorous statistical test there is no way one can conclude this is dropping.

Ok...

At the beginning of that thread at that site, when I initially showed my temp profiles, these people had no problem in accepting that winter TMin was increasing. That, they said, was well within AGW theory. So for these people this graph of increasing TMin is quite acceptable:


Definite trend as far as they are concerned. But look closely at the two graphs. They are the same! All I did in the top graph was to plot the lowest TMin and flip the image so the numbers were positive. Thus the top graph, which they assumed was another one of my TMax graphs, was in fact a TMin graph flipped.

So this test has exposed some interesting conflicts for these people.

If they claim that TMax is not dropping but has to be "properly" stat tested, then they also MUST apply this to TMin, which they fully accept that AGW would produce.

This shows so clearly their bias against TMax dropping. They will go to any lengths to discredit it. So a dropping TMax must be a real threat to the theory or they would not be so blatantly contradictory and require a double standard. Increasing TMin is OK, no stat test needed because it fits the theory, but oh no, we must apply rigorous stats to prove TMax is dropping.

So this test puts AGW into a real vice. They have two options:

1) TMax dropping is only due to "statistical noise" but not TMin it's increase is caused by AGW.

2) All both TMax and TMin profiles are the result of random variations of cycles within cycles producing the trends.

If they go with #2 then the temperature changes have nothing to do with CO2 emissions and AGW is dead.

If they go with #1 and ONLY TMax's profile is by random noise, but not TMin, then they have a serious conflict and double standard. I have shown that give them a TMin graph mirrored to LOOK like a TMax drop, they will demand a stats test be done. But not when it really shows that TMin is increasing.

Hence if a stats test proves that TMax is indeed dropping and dropping due to natural variation of cycles within cycles, then so too must TMins' increase be from normal variation of random cycles within random cycles. Hence #1 gets rejected and #2 becomes the only option left. CO2 has no effect on temperatures.

I thank the guys at illconcidered for playing the game and being test subjects.

Does the temperature profile follow a normalized curve?

You decide.

This is for the 29 stations for all across Canada. I took each station's baseline and subtracted the TMax for every day to get anomalies. I then created a matrix of how many days are at each interger deviation from the baseline per year. Years as columns, deviation degrees as rows and the intersection the number of days. I then did a percent calculation for each cell. That is, the percent of days for each year for each degree deviation (cleans up years with fewer days due to missing data). I then created a bell graph for each year and made this animation. Keep in mind that the area under the bars adds up to 100% for every year.


Saturday, December 18, 2010

Canadian Heat Waves Part 4E

Count of Days for each Degree Deviation from Baseline.

To see if there is any change over all in temps, not just the extreme ends, requires a counting of the number of days at each degree C deviation from each station's baseline.

This produces a large matrix of deviations from baseline of a range between -40 and 21 in columns, each year in rows and the count of days in each intersecting cell.

The highest number of days should be around the baseline (zero), and looks like this:


A nice normal distribution curve. Each individual year will show the same shape. But does the apex change over the years? Is "global warming" shifting the apex?


Not over all, however, the 1930's and 1940's has the highest shift in the apex.

Is the count changing for each degree above or below the baseline. That is, if summers are indeed cooling then the number of days at temps above the baseline should be dropping. Indeed they are:


Because there are a different number of over all days in the last years (due to missing records), this was done as a percent of each year's count of records. 5C, 6C and 7C anomalies above the baseline are dropping. Fewer days are in each of those ranges. The drop is steeper with 8C, 10C and 12C above baseline:


15C is a nice example of the 1930's being exceptionally hot with more days in this range than any other time in the past 100 years:

But what about the other end, below the baseline, is that changing with time? Doesn't appear to be, -5C, -10C and -15C deviations are flat trends:


So what is happening is since 1920, at least, there are fewer days in the hotter ranges of the year. Not changes in the mid-range temps. This shows a narrowing of the range of swings in temps. And it is not not just the highest of TMax, but all the range of temps of TMax above the average. Summers are not getting as hot today than they did in the 1930s and 1940's.

So if the above baseline is losing days, but the but the below baseline is not changes, where are those days going? They are going into the range very close to the baseline. This is the change in the share the deviation from the baseline is zero:

So over time since the 1920's at least, the bell shape of each year is tending to narrow with fewer days above the baseline, but taller near the baseline. Something to test for.

Thursday, December 16, 2010

Canadian Heat Waves Part 4D

If it's hot one year, what will it be next year?

The heartbeat-like pattern of the hottest TMax of the year begs a question. Can it be predicted what this year's hottest temp will be based on the previous year?

This is station 1111, Chreston, BC yearly highest TMax:

This is one of the few stations that shows an increase in TMax since 1990, but still not near the 1930's and 1940's. But notice the ups, then back downs. Is there at pattern to this? Once one year reaches its highest temps, what is the probability the next year will be higher or lower?

To answer that question I took the highest yearly TMax for each of the 28 stations, and matched it up to the next year's temp, and dumped those records into a new table. 3514 records in total. I then subtracted the main year from the next year (such that if 1900 was 35C and 1901 was 30C the diff would be -5C). Thus a negative number means the following year was cooler, while a positive number means the next year was hotter.

I then added up the total count of each integer TMax, and the total number of each percent lower and got a percent of how likely it was for the following year to be cooler for each TMax.

I then did the same for each percent would be hotter for each TMax to get this plot:




This is what this graph is saying: pick any TMax on the X-Axis, and the blue line is the percent chance the following year will be cooler. While the red line is the percent chance the following year will be hotter. At 42C it's 100% likely the following year will be cooler, but by how much?

At 27C, it's only a 5% chance the next year will be cooler, but a 70% chance it will be hotter, but by how much?

This is the range for each TMax that the following year would be lower:


So if a year is at 40C, there is a 95% chance that the next year will have a cooler maximum TMax of between 2 and 9C (38 t0 31C). So even though there have been cooler TMax in some years below 31C, from 40C there is virtually no chance that the following year will be under 30C for its highest TMax.

This next graph is the reverse. With low TMax for one year, what is the likely range of the increase in the following year?



Combining the two plots, if a year has a TMax of 32C, there is a 40% chance the next year will be higher or lower, and the range would be from 1C to 10C hotter, or from 1C to 5C cooler.

It almost appears that 32C is the mid point. Continuing the pendulum analogy, 32C is the bottom of the swing. Anything either side of that and the tendency is more to move back to 32C than it is to go higher or lower.

Is this median point changing over time? Unfortunately there are not enough data with only 28 stations to evaluate this and plot, so instead this is the matrix of years vs TMax and the number of times in each year for each TMax there are records.

The last test is how often when there is a rise between two years, what is the probability of the the following trend up or down? For this test I took all records where the first year was over 32C (our median) with an increase the following year. I then looked at the number of subsequent years showed a decline, the number of a successive increase and the number of no change. There were 771 records in total, of which 71% showed a drop after a rise. That is, when there was a spike up in temp from one year to the next, the following year after that has a 71% chance of being a drop producing a spike. 23% have a continued increase after the second year and 7% were the same, no change.

Canadian Heat Waves Part 4C

Hottest Day for each year for each station.

This requires a matrix of years as rows, stations as columns and cells as the days that reached the hottest temps. The spreadsheet for that can be downloaded from here.

This is the plot of how many of those hottest days land in each month for each year. If there is any migration of the hottest days moving into other months, the loss of one month should show up as a gain in another. But that is not what is happening. Most of the heat loss, the drop in summer temps, is occurring almost exclusively in July. This means the summers are forming more of a plateau than a crest since 1920.

















Wednesday, December 15, 2010

Canadian Heat Waves Part 4B

Number of days above each degree above anomaly for Part 4:






















Canadian Heat Waves Part 4

Top Ten Percent of Hottest Days
29 stations were selected to check for what is happening collectively with TMax. These stations were selected to have a start year before 1920 and have at least 95% of the records within their date range.
These are those stations:
StnIDStationProvinceStartsEndsRecordsBaseTMax
1111CRESTONBritish Columbia19122006270031.1
1340VAVENBYBritish Columbia19132009212032.4
1835CALMARAlberta19162007196528.2
1957RANFURLYAlberta19051991161329.1
2106LACOMBE CDAAlberta19081993199128.6
2205CALGARY INT'L AAlberta19002009262628.8
2247GLEICHENAlberta19032005229730.2
2364BANFFAlberta19001994236027.2
2409LAKE LOUISEAlberta19152007223026.
2490CAMPSIEAlberta19122009199828.2
2658BEAVERLODGE CDAAlberta19132007195027.3
2973MUENSTERSaskatchewan19042009193030.1
3259SCOTT CDASaskatchewan19112009190330.6
3270WASECASaskatchewan19072009192729.4
3328SASKATOON DIEFENBAKER INT'L ASaskatchewan19002009213631.2
3533RUSSELLManitoba19121990163130.1
3673SPRAGUEManitoba19161997153330.6
380BELLA COOLABritish Columbia19002002185928.3
3938FORT FRANCESOntario19131995134630.6
3943MINE CENTREOntario19152005156230.5
3952DRYDENOntario19141997112929.7
4333OTTAWA CDAOntario19002010166731.5
4547BRUCEFIELDOntario19031993111531.2
5208BERTHIERVILLEQuebec19191994111830.5
5348DRUMMONDVILLEQuebec19142009135631.
588FORT ST JAMESBritish Columbia19002009204427.2
6158FREDERICTON CDANew Brunswick19132000132730.3
6527CHARLOTTETOWN CDAPrince Edward Island19101992181427.2



Each station had their top 10% of high TMax from July, regardless of the year, dropped into a table. Those in turn were baselined for each station from 1961 to 1990, the common range used in the science (for what ever reason they give).

From there the anomalies from those baselines were plotted for the entire dataset.
This is the highest TMax anomaly:

The over all trend is down, fewer hot days across all of Canada. The heat waves peaked in the 1940's, dropping until the 1980s, and flat since. 20 years of no change in heatwaves.

This is the average of the TMax anomaly:


Basically flat over all, with the 1920's to the mid 1940's dominating.

The lowest of the TMax anomalies will be very close to a flat line, no fluctuation because this dataset starts from a minumum value for each location to give the top 10%.

This is the number of days in each year. This had to be more restrictive in the stations, 1920 to 2005 was the cut off range. So no stations with a start year before 1920 and no stations with an end year before 2005. This is only 16 stations that met that criteria. If the entire dataset was used the count would be skewed, the numbers too low, because of missing stations in those outside years.

The trend is also dropping. Fewer days in Canada are above the top 10% temperature now than in the mid 1940's.

No matter how you slice the data, either looking at single stations, or collectively from a range of stations across Canada, the over all trend of TMax is to cooler temps.




Canadian Heat Waves Part 3

Why Combining Stations Creates too much Error

In the right video I made at the top I got into the problems that arise when one station is used to fill in the gaps for another station. This post will expand on that to expose the problem more.



To know if one can use one location's data to fill in the gaps of another, the first thing you have to do is to check how the temps differ between each other where there are matching records.



So see what happens I picked to close stations, Belleville Ontario (4442) and Ottawa (4333) and matched the days together and subtracted Belleville's TMax, TMean and TMin for the summer months (May to Sept). Belleville is just south west of Ottawa, about a 2 hour drive.



The difference in the Tmax between these two locations is:

Ottawa was as much as 11.7C warmer than Belleville on the same day. Belleville was as much as 15C warmer on a different day, with the average difference of -0.18C and a standard deviation of 2.55C.

The difference in the Tmean between these two locations is:

Ottawa was as much as 8.1C warmer than Belleville on the same day. Belleville was as much as 11.7C warmer on a different day, with the average difference of -0.63C and a standard deviation of 2.20C.

The difference in the TMin between these two locations is:

Ottawa was as much as 12.8C warmer than Belleville on the same day. Belleville was as much as 13.9C warmer on a different day, with the average difference of -1.09C and a standard deviation of 2.20C.



Using just one year, 1974 (it has a large swing in the difference), this is what the summer TMax looks like. Even part of the season shows asymmetry in the difference.


This is the same year for TMin:


Notice that Belleville has warmer nights than Ottawa. This is because Belleville is on Lake Ontario, Ottawa isn't. The lake is buffering the temps for Belleville keeping their nights warmer than Ottawa.


So if you wanted to use Ottawa to fill in Belleville's missing data, the difference can be as much as 10C with a 65% range of +/-2.2C, a 97% range of +/-4.4C. But it gets more complicated than that. It appears the difference between the two locations is changing over time.


This is the difference in the same day TMax between the two locations per year. The top red line is when Ottawa is hotter than Belleville, the middle black line is the average difference of TMax and the blue line is when Belleville is hotter than Ottawa. Notice over time Belleville's hotter daytime TMax is dropping, less difference, compared to Ottawa.


This is the same graph showing the difference in the TMin temp, summer nighttime temps. Notice over time the two locations are showing less of a difference in temps between them.

So this begs the question. How does one merge records together from different locations to fill gaps when the degree of differences changes with time? You can't without introducing a high degree of error calculated from the standard deviations of matching records, which would have to change as the years progress.

Bottom line is using station data from different locations to fill in gaps is nothing more than guesses at best, and at worst is creating data ex nihilo.

Monday, December 13, 2010

Canadian Heat Waves Part 2

What do we do with missing records?

To get a proper analysis one needs a complete recordset. But that does not exist with EC data. I've got 570 stations downloaded that have data (that many again that did not return any data from EC).

I ran a scan of all the stations and counted all records that have data for each year for each station into one table. This is the actual count of records per year:


Of the 570 stations only 4 have a complete 100% dataset from 1900-2009 for all of Canada.
The spreadsheet of all 570 stations and their stats can be downloaded here.
Then even for those stations that have narrow start dates, not all of them have complete records between the start and end years.
This plot shows the start date vs percent of records they should have. Each dot is a station. Notice so few have complete records.


Expanded view of the above plot.

The claim is made that one can fill in gaps in one station with records from a near by station, combining the two into one recordset. If that was even possible to get anything meaningful from doing that, this data suggests one can't hope to be able to do that until well into the 20th century, losing the beginning years.

So how does one get a long term trend of combined stations to get anomalies with such scant data? There isn't any way to do it, period. This is why one can only look at specific stations.

Canadian Heat Waves Part 1

This will be a four part blog post on different ways of seeing how heatwaves and the hottest days of the year are trending across Canada. This part will look at the process of getting that analysis.

This blog protests why I will be using only 15 stations from across Canada to get a trend. So let's put that to bed right off.

The surface records at Environment Canada are at best pathetic. The number of stations peaked in the mid 1980's. This is the count of stations' starting years:




This graph shows the same trend, except this time the years are on the Y-Axis with the station count in the X-Axis.

Thus if you want 50 stations in the analysis it would have to start in year 1911 to prevent the lower number of stations before that data skewing the results. If you wanted 100 stations, it would have to start in 1930.

Recall that the warming trend that the AGW proponents says is happening is increasing in temps from 1850 to 1945, then a drop until 1975, then an increase since 1975. So starting the analysis too far from 1900 will not show a proper over all trend. Especially if, as we have seen in some of the stations, the warmest and coldest years were in the mid 1920's

Thus there is a trade off. The earliest is best, but that restricts us to 15 records.

The following analysis will look at 15 stations only, those that start in 1900 and have a good set of records. Not all of the stations that start in 1900 have continuous records, some have gaps, but it is the closest we will get to as long a recordset as possible. These 15 are the earliest with the most number of records (few gaps).

The 15 stations are from a wide range of stations across southern Ontario. Station data from the far north is so little so late that, if one wants to make claims of trends, one must put that in context of the duration of the records.

For example, Sach's Harbour, way up in the NWT doesn't have records before the mid 1950's. Hence any claim that the trend is increasing (it hasn't since 1970) is ONLY from the mid 1950's. With no prior data there is no way anyone can know what the over all trend is from 1900. Half that data range is no measure of trends.

This is the 15 stations used for the rest of these posts:

StnIDStationProvince
2205CALGARY INT'L AAlberta
2364BANFFAlberta
2971MOOSOMINSaskatchewan
3080CHAPLINSaskatchewan
3328SASKATOON DIEFENBAKER INT'L ASaskatchewan
3509MINNEDOSAManitoba
380BELLA COOLABritish Columbia
4333OTTAWA CDAOntario
4442DURHAMOntario
4576LUCKNOWOntario
4862BLOOMFIELDOntario
5168HALIBURTON AOntario
5325BROMEQuebec
588FORT ST JAMESBritish Columbia
735CHILLIWACKBritish Columbia

The other interesting problem that arises is which months to use. The ideal would be to use just July or August. But hot temps occur in other months, and depending on the method used to define a heat wave or hot day will be influenced by the months selected.

This is Station 5325's TMax range for all years:

Red line is the hottest record setting day (highest TMax in 109 years), the blue line is the minimum TMax, and the black is the average TMax. Notice the crest is mid July, but with a few above 30C prior to and after that.

Two of the tests in the next posts will be choosing a threshold temperature by a relative means. One will be the top 10% of hot records, the other being those above the upper second standard deviation. But including TMaxs from month's prior to the crest will lower the threshold.

Using this station as an example, using all the months the top 10% hottest records will return 1582 records (from a total of 15,812). The month distribution of those records looks like this:

However, looking at this from the percent of days in each month we get this:



So the top 10% of the hottest records will have 20% of those records from July. This isn't really the top 10% as it will get dominated by July temps that are actually below the 10% for July.

The other option is to pick just July for the top 10%. Take the lowest temp found in that, and use any days above that low TMax regardless of the month. If we do that we get the lowest temp at 28.3C, so any records above that value regardless of the month. This reduces the records significantly to 164 records. July still dominates at 75% of the records, but it produces a much closer definition of what constitutes a "hot" day for that location.

The other option is to use the top 10% for EACH MONTH so they are all equally weighed. The problem with that is that in May and Sept the top 10% hottest days aren't when compared to mid summer days.

So the best approach to determine what is a "hot" day is to use July as the baseline to find the lowest temp of the top 10% of July's hottest days, and use that lowest threshold to get all those hot days for that location. This process will also be used to get those temps above the upper second standard deviation.

Friday, December 10, 2010

Hottest Day of the year

Station 2973 hottest day of each year. Y Axis is number of days since Jan 1. Some years will have more than one day share the same highest temp.How does AGW force the hottest day of the year further into the year as both the linear trend (dashed line) and the 10 year moving average show?

Number of hot days per month. Evenly spread between July and Aug.
Hottest day of 1952 was April 28. Then what did the rest of the summer look like? Quite flat shaped, not like a normal year with a crest around July:

What would cause this?

Hottest days of each year since 1900:

Year

MaxOfMax Temp

DayNumber

Date

1904

30.6

204

23-Jul-04

1905

28.9

152

02-Jun-05

1906

30.6

187

07-Jul-06

1908

31.7

206

25-Jul-08

1909

33.3

202

22-Jul-09

1910

32.2

176

26-Jun-10

1911

30.6

169

19-Jun-11

1912

32.2

173

22-Jun-12

1913

27.8

160

10-Jun-13

1913

27.8

161

11-Jun-13

1914

32.8

207

27-Jul-14

1914

32.8

208

28-Jul-14

1915

32.2

219

08-Aug-15

1916

28.9

213

01-Aug-16

1916

28.9

220

08-Aug-16

1917

30.6

265

23-Sep-17

1917

30.6

227

16-Aug-17

1918

33.9

164

14-Jun-18

1919

36.1

196

16-Jul-19

1920

33.3

198

17-Jul-20

1921

32.2

241

30-Aug-21

1922

35

213

02-Aug-22

1923

31.1

166

16-Jun-23

1924

33.3

187

06-Jul-24

1925

34.4

214

03-Aug-25

1926

28.9

235

24-Aug-26

1927

31.1

204

24-Jul-27

1928

32.2

222

10-Aug-28

1929

35

206

26-Jul-29

1930

35.6

235

24-Aug-30

1931

35.6

176

26-Jun-31

1932

32.2

231

19-Aug-32

1933

37.2

225

14-Aug-33

1934

33.3

214

03-Aug-34

1934

33.3

227

16-Aug-34

1934

33.3

145

26-May-34

1935

36.7

225

14-Aug-35

1936

34.4

191

10-Jul-36

1937

37.2

184

04-Jul-37

1937

37.2

185

05-Jul-37

1938

32.8

146

27-May-38

1938

32.8

191

11-Jul-38

1938

32.8

209

29-Jul-38

1939

35.6

191

11-Jul-39

1940

36.1

221

09-Aug-40

1941

41.1

199

19-Jul-41

1942

30.6

202

22-Jul-42

1943

35

187

07-Jul-43

1944

32.8

205

24-Jul-44

1945

34.4

228

17-Aug-45

1946

37.8

210

30-Jul-46

1947

35

208

28-Jul-47

1948

33.3

188

07-Jul-48

1948

33.3

242

30-Aug-48

1949

37.2

217

06-Aug-49

1950

36.7

207

27-Jul-50

1951

31.7

215

04-Aug-51

1952

32.2

118

28-Apr-52

1952

32.2

234

22-Aug-52

1952

32.2

237

25-Aug-52

1953

31.7

194

14-Jul-53

1954

32.8

200

20-Jul-54

1955

33.3

196

16-Jul-55

1955

33.3

198

18-Jul-55

1956

35

161

10-Jun-56

1957

33.9

206

26-Jul-57

1958

35.6

202

22-Jul-58

1959

35

204

24-Jul-59

1961

38.3

227

16-Aug-61

1962

33.3

176

26-Jun-62

1963

33.9

202

22-Jul-63

1964

32.8

193

12-Jul-64

1965

30.6

221

10-Aug-65

1966

32.2

196

16-Jul-66

1967

33.3

222

11-Aug-67

1967

33.3

246

04-Sep-67

1967

33.3

248

06-Sep-67

1968

30.6

170

19-Jun-68

1969

33.3

232

21-Aug-69

1970

32.8

219

08-Aug-70

1970

32.8

153

03-Jun-70

1970

32.8

218

07-Aug-70

1971

32.8

233

22-Aug-71

1971

32.8

217

06-Aug-71

1972

35

241

29-Aug-72

1973

32.2

224

13-Aug-73

1974

33.3

217

06-Aug-74

1974

33.3

199

19-Jul-74

1975

33.9

209

29-Jul-75

1976

32.2

235

23-Aug-76

1977

30

174

24-Jun-77

1978

33.9

221

10-Aug-78

1979

32.8

200

20-Jul-79

1980

32.8

191

10-Jul-80

1981

31

221

10-Aug-81

1981

31

251

09-Sep-81

1981

31

184

04-Jul-81

1981

31

183

03-Jul-81

1982

30

217

06-Aug-82

1982

30

245

03-Sep-82

1982

30

210

30-Jul-82

1983

35

242

31-Aug-83

1983

35

217

06-Aug-83

1984

36

208

27-Jul-84

1985

32

236

25-Aug-85

1985

32

185

05-Jul-85

1986

32.5

174

24-Jun-86

1986

32.5

147

28-May-86

1987

33

165

15-Jun-87

1987

33

207

27-Jul-87

1987

33

208

28-Jul-87

1988

40

157

06-Jun-88

1989

37

201

21-Jul-89

1990

34

217

06-Aug-90

1991

34

221

10-Aug-91

1991

34

242

31-Aug-91

1991

34

220

09-Aug-91

1992

32

226

14-Aug-92

1992

32

227

15-Aug-92

1993

31

171

21-Jun-93

1993

31

209

29-Jul-93

1994

30.5

208

28-Jul-94

1994

30.5

226

15-Aug-94

1995

32

149

30-May-95

1995

32

216

05-Aug-95

1996

33

242

30-Aug-96

1997

35.5

218

07-Aug-97

1998

35

217

06-Aug-98

1999

32.5

236

25-Aug-99

2000

31.5

235

23-Aug-00

2001

36

218

07-Aug-01

2002

34.5

198

18-Jul-02

2002

34.5

194

14-Jul-02

2003

36.5

227

16-Aug-03

2004

31

200

19-Jul-04

2004

31

198

17-Jul-04

2005

32.5

212

01-Aug-05

2006

32

179

29-Jun-06

2007

33.5

210

30-Jul-07

2008

35

237

25-Aug-08

2009

31

266

24-Sep-09

2009

31

164

14-Jun-09

About Me

jrwakefield (at) mcswiz (dot) com