Please click on my picture at left to return to the home page.

Summary

In 2009 the BBC on-line weather forecast changed substantially with new symbols and new information. A new analysis of weather forecasting accuracy is underway and will be published here when completed. This will be more comprehensive than the results shown below and will include the accuracy with which maximum and minimum temperatures for Weymouth are forecast.

Please click here to see the provisional results for 2009.

Meanwhile, back in 2005 a scheme for testing the accuracy of BBC weather forecasts for Weymouth was used based on the weather symbols displayed on the website [1].

This shows that forecasts are correct on only about one day in four irrespective of time ahead of the forecast from one day to four days.

About half the forecasts are unacceptable in the sense that the weather is annoyingly different from that forecast, e.g., forecasting a sunny day when it actually rains.

Choosing weather symbols at random and taking this as a 'forecast' works acceptable on about one day in five - not as good as professional forecasting. 

However, assuming that the weather tomorrow will be broadly the same as today produces about 40% acceptable forecasts. This works about as well as professional forecasters' predictions.

Introduction

This web page examines the reliability of weather forecasts for Weymouth.

What's the problem?

Well, on Wednesday 27th September 2005 at 09.00 local time the BBC local weather forecast [1] for Weymouth predicted the following:-

27th Sept 28th Sept 29th Sept 30th Sept 1st Oct
whereas the Countryside Online website [2] predicted 
You can see that, even forecasting at 9 am for the actual day, one site predicts 'Light Showers' whilst the other predicts 'Sunny Intervals'. That may not be a devastating difference - a matter of life or death - but it shows that forecasters cannot even get the same results. It certainly matters if deciding whether to hang out your washing before leaving for work!

When the 29th September finally arrived we had the following forecasts for the day as shown at 09.00 BST:

BBC Online
Countryside Online

The day actually was clear and sunny in the morning changing to light cloud in the afternoon but remaining dry all day!

As another example of bad forecasting, on 8th October 2005 the forecast for the 11th was

but by the 10th the forecast had changed to  - a very bad mistake!

The Meteorological Office assesses the general accuracy of its weather forecasting using a panel drawn from the public who look at the forecast and then what actually happened. Inevitably, such tests have a subjective element in them. 

For example, if a forecast was

"Some sunny intervals but with a chance of light rain which might be heavy locally."

That could be judged to be correct whether it was sunny or not and whether it rained or not. It's rather like reading those vague tabloid horoscopes.

Forecast Accuracy

What I have done is to perform a strictly quantitative test of forecasts for Weymouth based on the summary weather symbols on the BBC website for Weymouth, [1]. These symbols are typically as shown above.

Each day at 09.00 local time, I recorded the forecast weather symbols as shown above. I then recorded whether the symbol for a specific day changed as the day drew closer. Finally, I recorded the weather symbol for the day on which the forecast was shown.

Note that this does not actually test the forecast reliability - it is rather a test of the consistency of the forecasts.

Figure 1

Figure 1 shows the fraction of forecasts that were the same either 1, 2, 3 or 4 days ahead when compared with the weather symbol displayed on that actual day. In other words, on how many occasions was the forecast changed when the forecast day eventually arrived?

It can be seen that the weather symbol changed on about 70% - 75% of forecasts - even from one day to the next there were two chances in three that the weather forecast would change as indicated by the choice of symbol.

Not very good!

Perhaps I am being too harsh on the forecasters. If they change the weather symbol this may not be important if the change is small. 

For example, changing from    to  isn't going to worry me much if I am going 

out for the day and intend to leave my washing out to dry although it might affect my decision whether to sunbathe on Weymouth Beach or potter in the garden instead.

However, forecasting  and the day is  would really annoy most people who 

look to weather forecasts when planning events or activities.

Accordingly, I drew up a scale of significant changes to see whether changes in forecast were socially significant or not. This errs on the side of fairness to the forecasters.

I grouped weather symbols together such that each group was given a number and the symbols within each group were judged to be acceptably close in meaning not to be considered inconsistent.

Group

Compatible Symbols

1
2   
3     
4
5   

Figure 2

Figure 2 shows my groupings.

I have omitted winter symbols (snow and sleet) because this experiment was run throughout the Summer and Autumn of 2005.

If the forecast changes between symbols within a group this is taken as an acceptable forecast. However, if the forecast symbol moves from one group to another this is taken to be an inaccurate and unacceptable forecast accuracy.

Figure 3

Figure 3 shows the average differences in the grouping number as a function of the number of days the forecasts are made in advance.

It can be seen that attempts to forecast three or more days ahead result in unacceptable forecasting errors - not very good!

Figure 4

Figure 4 shows the proportion of forecasts that come up with weather symbols in the same group (purple bars). Also shown on this graph is the proportion of weather forecasts that would be acceptable if weather symbols were chosen at random (blue bars). It can be seen that forecasters are generally better than guesswork except four days ahead when the forecasters do no better than random guessing.

Another type of 'forecast' is known as 'persistence'; which means assuming that the weather will be broadly the same in the future as it is today. The produces acceptable forecasts (weather symbols in the same grouping on consecutive days) at the rate shown by the orange bars on Figure 4. It can be seen that the forecasters perform better than persistence up to three days ahead - but no better after four days!

Forecasting four days into the future is no better than guesswork.

Comparing figures 1 and 2 shows that, although the forecasters get their forecasts wrong on about three days in four, they come up with forecasts that are acceptable about one day in two,  two and three days ahead but only one forecast in four is acceptable four days ahead.

Another test is to check the weather forecast at 09.00 Local Time and then, at the end of that day, see whether the forecast was accurate. We might expect that the forecasters would be very accurate in predicting what would be happening in the next fifteen hours.

Not so!

On only 40% of morning forecasts did the day turn out as predicted by the weather symbol and on only 59% of forecasts did the day turn out in the same group of symbols.

As an example of what can go wrong, at 09.00 on 30th September 2005 the forecast for that same

day was - a good day for hanging out the washing or going for a country walk. 

In fact, it was overcast and rained hard in the afternoon. My washing - and I - would have been soaked had I hung it out and gone for that walk!

DAILY MAXIMUM TEMPERATURES

The Meteorological Office records the accuracy of its forecasts of daily maximum temperature on a website reached by clicking here. These results are averaged for forecasts on eleven cities in the UK.

Figure 5

Figure 5 shows the Meteorological Office assessment of its forecasts of daily maximum temperature between 2001 and 2005. The graph shows the percentage of forecast temperatures than are within 20C of the measured values.

Also plotted are the accuracies for Weymouth using persistence; namely, assuming that the forecast is given by a previous day's actual value.

It can be seen that the Meteorological Office forecasts for daily maximum temperature are about 10 - 15% more accurate than one-day persistence.

PROVISIONAL RESULTS FROM 2009 ONWARDS

Changes to the BBC Weather Forecast website in 2009 included a new set of symbols so the analysis performed in 2005 could not be continued. However, using the new symbols the following results have so far been obtained. The analysis is continuing.

The new symbols were divided into three groups as follows

Group 1   Sunny, Sunny Intervals

Group 2   White Cloud, Grey Cloud

Group 3   Drizzle, Light Showers, Heavy Showers, Light Rain, Heavy Rain

Figure 6

Figure 6 shows the actual weather on any day compared with the forecast 'Group' up to four days ahead.

Picking a random grouping would be expected to give an accurate forecast one day in three which is the horizontal line at 33%.

For comparison, the accuracy of using persistence is shown - this meaning that it is assumed that the weather will not change from that observed at the appropriate number of days in the past.

It can be seen that persistence is slightly worse than the BBC Weather website forecast - but not by a significant amount.

Thus, assuming that the weather will be the same over the next four days as today works pretty well.

Figure 7

Figure 7 shows how well the BBC Weather website and the simple persistence assumption compare when applied to the actual nine symbols used.

The horizontal line shows the result of guessing each day from the nine symbols; a probability of 11%.

It can be seen again that there is essentially no significant difference between persistence and the BBC Weather forecast website.

New for 2009 I analysed the accuracy with which the BBC Weather website forecast maximum and minimum temperatures each day compared with the actual values from the Weymouth Weather Station.

Figure 8

Figure 8 shows the actual and forecast maximum temperatures in Weymouth. It can be seen that the predictions are good with the exception of a few outlier points.

Figure 9

Figure 9 shows the predicted and actual minimum temperatures for Weymouth. It can be seen that the accuracy is poor. Perhaps this is expected as timing of the arrival of a weather front is difficult and an unexpected arrival of clear skies after heavy cloud during the night will have a substantial effect on minimum temperatures experienced and forecast.

Figure 10

Figure 10 shows the Root Mean Square (RMS) errors for minimum and maximum temperature forecasts compared with persistence. It can be seen that assuming that temperatures will be the same as up to four days previously works as well as using the BBC Weather website forecasts of maximum and minimum temperature.

REFERENCES

1.    BBC Weymouth weather forecasts http://www.bbc.co.uk/weather/5day.shtml?id=1738&links

2.    Countryside weather forecast for Dorset http://www.countrysideonline.co.uk/weather(7)-28.htm

3.    Accuracy of Daily Maximum Temperatures forecasts http://www.met-office.gov.uk/corporate/verification/maxtemp.html

 

Please click on my picture at left to return to the home page.

GJK01305

Keywords Weather climate forecasting local accuracy Weymouth Dorset