Summary

A scheme for testing the accuracy of BBC weather forecasts for Weymouth has been used based on the weather symbols displayed on the website [1].

This shows that forecasts are correct on only about one day in four irrespective of time ahead of the forecast from one day to four.

About half the forecasts are unacceptable in the sense that the weather is annoyingly different from that forecast, e.g., forecasting a sunny day when it actually rains.

Choosing weather symbols at random and taking this as a 'forecast' works acceptable on about one day in five - not as good as professional forecasting. 

However, assuming that the weather tomorrow will be broadly the same as today produces about 40% acceptable forecasts. This works about as well as professional forecasters' predictions.

The Meteorological Office tests its forecasts by measuring the accuracy of the forecasts for daily maximum temperature. The accuracy is better than persistence (e.g., assuming that tomorrow will be the same as today) but only by about 10 - 15%.

For an introduction to the weather symbols please click here.

This web page examines the reliability of weather forecasts for Weymouth.

What's the problem?

Well, on Wednesday 27th September 2005 at 09.00 local time the BBC local weather forecast [1] for Weymouth predicted the following:-

When the 29th September finally arrived we had the following forecasts for the day as shown at 09.00 BST:

BBC Online
Countryside Online

The day actually was clear and sunny in the morning changing to light cloud in the afternoon but remaining dry all day!

As another example of bad forecasting, on 8th October 2005 the forecast for the 11th was

but by the 10th the forecast had changed to

- a very bad mistake!

The Meteorological Office assesses the general accuracy of its weather forecasting using a panel drawn from the public who look at the forecast and then what actually happened. Inevitably, such tests have a subjective element in them. 

For example, if a forecast was

"Some sunny intervals but with a chance of light rain which might be heavy locally."

That could be judged to be correct whether it was sunny or not and whether it rained or not. It's rather like reading those vague tabloid horoscopes.

What I have done is to perform a strictly quantitative test of forecasts for Weymouth based on the summary weather symbols on the BBC website for Weymouth, [1]. These symbols are typically as shown above.

Each day at 09.00 local time, I recorded the forecast weather symbols as shown above. I then recorded whether these symbol for a specific day changed as the day drew closer. Finally, I recorded the weather symbol for the day on which the forecast was shown.

Note that this does not actually test the forecast reliability - it is rather a test of the consistency of the forecasts.

Figure 1 shows the fraction of forecasts that were the same either 1, 2, 3 or 4 days ahead when compared with the weather symbol displayed on that actual day. In other words, on how many occasions was the forecast changed when the forecast day eventually arrived?

It can be seen that the weather symbol changed on about 70% - 75% of forecasts - even from one day to the next there were two chances in three that the weather forecast would change as indicated by the choice of symbol.

Not very good!

Perhaps I am being too harsh on the forecasters. If they change the weather symbol this may not be important if the change is small. 

For example, changing from

to

isn't going to worry me much if I am going

out for the day and intend to leave my washing out to dry although it might affect my decision whether to sunbathe on Weymouth Beach or potter in the garden instead.

However, forecasting

and the day is

would really annoy most people who

look to weather forecasts when planning events or activities.

Accordingly, I drew up a scale of significant changes to see whether changes in forecast were socially significant or not. This errs on the side of fairness to the forecasters.

I grouped weather symbols together such that each group was given a number and the symbols within each group were judged to be acceptably close in meaning not to be considered inconsistent.

Group	Compatible Symbols
1
2
3
4
5

Figure 2

Figure 2 shows my groupings.

I have omitted winter symbols (snow and sleet) because this experiment was run throughout the Summer and Autumn of 2005.

If the forecast changes between symbols within a group this is taken as an acceptable forecast. However, if the forecast symbol moves from one group to another this is taken to be an inaccurate and unacceptable forecast accuracy.

Figure 3

Figure 3 shows the average differences in the grouping number as a function of the number of days the forecasts are made in advance.

It can be seen that attempts to forecast three or more days ahead result in unacceptable forecasting errors - not very good!

Figure 4 shows the proportion of forecasts that come up with weather symbols in the same group (purple bars). Also shown on this graph is the proportion of weather forecasts that would be acceptable if weather symbols were chosen at random (blue bars). It can be seen that forecasters are generally better than guesswork except four days ahead when the forecasters do no better than random guessing.

Another type of 'forecast' is known as 'persistence'; which means assuming that the weather will be broadly the same in the future as it is today. The produces acceptable forecasts (weather symbols in the same grouping on consecutive days) at the rate shown by the orange bars on Figure 4. It can be seen that the forecasters perform better than persistence up to three days ahead - but no better after four days!

Forecasting four days into the future is no better than guesswork.

Comparing figures 1 and 2 shows that, although the forecasters get their forecasts wrong on about three days in four, they come up with forecasts that are acceptable about one day in two,  two and three days ahead but only one forecast in four is acceptable four days ahead.

Another test is to check the weather forecast at 09.00 Local Time and then, at the end of that day, see whether the forecast was accurate. We might expect that the forecasters would be very accurate in predicting what would be happening in the next fifteen hours.

Not so!

On only 40% of morning forecasts did the day turn out as predicted by the weather symbol and on only 59% of forecasts did the day turn out in the same group of symbols.

As an example of what can go wrong, at 09.00 on 30th September 2005 the forecast for that same

day was

- a good day for hanging out the washing or going for a country walk.

In fact, it was overcast and rained hard in the afternoon. My washing - and I - would have been soaked had I hung it out and gone for that walk!

The Meteorological Office records the accuracy of its forecasts of daily maximum temperature on a website reached by clicking here. These results are averaged for forecasts on eleven cities in the UK.

Figure 5

Figure 5 shows the Meteorological Office assessment of its forecasts of daily maximum temperature between 2001 and 2005. The graph shows the percentage of forecast temperatures than are within 2⁰C of the measured values.

Also plotted are the accuracies for Weymouth using persistence; namely, assuming that the forecast is given by a previous day's actual value.

It can be seen that the Meteorological Office forecasts for daily maximum temperature are about 10 - 15% more accurate than one-day persistence.