Do Climate Models Work?

Figure 1. Indicators of global climate change from CMIP6 historical and scenario simulations. (a) Global surface air temperature changes relative to the 1995–2014 average (left axis) and relative to the 1850–1900 average (right axis). (b) Global land precipitation changes relative to the 1995–2014 average. (c) September Arctic sea ice area. (d) Global mean sea level (GMSL) change relative to the 1995–2014 average. (a), (b) and (d) are annual averages, (c) are September averages. In (a–c), the curves show averages over the CMIP6 simulations, the shadings around the SSP1‑2.6 and SSP3‑7.0 curves show 5–95% ranges, and the numbers near the top show the number of model simulations used. Results are derived from concentration-driven simulations. In (d), the barystatic contribution to GMSL (i.e., the contribution from land-ice melt) has been added offline to the CMIP6 simulated contributions from thermal expansion (thermosteric). The shadings around the SSP1‑2.6 and SSP3‑7.0 curves show 5–95% ranges. The dashed curve is the low confidence and low likelihood outcome at the high end of SSP5‑8.5 and reflects deep uncertainties arising from potential ice-sheet and ice-cliff instabilities.

The basis of all statements from Climate Activists saying we must "follow the science" are the climate models. This page asks do these models "work" with the meaning do these models agree with the past observed data and give any meaningful predictions for the future.

Figure 1, taken from the IPCC report: AR6 Climate Change 2021: The Physical Science Basis where it is Figure 4.2, show four of the indicators generated by the climate models. The shaded areas in these plots show the large range of predictions from the multiple models for each of the emission scenarios. These models are the CMPI6 models produced for the AR6 cycle of IPCC reports. For the full range of  models and scenarios the prediction for the surface temperature rise is between 1.3°C and  8.1°C in 2100 compared to the 1850-1900 baseline period. The increase in precipitation is between -2% and >+10% in 2100 relative to 1995-2014. The September Arctic sea ice cover ranges from ice free to nearly twice the 2014 ice cover. The sea level rise ranges from 0.3m to 0.9m increase in 2100  relative to 2010.

These enormous ranges of predictions give little useful information for policy decisions for the future, however the climate alarmists focus on the most extreme changes to make their predictions of immediate breakdown of society.

The IPCC report gives little emphasis to either the likely range of predictions or the agreement between the model predictions and existing data. This page explores these issues and studies whether some models should be rejected because they badly reproduce observed temperature anomalies in the period where measurements already exist.

Climate Models

The climate models, used to produce figure 1, are complicated computer simulation of the Earth's atmosphere and oceans. Figure 2, taken from the book "Unsettled" by Steven Koonin, illustrates how these simulations operate. The atmosphere is covered with a three-dimensional grid, typically about 10-20 layers vertically and cells of 100 km x 100 km horizontally. The grid for the oceans is similar but finer, with up to 30 vertical layers and 10 km x 10 km horizontally. With this the whole Earth is covered with typically one million cells for the atmosphere and 100 million cells for the oceans.

Figure 2 (Left). Example of division of atmosphere into cells (boxes). (Right) Example of one of the major problems with the models: clouds.

With the grid in place the computer simulations use the fundamental laws of physics to calculate how air, water and energy in each cell at a given time move to neighbouring grid cells, at a slightly later time which can be as small as 10 minutes. This process is repeated millions of times to simulate the climate for a century into the future. These simulations can take months of computer time on the world's most powerful supercomputers.

One of the intrinsic problems of these models is the size of the cells which cannot be smaller because of the limits on computer power. Many phenomena, in particular clouds, are smaller than the possible cell size and these phenomena must be simulated with approximations on the grid.

Climate modelling has a long history and in 1995 started to be coordinated by the World Climate Research Programme (WCRP)  with the Coupled Model Intercomparison Project (CMIP). CMIP is now a global enterprise with more than 50 modelling centres participating in the latest CMIP6 cycle, across six continents.

The climate models rely on greenhouse gas emissions and these inputs have been agreed on by international structures and stored at Earth System Grid Federation. The emissions inputs for the CMIP5 models used for the IPCC ARS5 reports were the Representative Concentration Pathway (RCP) scenarios and those for the CMIP6 models used for ARS6 the Shared Socio-economic Pathways (SSP) scenarios. Each of these cycles of models uses several different scenarios with different predictions for future emissions and figure 3 (from IPCC ARS6 WGI report) presents the atmospheric concentrations of CO₂ CH₄ and N₂O. The scenarios with the highest emissions RCP 8.5 and SSP5 8.5 are now considered to be too extreme and unlikely. The weather parameter predictions from CMIP6 shown in figure 1 use the full range of SSP scenarios. The plots presented in the rest of this page use the RCP 4.5 for CMIP5 and SSP2 4.5 for CMIP6 predictions.

Figure 3. Green House Gas atmospheric concentrations in the Shared Socio-economic Pathways (SSP) scenarios and the Representative Concentration Pathway (RCP) scenarios.

Climate Models Predictions Compared to Observed Temperature Data

The plots presented below use temperature data (at 2-metre from the surface) for CMIP5 and CMIP6 models and observed ERA5 data downloaded from the ECMFW database using the Gridded monthly climate projection dataset underpinning the IPCC AR6 Interactive Atlas.

Figure 4 compares model predictions and observed data for the Global Mean Surface Temperature (GMST) anomalies over the whole globe. The temperature anomalies are relative to the baseline period 1961-1990. The model predictions show the full range of predictions for different models with the shaded regions and the mean of the model ensemble with the solid blue and grey lines. The observed ERA5 temperature data is the red line. All the lines use a moving average over 10 years.

Figure 4. GMST anomalies relative to the baseline period 1961-1990 for CMIP5 and CMIP6 models and ERA5 observed data, averaged over all global latitudes. CMIP5 is in grey and CMIP6 is blue  with the minimum to maximum range of models with the shaded region and the average of all models as the line. The observed ERA5 temperature data is the red line. All the lines use a moving average over 10 years.

The CMIP5 data are historical hindcasts until 2005 and predictions later, while CMIP are hindcast before 2014. From the figure it can be seen that the different models have a large range of predictions between 1.1°C and 3.2°C by 2050. Also that the more developed CMIP6 models have a wider range than the earlier CMIP5 models. After 1998 the observed data starts to diverge from the mean of the models and these differences between observations and models will be explored further below.

The increase in observed global temperature is very different at different latitudes, as shown in detail on the page on this website: Does the focus on one iconic global temperature plot make sense? Figure 5 compares the model predictions and observed temperature as a function of latitude with the same colourings as figure 4.

Figure 5. GMST anomalies averaged over the period 2010-2019 relative to the baseline period 1961-1990 for CMIP5 and CMIP6 models and ERA5 observed data, as a function of latitude latitudes. CMIP5 is in grey and CMIP6 is blue  with the minimum to maximum range of models with the shaded region and the average of all models as the line. The observed ERA5 temperature data is the red line. 

Can some models be rejected because of disagreements with observations?

Figures 4 and 5 show that the models have a wide range of temperature predictions inside the time period when observed data exists. Studies have been published to explore if some models can be rejected already because of disagreement with the surface temperature record. A recent publication which does that is Nicola Scafetta, has received much media attention and criticism for example from Gavin Schmidt et al. This section makes an analysis with a similar objective as Scafetta but dividing the globe in a different way and with a different statistical evaluation.

Figure 6 compares CMIP5 and CMIP6 models with observed 2-metre observed temperature from the ERA5 re-analysis for the mid-range of global latitudes 60°S to 60°N. The plots gives 22 CMIP5 models and 34 CMIP6 in blue with the observed temperature anomalies in red. The plot clearly shows that after the reference period (1961-1990) the observed temperature anomalies diverge below the predictions of the majority of models for both CMIP5 and CMIP6.

Figure 6. Temperature anomalies for CMIP5 and CMIP6 models (blue) compared to ERA5 observed data (red), for the latitude range 60° S to 60°. For the CMIP5 (CMIP6) models up to 2005 (2014) the values are hindcasts and after they are projections. All curves use a moving averages over 10 years.

To quantify these disagreement, two statistical analyses are made. The first of these methods (Δslope) compares the fitted slopes of the temperature trends between 1979 and 2022. The second method (Χ² or chis-squared) calculates the Χ² between 2013 and 2022. These two estimator are defined as:

Figure 7 uses the examples of two extreme models to illustrate the model and data comparison and the estimators use to evaluate the differences. Figure 8 plots all the fitted temperature anomaly slopes (b) for the CMIP5 and CMIP6 models studied here. The fits to the ERA5 data are displayed as the horizontal red lines together with the corresponding error estimates (errb_obs). The errors on the models (errb_model) are not displayed in the plot but are typically 1.5 times larger than that on the observed data..

These temperature measurement uncertainties ("error bars") are basic to the statistical analysis but must be derived in an empirical fashion from the data. The ECMWF database provides no estimates of uncertainties but HadCRUT5 provides error bars which are frequently used in publications. These HadCRUT temperature measurement errors are year-to-year for latitudes 90°S to 90°N, about 0.02°C. To obtain slope fits with acceptable chis-squared, these HadCRUT error must be scaled by factors between 5 and 20. In method 1) the temperature errors were adjusted such that the fitted chi-squared was 1/degree of freedom and for method 2) using the standard deviation of the fit residuals, both giving very similar values. For the latitude range 60°S to 60°N these errors are 0.10°C for observations and 0.15-0.20°C for models. Each latitude range and model is used with different errors derived from the corresponding data.

Figure 7. Examples of two CMIP6 models compared to observed temperature anomalies. Left: MPI-ESM1-2-LR model and right: UKESM1-0-LL model. The plots show the observed data in red and the models in blue. The observed data and models are plotted with errors bars which are derived as described in the text and the two statistical quantities used to yield the probabilities of agreement are shown.

Figure 8. Slopes from linear fits to CMIP5 (top row) and CMIP6 (bottom row) model temperature anomaly data as a function of the Equilibrium Climate Sensitivity (ECS) for each model shown in three ranges of latitude: Antarctic, mid-latitudes and Arctic. The fits are made over the time period 1979-2022 and are applied to yearly data. The slopes for observed ERA5 data are indicated by the red line with the 95% confidence level range indicated by the shaded area. The Pearson Correlation coefficient for the slopes compared to the ECS for the models are given in each plot.

The Equilibrium Climate Sensitivity (ECS) used throughout this analysis are taken from Schlund et al.  From figure 8 it can be seen that most models have higher slopes than the observations and have less correlation with the ECS for the Antarctic latitudes

With the two estimators, probabilities can be calculated using the appropriate statistics: 1) with a Normal Distribution and 2) with a Chi-Squared Distribution. For the second example in figure 8 (UKESM1-0-LL), the Δslope = 16.1 with the the probability to exceed this value <0.001% and Χ² = 80.7 for 10 years, with the the probability to exceed this value also <0.001%. For the first example Δslope = -0.2 and Χ² = 5.7 for 10 years, both values giving agreement between the model and observations within the 95% confidence level range. Table 1 gives the probabilities for these two statistical analyses for the high ECS CMIP6 models where the disagreement between models and observations is greatest. 

Table 1. Analysis of agreement/disagreement between CMIP6 models with high ECS for three latitude ranges with the Δslope and Χ² methods. The probabilities for exceeding the estimator values are given, with the label "Agree" meaning that the values have agreement within a 95% confidence range. The green highlight is used to indicate values with agreement; red with disagreement and purple with disagreement but model lower than data.

In this table 1, it can be seem that the two estimators lead to the same conclusion when the difference between model and observations are large but when the disagreement is less, the Χ² value does not show significant probabilities beyond the 95% confidence level range. This is because the Δslope method uses the full  44 year range from 1979-2022 while the Χ² method only uses the 10 year range 2013-2022. The Χ² statistical method is similar to the Student-t method used in the Scafetta and Schmidt et al papers.

Table 2 focuses just on the more sensitive Δslope statistical method showing the state of model agreement with ERA5 data for all CMIP5 and CMIP6 models considered. Table 3 gives a summary of the results in table 2.

Table 2. Analysis of agreement/disagreement between CMIP5 and CMIP6 models and observed temperature anomalies for 30 three latitude ranges with the Δslope method. The green highlight is used to indicate values with agreement; purple with disagreement but model slope lower than data and red with disagreement with model slope higher than data. The Equilibrium Climate Sensitivity (ECS) for each model is given and the models are ranked from highest ECS at the top to lowest at the bottom. 

Table 3. Summary of results in table 2 showing, for the CMIP5 and CMIP6 models in 3 latitudes ranges, the proportion of models which agree; disagree with lower slope than the ERA5 observed data and disagree with higher slope. The table splits the data in three ranges of ECS: high; medium and low corresponding to the divisions in the Scafetta paper.

From table 3 it can be seen that for the mid-latitude range 60°S to 60° gives a clear picture of model agreement with ERA5 being correlated with the ECS. In the Antarctic and Artic ranges the correlation is less clear. Figure 9 summaries the situation for the mid-latitude range, giving the model prediction for the year 2100 versus the Equilibrium Climate Sensitivity with the point colours indicating which model agree/disagree with data according to the summaries the situation for the mid-latitude range, giving the model prediction for the year 2100 versus the Equilibrium Climate Sensitivity with the point colours indicating the agreement/disagreement with data according to the Δslope analysis.

Figure 9. Summary plot for the mid-latitude range( 60°S to 60°N), giving the model prediction for the year 2100 versus the Equilibrium Climate Sensitivity with the point colours indicating the agreement/disagreement with data according to the Δslope analysis. The colour code indicates: green agreement; purple with disagreement but model lower slope than data and red with disagreement but model with higher slope than data.

Conclusions

The analysis presented on this page shows that most CMIP5 and CMIP6 Climate Models with Equilibrium Climate Sensitivity > 3.0 disagree with the observed temperature anomalies because they give a temperature increase  up to 2022 which is significantly larger than that which is observed. This is the same conclusion as the Scafetta paper.

The answer to the title question of this page "Do Climate Models Work" is no, for the large majority of CMIP6 models using the assessment of observed surface temperature anomalies. This means that using the ensemble average of the models to make predictions for the future is wrong. 

To conclude that any models do "work" requires more studies.