Among validation strategies, k-fold cross validation (CV) is popular
to estimate the performance of the model in view to data that have not
been used for model training. During CV, models are repeatedly trained
(k models) and in each model run, the data of one fold are put to the
side and are not used for model training but for model validation. In
this way, the performance of the model can be estimated using data that
have not been included in the model training. Summary statistics can be
calculated either by averaging the performance per fold (default in
caret) or by calculating the statistics over all held-back data at once
(CAST::global_validation).
The Standard approach: Random k-fold CV
In the example above we used a random k-fold CV that we defined in
caret’s trainControl argument. More specifically, we used a random
3-fold CV. Hence, the data points in our dataset were RANDOMLY split
into 3 folds. To assess the performance of the model let’s have a look
on the output of the Random CV:
## Random Forest
##
## 703 samples
## 11 predictor
##
## No pre-processing
## Resampling: Cross-Validated (3 fold)
## Summary of sample sizes: 470, 469, 467
## Resampling results:
##
## RMSE Rsquared MAE
## 25.75544 0.6651004 15.08417
##
## Tuning parameter 'mtry' was held constant at a value of 2
global_validation(model_default)
## RMSE Rsquared MAE
## 25.9283336 0.6672803 15.0845659
We see that species richness could be modelled with a R² of 0.66
which indicates a good fit of the data. Sounds good, but unfortunately,
the random k fold CV does not give us a good indication for the map
accuracy if the reference data are not a random sample of the prediction
area. Random k-fold CV means that each of the three folds (with the
highest certainty) contains data points from each spatial cluster.
Therefore, a random CV cannot indicate the ability of the model to make
predictions beyond the location of the training data (i.e. to map
species richness). Since our aim is to map species richness, we rather
need to perform a target-oriented validation which validates the model
in view to spatial mapping.
Target-oriented validation
We are not interested in the model performance in view to random
subsets of our vegetation plots, but we need to know how well the model
is able to make predictions for areas without reference samples. To find
this out, we need to repeatedly leave larger spatial regions of one or
more vegetation plots out and use them as test data during CV. Several
suggestions of spatial CV exist. CAST implements a straightforward way
(CAST::createSpaceTimeFolds) and the sophisticated method of nearest
neighbor distance matching (Mila et al 2022, Linnenbrink
et al 2023) as either leave-one-out CV (CAST::nndm) or as k-fold CV
(CAST::knndm).
CAST’s function “CreateSpaceTimeFolds” is designed to provide index
arguments used by caret’s trainControl. The index defines which data
points are used for model training during each model run and reversely
defines which data points are held back. Hence, using the index argument
we can account for the dependencies in the data by leaving the complete
data from one or more regions out (LLO CV), from one or more time steps
out (LTO CV) or from locations and time steps out (LLTO CV). In this
example we’re focusing on LLO CV. We use the column “Country” to define
the location of the samples and split the data into folds using this
information, i.e. we avoid that data from the same country are located
in both, training and test data. Analog to the random CV we split the
data into three folds, hence three model runs are performed each leaving
one third of all data out for validation.
Knndm tries to find a k-fold configuration such that the integral of
the absolute differences (Wasserstein W statistic) between the empirical
nearest neighbour distance distribution function between the test and
training data during CV, and the empirical nearest neighbour distance
distribution function between the prediction and training points, is
minimised. In other words, we try to split the data into folds so that
the difficulty of the prediction during cross-validation is comparable
to the difficulty when deploying the model for entire South America,
where the difficulty is defined by spatial distances. Note, that feature
space distances are also an implemented option.
See tutorial on the cross-validation in this package for more
information.
Example for CreateSpacetimeFolds:
set.seed(10)
indices_LLO <- CreateSpacetimeFolds(splotdata,spacevar = "Country",
k=3)
Example for knndm:
set.seed(10)
indices_knndm <- knndm(splotdata,predictors_sp,k=3)
Let’s compare how well these strategies fit the prediction task,
especially in comparison to a random cross-validation. We will use
CAST’s geodist function for that. Geodist calculates nearest neighbor
distances in geographic space or feature space between training data as
well as between training data and prediction locations. Optional, an
that is what we want to focus on here, the nearest neighbor distances
between training data and CV folds is computed. See tutorial on
geodistance visualizations in this package for more information.
plot(geodist(splotdata,predictors_sp,cvfolds =model_default$control$indexOut))+
scale_x_log10(labels=round)
plot(geodist(splotdata,predictors_sp,cvfolds =indices_LLO$indexOut))+
scale_x_log10(labels=round)
plot(geodist(splotdata,predictors_sp,cvfolds =indices_knndm$indx_test))+
scale_x_log10(labels=round)
We see that using random folds, we’re only testing how well the model
can make predictions for new areas that are a few hundreds of meters
away from the training data. Using createSpaceTimeFolds with the Country
as spatial unit (i.e. leave-country-out CV), we produce prediction
situations that are even harder than the prediction task. Knndm provides
the best option, since the CV distances that we create during
cross-validation are comparable to what is required when we predict for
the entire area.
model <- train(st_drop_geometry(splotdata)[,predictors],
st_drop_geometry(splotdata)$Species_richness,
method="rf",
tuneGrid=data.frame("mtry"=2),
importance=TRUE,
trControl=trainControl(method="cv",
index = indices_knndm$indx_train,
savePredictions = "final"))
model
## Random Forest
##
## 703 samples
## 11 predictor
##
## No pre-processing
## Resampling: Cross-Validated (10 fold)
## Summary of sample sizes: 520, 442, 444
## Resampling results:
##
## RMSE Rsquared MAE
## 34.13482 0.4754385 20.79699
##
## Tuning parameter 'mtry' was held constant at a value of 2
## RMSE Rsquared MAE
## 34.5314356 0.4269648 20.9480996
By inspecting the output of the model, we see that in view to new
locations, the R² is much lower and the RMSE much higher compared to
what was expected from the random CV.
Apparently, there is considerable overfitting in the model, causing a
good random performance but a poor performance in view to new locations.
This might partly be attributed to the choice of variables where we must
suspect that certain variables are misinterpreted by the model (see Meyer et al
2018 or talk
at the OpenGeoHub summer school 2019).
Let’s have a look at the variable importance ranking of Random
Forest. Assuming that certain variables are misinterpreted by the
algorithm we should be able to produce a higher LLO performance when
such variables are removed. Let’s see if this is true…
