Cross Validation

In Design Manager, before obtaining the surrogate function for the entire design space, sample data must first be generated by running designs in a design study. From the sample data generated in these real simulations, a surrogate function is calculated using the specified surrogate type. After computing the surrogate, you can use cross validation to assess the quality of the surrogate for predicting unknown response values.

Cross validation assesses the prediction accuracy by removing some of the sample data, fitting the remaining data with the surrogate function, and using the removed data as out-of-sample data by which to test the reduced surrogate.

The following table gives an overview of methods for assessing the prediction accuracy of the surrogate function:

Table 1. Surrogate Prediction
Surrogate Type Assessing the Surrogate Fit of the Known Data Assessing the Surrogate Fit of the Unknown Data
Kriging

RBF

Since the structure of the surrogate function defines the exact fit of the known data, cross validation is the only method by which to assess the prediction accuracy.
  • Cross V
  • PRESS

Correlation Coefficient (R2) is always equal to 1 and the Root Mean Squared Error (RMSE) is always equal to 0.

  • Cross V
  • PRESS
Least Squares
  • Correlation Coefficient (R2)
  • Root Mean Squared Error (RMSE)
  • Adjusted Correlation Coefficient (R2adj)
  • Cross V
  • PRESS

For example, if you have m data points and they are divided in k groups, then each group has n = m k data points. One group is used as training data to validate the fit of the reduced surrogate function, which is calculated from the remaining data points. For the i t h data point, the predictive residual c v i is computed as follows:

Figure 1. EQUATION_DISPLAY
c v i = y i y ^ i
(5173)
where
  • y i is the true value of the i t h data point.
  • y ^ i is the predicted value of the i t h data point when evaluated with the reduced surrogate model.

The predictive residual is computed for all the m data points. The Mean Square Error (MSE) is calculated as follows:

Figure 2. EQUATION_DISPLAY
M S E = 1 m i = 1 m ( c v i ) 2
(5174)

The Cross V value of a design study is the Root Mean Squared Error (RMSE):

Figure 3. EQUATION_DISPLAY
c v s t u d y = M S E
(5175)