Least Squares

Least squares linear and quadratic models create a polynomial that approximates the data sample globally as a best-fit.

For example, for one design study containing two input parameters $x_1$ and $x_2$ , a linear polynomial for predicted value $\hat{y}$ including constant and linear terms is defined as:

A quadratic polynomial including constant, linear terms, interaction, and squared terms is defined as:

The fit quality of a polynomial to a data point is measured by its residual ${\epsilon }_i$ defined as the difference between the actual value $y_i$ and the predicted value $\widehat{y_i}$ :

In matrix notation, the equation is:

Here, $X$ represents the design matrix. For example, the design matrix of a quadratic model with two input parameters $x_1$ and $x_2$ from m simulations is:

The vector $\beta$ of the unknown coefficients is:

Best possible fit means that the residual sum of squares $L$ is minimized for the sample data:

In the form of linear approximation:

After expanding the linear approximation:

Due to the dimensions of the vector $y$ , $\beta$ , and the matrix $X$ , ${\beta }^t{X}^{t}y$ and $y^{t}X\beta$ are two scalars. Therefore, $y^{t}X\beta \text{\hspace{0.17em}=\hspace{0.17em}}{\left({\beta }^t{X}^{t}y\right)}^t={\beta }^t{X}^{t}y$ . The equation can be re-written as:

The minimum of the sum of squares is found when the gradient of linear approximation is zero:

Least squares surrogates usually do not pass through the sample points directly. They go beside the sample points in the design space to obtain a best global fit. The least squares solution can be computed using the singular value decomposition of the matrix $X$ . This method is efficient with a large number of sample data.