3 commonly used metrics in Regression
In our previous post, we talked about different metrics for evaluating a classification model. Here, we are going to extend the topic from classification to regression. In this post, we are going to introduce 3 commonly used metrics to evaluate regression model.
Notations
Let’s go through some basic concept in regression before we go into the details of the metrics.
The goal in regression is to find the relationship between the dependent variable \(y\) and the independent variables \(x = [x_1, \cdots, x_N]\). This relationship is represented in the form of a function \(f\) and a parameter set \(\theta\).
In most of the case, the model or estimation is not perfect and there certain error \(e\) between the true value \(y\) and the estimated value (\(\hat{y}\)). Hence the model can be written as
$$y = f(x,\theta) +e$$ or
$$y = \hat{y} +e$$
When comes to the evaluation of a regression model, we are mostly interested in the error term \(e\) (or some time called the residuals). You may find later many of the metrics are built on top of error.
RMSE
RMSE stands for root mean squared error and it is literally the root of mean squared error:
$$RMSE = \sqrt{\frac{1}{K}\sum\limits_i e_i^2}$$
The RMSE summarizes how much error between the observations \(y\) and the estimation \(\hat{y}\).
Unlike recall and precision in classification, RMSE itself may not provide much information about the model’s performance, however, comparing RMSE of different models can help you decide which model to choose.
MAE
The idea of MAE (Mean absolute error) is similar to RMSE, but instead of squared errors, MAE employs absolute values of the error:
$$MAE = \frac{1}{K}\sum\limits_i |e_i|$$
MAE is designed to lower the effect of outliers.
This is because each error term in RMSE is being squared, hence an outlier (data point/observation with high error) will contribute a lot in the calculation. But in MAE instead of square, we take the absolute value of the error so the effect of outlier is reduced.
It maybe a little bit difficult to visualize why MAE has this effect. Let’s show it by numbers.
Consider the following example:
y | f(x) | e | |
Data point 1 | 1 | 1.5 | -0.5 |
Data point 2 | 2 | 3 | -1 |
Data point 3 | 3 | 1.7 | 1.3 |
Data point 4 | 4 | 4.2 | -0.2 |
Data point 5 | 5 | 4.5 | 0.5 |
Outlier | 6 | 20 | -14 |
RMSE and MAE with or without outliers can be easily calculated:
- RMSE with outlier = 7.06
- RMSE without outlier = 0.90
- MAE with outlier = 2.92
- MAE without outlier = 0.7
When we take the outlier into accounts, the RMSE is way larger than the MAE (~3x) but if we take away the outlier the RMSE and MAE are very similar.
R-Squared (\(R^{2}\))
Differ from RMSE and MAE, R-Squared does not only look at the error terms, it also take the variation in the data into account.
The R Squared value of a model is capturing “How much percentage of the variation of the data is captured by the model”.
It is defined as
$$R^{2}\equiv 1- \frac{ \sum\limits_i e_i^2 } {\sum\limits_i (y_i – \bar{y})^2}$$
It is a nice metric for regression due to its interpretability, but it sometimes also hide some problems of a model.
For instance you can have a biased model with a high R-squared value (99%). So don’t draw conclusion solely from R-squared value.