Machine learning does not always have to decide between two or three labels. Quite often, the problem is much more practical: What will the number be?
How much will a house cost? How many products could a store sell next month? What might the electricity demand look like tomorrow? How long could a delivery take?
These are regression problems.
Regression algorithms are designed to predict continuous numerical values from historical data. They sit behind many forecasting and predictive analytics systems, from financial modelling and sales forecasting to real estate pricing and demand planning.
There is no single regression algorithm that works for every dataset. A simple linear relationship may call for Linear Regression, while a messy dataset with nonlinear patterns may benefit from a tree-based model.
For anyone learning machine learning, six algorithms are particularly useful to understand.
1. Linear Regression
Linear Regression is usually where regression starts.
The idea is simple. The model tries to find a relationship between one or more input variables and a numerical target. With one variable, that relationship can be represented as a straight line.
For example, suppose a retailer wants to estimate monthly sales from advertising expenditure. If sales generally increase as advertising spending increases, Linear Regression can estimate the relationship and use it to predict future sales.
The basic equation is:
y = β₀ + β₁x + ε
With several input variables, the equation expands accordingly.
The model calculates coefficients that minimise the difference between its predictions and the actual values in the training data. Ordinary Least Squares is one of the standard approaches used for this purpose.
Linear Regression remains popular because it is fast, straightforward and relatively easy to explain. In a business environment, that last point matters. A model that produces a prediction is useful. A model whose reasoning can also be explained to decision-makers is often even more useful.
Still, Linear Regression has limits.
Real-world relationships are rarely perfectly straight. Outliers can also pull the fitted line away from where it would otherwise sit. When several predictors are strongly related to each other, coefficient estimates can become unstable.
That is where other regression methods start to earn their place.
2. Polynomial Regression
A straight line cannot describe every relationship.
Consider temperature and air-conditioning demand. Electricity consumption may remain fairly stable across a comfortable temperature range, then rise sharply as temperatures climb. A straight line may miss that curve.
Polynomial Regression introduces powers of the input variables to create a curved relationship.
A simple example is:
y = β₀ + β₁x + β₂x² + ε
Adding an x² term allows the model to bend instead of remaining completely straight. Higher-degree terms can create more complex curves.
The catch is obvious: more flexibility can also mean more trouble.
A polynomial with a very high degree may fit the training data almost perfectly while producing poor predictions on new data. In other words, the model starts learning the noise rather than the useful pattern.
That is classic overfitting.
Polynomial Regression works best when there is a reasonable reason to expect a curved relationship and the degree of the polynomial is kept under control. Cross-validation can help determine whether the added complexity actually improves prediction.
3. Ridge Regression
Datasets can contain a lot of information. That sounds helpful until several variables start saying almost the same thing.
Imagine a marketing dataset containing website visits, page views, sessions and several other traffic-related measurements. These variables may be strongly correlated. Ordinary Linear Regression can struggle with this type of multicollinearity.
Ridge Regression tackles the problem by adding L2 regularisation.
Instead of simply trying to minimise prediction error, the model also penalises large coefficients. The result is a model that tends to keep coefficient values smaller and more stable.
The strength of this penalty is controlled by a regularisation parameter, commonly represented by λ.
A larger value applies stronger shrinkage.
Ridge does not normally eliminate variables altogether. Instead, it pulls their coefficients toward zero. This makes the method useful when many predictors may contain information about the target and removing variables entirely is not desirable.
For machine learning learners, Ridge is worth understanding because it introduces a broader idea that appears repeatedly in modern modelling: regularisation.
The goal is not simply to fit the training data as closely as possible. The model also needs to behave sensibly when it encounters data it has never seen.
4. Lasso Regression
Lasso Regression takes regularisation in a slightly different direction.
Lasso stands for Least Absolute Shrinkage and Selection Operator. It uses L1 regularisation, which penalises the absolute values of model coefficients.
The interesting part is what can happen as a result.
Some coefficients can become exactly zero.
That effectively removes those variables from the model.
Suppose a dataset contains hundreds of potential predictors. Not all of them will contribute useful information. Lasso can shrink weaker predictors down to zero while retaining variables that contribute more strongly to the prediction.
This makes Lasso useful for both prediction and feature selection.
There is, however, a wrinkle.
When several variables are highly correlated, Lasso may favour one and reduce the coefficients of others. That selection is not necessarily proof that the chosen variable is inherently more meaningful. It can simply be a consequence of how the optimisation behaves with correlated predictors.
Ridge and Lasso therefore solve related problems in different ways:
- Ridge generally keeps all predictors but shrinks their coefficients.
- Lasso can remove some predictors by reducing their coefficients to zero.
- Ridge is often useful when many correlated predictors contribute to the target.
- Lasso can be useful when a smaller set of predictors is preferred.
Both methods are commonly paired with cross-validation to select an appropriate regularisation strength.
5. Decision Tree Regression
Decision Tree Regression does not try to fit one equation across the entire dataset.
Instead, it keeps asking questions.
- Is the house larger than a certain size?
- Is the location in a particular category?
- Is the number of bedrooms above a certain threshold?
Each split divides the data into smaller groups. Eventually, observations reach a leaf, where the model produces a numerical prediction.
The structure is easy to picture as a flowchart.
That is one reason decision trees are popular among machine learning practitioners. They can capture relationships that are difficult to represent with a straight line.
A decision tree can also pick up interactions between variables without requiring every interaction to be manually created.
But there is a catch.
Give a tree too much freedom and it can keep splitting until it practically memorises the training dataset. Training performance may look excellent while performance on unseen data falls sharply.
Tree depth, minimum samples per split and minimum samples per leaf are among the parameters used to control this behaviour.
Feature scaling is generally not required for decision trees, which is another practical advantage.
6. Random Forest Regression
A single decision tree can be unstable. Change the training data slightly and the resulting tree may look quite different.
Random Forest Regression approaches the problem by building many decision trees and combining their predictions.
Each tree is trained using a different bootstrap sample of the data, while random subsets of features are considered when determining splits. For a regression task, the predictions from the individual trees are generally averaged to produce the final result.
The logic is fairly intuitive: instead of trusting one tree, use a large group of them.
This often produces a more stable model than a single decision tree.
Random Forest Regression is particularly useful for tabular datasets containing nonlinear relationships, numerous predictors and interactions between features. It also requires less preprocessing than many traditional statistical approaches.
There are trade-offs. A forest containing hundreds of trees is harder to interpret than a small decision tree. Training and prediction can also require more computational resources.
Still, Random Forest is often a sensible model to test when a dataset does not appear to follow a simple mathematical relationship.
Which Regression Algorithm Should Be Used?
There is no universal winner.
That point gets lost surprisingly often when machine learning algorithms are discussed. A model that performs well on one dataset can perform poorly on another.
A useful starting point is:
| Regression algorithm | Typical use |
|---|---|
| Linear Regression | Straightforward linear relationships |
| Polynomial Regression | Curved relationships |
| Ridge Regression | Correlated predictors and regularised linear modelling |
| Lasso Regression | Regularisation with feature selection |
| Decision Tree Regression | Nonlinear relationships and feature interactions |
| Random Forest Regression | More complex tabular datasets |
The next step is testing.
Common regression metrics include Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and R². Each metric tells a slightly different story.
MAE, for example, gives an average absolute prediction error and is relatively easy to interpret. RMSE gives greater weight to larger errors. R² indicates how much of the variation in the target is explained by the model under the assumptions of the metric.
Cross-validation should also be part of the process when the dataset permits it. A model that looks impressive on training data has not proved much until it performs well on data outside that training set.
Final Thoughts
Regression is much bigger than fitting a line to a collection of points.
Linear Regression provides the basic foundation. Polynomial Regression introduces controlled curvature. Ridge and Lasso demonstrate how regularisation can improve linear models. Decision Trees handle nonlinear relationships through sequential splits, while Random Forest combines many trees to produce a more stable prediction.
These six algorithms cover several fundamentally different ways of approaching a regression problem.
For anyone learning machine learning, the real value comes from understanding why the algorithms behave differently.
Memorising formulas is useful, but knowing when a model is likely to overfit, when correlated features can cause trouble, and when a nonlinear method makes more sense is what turns algorithm knowledge into practical machine learning skill.
Also Read:
