Assumptions of Linear Regression: 5 Assumptions With Examples

In statistical analysis, regression is a vital tool for uncovering cause-and-effect relationships. Personally, I find it to be indispensable in my work. At its core, regression analysis scrutinizes the interplay between observed patterns and the variables believed to influence them. By assessing the magnitude and direction of these relationships, it offers insights into the dynamics of various phenomena. Regression is used to gauge and quantify cause-and-effect relationships. For instance, if there is a 20% reduction in the price of a product, say, a moisturiser, people are likely to buy it, and sales are likely to increase.

Here, the observed pattern is an increase in sales (also called the dependent variable). The variable assumed to impact sales is the price (also called the independent variable). 

In our exploration of Assumptions of Linear Regression, we aim to shed light on five fundamental principles underlying this technique. Through examples, we’ll illustrate how these Assumptions of Linear Regression guide our understanding of causal relationships and shape the interpretation of regression results. By grasping these foundational concepts, analysts like me can wield regression with confidence, extracting meaningful insights from complex datasets.

Enrol for the Machine Learning Course from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

What Is Linear Regression?

Linear regression is a statistical technique that models the magnitude and direction of an impact on the dependent variable explained by the independent variables. Linear regression is commonly used in predictive analysis.

Linear regression explains two important aspects of the variables, which are as follows:

• Does the set of independent variables explain the dependent variable significantly?
• Which variables are the most significant in explaining the dependent available? In which way do they impact the dependent variable? The impact is usually determined by the magnitude and the sign of the beta coefficients in the equation.

Now, let’s look at the assumptions of linear regression, which are essential to understand before we run a linear regression model.

Read more: Linear Regresison Model & How it works?

Assumptions of Linear Regression

Linear relationship

One of the most important assumptions is that a linear relationship is said to exist between the dependent and the independent variables. If you try to fit a linear relationship in a non-linear data set, the proposed algorithm won’t capture the trend as a linear graph, resulting in an inefficient model. Thus, it would result in inaccurate predictions.

FYI: Free nlp course!

How can you determine if the assumption is met?

The simple way to determine if this assumption is met or not is by creating a scatter plot x vs y. If the data points fall on a straight line in the graph, there is a linear relationship between the dependent and the independent variables, and the assumption holds.

What should you do if this assumption is violated?

If a linear relationship doesn’t exist between the dependent and the independent variables, then apply a non-linear transformation such as logarithmic, exponential, square root, or reciprocal either to the dependent variable, independent variable, or both. 

No auto-correlation or independence

The residuals (error terms) are independent of each other. In other words, there is no correlation between the consecutive error terms of the time series data. The presence of correlation in the error terms drastically reduces the accuracy of the model. If the error terms are correlated, the estimated standard error tries to deflate the true standard error.

How to determine if the assumption is met?

Conduct a Durbin-Watson (DW) statistic test. The values should fall between 0-4. If DW=2, no auto-correlation; if DW lies between 0 and 2, it means that there exists a positive correlation. If DW lies between 2 and 4, it means there is a negative correlation. Another method is to plot a graph against residuals vs time and see patterns in residual values.

What should you do if this assumption is violated?

If the assumption is violated, consider the following options:

• For positive correlation, consider adding lags to the dependent or the independent or both variables.
• For negative correlation, check to see if none of the variables is over-differenced.
• For seasonal correlation, consider adding a few seasonal variables to the model.

No Multicollinearity

The independent variables shouldn’t be correlated. If multicollinearity exists between the independent variables, it is challenging to predict the outcome of the model. In essence, it is difficult to explain the relationship between the dependent and the independent variables. In other words, it is unclear which independent variables explain the dependent variable.

The standard errors tend to inflate with correlated variables, thus widening the confidence intervals leading to imprecise estimates.

How to determine if the assumption is met?

Use a scatter plot to visualise the correlation between the variables. Another way is to determine the VIF (Variance Inflation Factor). VIF<=4 implies no multicollinearity, whereas VIF>=10 implies serious multicollinearity.

What should you do if this assumption is violated?

Reduce the correlation between variables by either transforming or combining the correlated variables.

Must Read: Types of Regression Models in ML

Homoscedasticity

Homoscedasticity means the residuals have constant variance at every level of x. The absence of this phenomenon is known as heteroscedasticity. Heteroscedasticity generally arises in the presence of outliers and extreme values.

How to determine if the assumption is met?

Create a scatter plot that shows residual vs fitted value. If the data points are spread across equally without a prominent pattern, it means the residuals have constant variance (homoscedasticity). Otherwise, if a funnel-shaped pattern is seen, it means the residuals are not distributed equally and depicts a non-constant variance (heteroscedasticity).

What should you do if this assumption is violated?

• Transform the dependent variable
• Redefine the dependent variable
• Use weighted regression

Normal distribution of error terms

The last assumption that needs to be checked for linear regression is the error terms’ normal distribution. If the error terms don’t follow a normal distribution, confidence intervals may become too wide or narrow.

How to determine if the assumption is met?

Check the assumption using a Q-Q (Quantile-Quantile) plot. If the data points on the graph form a straight diagonal line, the assumption is met.

You can also check for the error terms’ normality using statistical tests like the Kolmogorov-Smironov or Shapiro-Wilk test.

What should you do if this assumption is violated?

• Verify if the outliers have an impact on the distribution. Make sure they are real values and not data-entry errors.
• Apply non-linear transformation in the form of log, square root, or reciprocal to the dependent, independent, or both variables.

Number of observations Greater than the number of predictors  

The number of training data, or observations, must always exceed the number of tests or prediction data for the model to perform well. On the other hand, the performance of the model improves with more data. When there are more predictors than observations, the model becomes over-parameterized and causes problems like overfitting. 

Let’s say we are forecasting home values using characteristics like size, number of bedrooms, and demographics of the surrounding area. The assumption is satisfied if there are 1000 houses(observations) and five predictor variables. 

How to Determine if the Assumption is Met? 

To verify this assumption, count the number of observations (n) and the number of predictors (p). Ensure that n>p. 

What Should You Do if This Assumption is Violated? 

If the number of predictors exceeds the number of observations, consider reducing the number of predictors through feature selection techniques like forward selection, backward elimination, or dimensionality reduction methods such as principal component analysis (PCA). 

Each observation is unique 

Linear regression assumes that each observation in the dataset is independent and identically distributed (IID). This means that the error terms (ε) associated with each observation are not correlated with each other. 

Suppose in a study measuring the effects of a new drug on patients, each patient’s response to the drug is considered an independent observation. 

How to Determine if the Assumption is Met? 

To assess independence, examine whether there are any patterns or correlations in the residuals (errors) of the model. A plot of residuals against the predicted values should show random scattering around zero, indicating independence. 

What Should You Do if This Assumption is Violated? 

If the assumption of independence is violated, it suggests that there may be hidden patterns or dependencies in the data. Consider collecting more data, accounting for any inherent dependencies, or using techniques like time-series analysis for temporal data.

Conclusion

Understanding the assumptions of linear regression is pivotal for anyone looking to harness the power of this fundamental statistical tool. These assumptions form the bedrock upon which reliable, valid, and interpretable linear regression analyses are built. By meticulously examining and ensuring the adherence to these assumptions with practical examples, researchers and data scientists can significantly improve the accuracy and applicability of their findings. It is through this rigorous adherence to the foundational principles of linear regression that one can unlock the full potential of their data, leading to insights that are both profound and actionable. Remember, the strength of any linear regression analysis lies not just in the application of the technique but in the careful consideration and validation of its underlying assumptions. As we continue to navigate through vast seas of data in various fields, the role of these assumptions will remain undeniably central to achieving clarity, precision, and effectiveness in our analytical endeavors.

If you’re interested to learn more about regression models and more of machine learning, check out IIIT-B & upGrad’s PG Diploma in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.