Regression Analysis

Regression analysis is a fundamental statistical technique in research, widely used across disciplines to examine relationships between variables. Regression is used to understand relationships between variables and helps researchers determine the nature and strength of the relationship between independent (predictor) and dependent (outcome) variables.

With regression researchers can predict patterns based on existing data and observations. This statistical technique allows for control of confounding variables or for multiple factors simultaneously, reducing bias. Regression is optimized for testing hypotheses determining whether relationships between variables are statistically significant.

Using regression, you can assess impacts. The model coefficients provide estimates of how much a change in one variable influences another. The robustness of regression techniques enables handling of complex data structures through construction of advanced regression models such as multiple regression, logistic regression, and panel data regression even in cases of nonlinear relationships and interactions between variables.

What is Regression Analysis?

Regression analysis is a statistical technique in quantitative data analysis for examining the relationship between two or more variables. These variables are usually labelled as independent and dependent. Determining how to model the variables as dependent and independent is based on a supposition that one or more variables are directly influenced by the other variables in the dataset. Therefore, a regression model evaluates whether one or more independent variables influence one or multiple dependent variables.

A simple regression model includes one independent and one dependent variable. Alternatively, a regression model that examines the impact of more than one independent variable on one dependent variable is known as a multiple regression model. 

Regression analysis is a reliable method of identifying which variables impact a topic of interest. Using regression analysis, a researcher can confidently determine which factors matter most, which can be ignored, and how these factors influence each other. Like other inferential analyses, regression analysis draws a sample from a population of interest and uses it to understand the characteristics of the population.

Many dissertations or theses adopt either a simple or multiple regression model to investigate the influence of one or more independent variables on a dependent variable. But, in multivariate regression models, researchers evaluate the impact of independent variables on multiple correlated dependent variables in a single model. 

While simple and multiple regression analyses focus on the conditional probability distribution of the dependent variable given the values of the independent variables, multivariate analyses look at the joint probability distribution of all variables.

When considering a regression analysis in a dissertation or thesis, researchers should: 

• Identify the appropriate regression model based on the data type and context of the research question.
• Check the assumptions for the selected model before conducting the regression.
• Summarize regression statistics like model fit, parameter estimates (regression coefficients, confidence intervals, and p-values) in tables.
• Interpret the results correctly, answering the research hypotheses and questions.

This article comprehensively covers the most common types of regression analyses used in dissertations or theses, focusing on univariate regression (one dependent variable).

What is the Purpose of Regression Analysis?

By conducting a regression analysis, researchers can:

• Identify causal relationships: Researchers can determine which independent variables significantly influence the dependent variable and which do not have an impact by assessing their p-values.
• Compare variables: In regression analysis, the coefficients of the independent variables provide the direction and extent of their impact on the dependent variable. This helps researchers to identify the most influential variables on the outcome of interest.
• Control Variables: Regression analysis can be used to control the effect of other variables that may influence the relationship between the independent variable(s) and the dependent variable. Such variables are known as confounding or control variables.
• Predict an outcome: Using the coefficient estimates obtained in regression analysis, researchers can use the model to predict the values of thedependent variable.
• Validate models: The model’s expected values can be compared to actual values, thereby assessing its accuracy.

Regression Model Building

A regression analysis formulates a mathematical model that describes the relationship between the variables. This relationship is represented as a line or curve in simple regression and hyperplane when conducting a multiple regression. Based on the collected data, a regression model aims to find the best-fit line that accurately represents the relationship between the variables in the data. This line minimizes the prediction error which is the distance between the observed data points and the predicted values from the model.

Depending on the characteristics of the data and research aim, different types of regression analyses can be conducted. Each type of regression analysis is founded on specific assumptions that must be met for a regression model to be reliable and valid.

Types of Regression Analysis

Linear Regression

Linear regression is a statistical model that evaluates the linear relationship between a continuous dependent variable and one or more independent variables. The independent variables can be measured on a continuous or categorical scale.

Generally, the concept of regression analysis began with modeling linear regression and has been extensively used in practical applications. This is because linear relationships are easier to fit than those that are non-linear. Basically, it is easier to determine the statistical properties of the resulting estimators in linear regression.

Often, linear regression models use the ordinary least squares approach to fit the parameters but other methods can be applied as well. These methods focus on minimizing different cost functions depending on the arrangement of the data. For example, least absolute deviations regression aims to minimize the “lack of fit” whereas lasso and ridge regression models minimize L1-norm penalty and L2-norm penalty respectively, which are penalized versions of the least squares cost function. Note that the least squares method can also be used to model non-linear relationships. Therefore, linear model and least squares are closely linked but are not the same.

While linear models typically fit a straight line, they can also fit curves, depending on the linear regression equation. Specifically, linear models can fit simple curvature relationships in the data but not complex curves. 

Important fact: The defining characteristic of linear regression is the functional form of the parameters rather than the ability to model curvature. Below is a definition of a linear regression model that clarifies this aspect.

A regression model is linear when all the terms in the model are either a constant or a parameter multiplied by an independent variable.

This implies that a linear model equation is formed by adding the terms. This limits the linear regression model to the form represented in Equation 1 – 1 below (for a simple linear model with one independent variable (IV)):

DV = constant + estimated parameter₁ * IV₁ + Random error ………….Equation 1 – 1

Curvilinear Models

Linear models can include nonlinear variables such as polynomials, exponential functions, or other functions to model curvature. This happens when it is necessary to transform the variables to meet the linear regression assumptions (OLS estimation assumptions).  Applying such remedial measures modifies the initial linear regression model to a curvilinear model.

For example, assuming a linear relationship between n independent variables (X₁ … X_n) and one dependent variable (Y) where the independent variable X₁ necessitatedtransformation to a squared power to satisfy the linear regression assumptions. This results in a linear regression model that is curved and can be represented as Equation 1 – 2 below:

Y = β₀ + β₁ X₁ ²+ β₂X₂ + … + β_nX_n + ε …………………….Equation 1 – 2

Linear regression analysis allows researchers to apply a simple and interpretable model to answer complex research questions by examining relationships between variables. Computationally, a linear model is efficient, hence the preferred technique for most researchers before opting for advanced nonlinear modelling methods. 

Due to its clear framework for gauging the impact of independent variables on dependent variables, researchers use linear regression to understand and predict diverse real-world phenomena in social sciences, economics, and biology.

Types of Linear Regression Models

Simple Linear Regression

A simple linear regression model evaluates the relationship between exactly two variables, one independent variable and a continuous dependent variable.

Multiple Linear Regression

A multiple linear regression model evaluates the relationship between multiple independent variables and a continuous dependent variable.

Multivariate Linear Regression

A multivariate linear regression model evaluates the relationship between multiple independent variables and multiple continuous dependent variables that are correlated.

Assumptions of Linear Regression (OLS estimation)

When the linear regression model satisfies the OLS assumptions, the method generates the best possible estimates that are close to the actual population values. According to the Gauss-Markov theorem, OLS estimation generates better estimates (regression coefficients) than all other linear model estimation methods when the assumptions hold.

Violations of these assumptions require the researcher to employ corrective measures or apply other appropriate estimation methods to ensure the results are reliable and valid.

Most of the assumptions in OLS estimation relate to the error term properties. While the error term is an unknown population value, residuals are used. Residuals represent the error for each observation in a sample and calculated as shown in Equation 1 – 4 below:

Residuals = Observed value – Expected (fitted or predicted) value ……Equation 1 – 4

In the following section, we explore six classical OLS assumptions for a linear regression model to generate the best estimates.

Linearity

The first assumption to check is that the relationship between the independent and dependent variables is linear. It means that the change in the independent variable(s) is directly proportional to the change in the dependent variable.

A researcher can check for linearity between each set of the independent and dependent variables. This is usually assessed by creating scatter plots of each continuous independent variable and the dependent variable. If the points represent a straight line, it indicates a linear relationship, and the assumption is met.

In linear regression, linearity is not assessed when handling categorical independent variables, it is assumed that the effect of each category on the dependent variable is constant. This is because linearity assumes a constant effect.

Violating this assumption may underfit the data and lead to inaccurate estimations. Therefore, for nonlinear relationships, either transform the independent or dependent variable using log, reciprocal, or square root to ensure the assumption is met or apply nonlinear regression models.

The error term has a population mean of zero

The error term in a regression model accounts for the unexplained variation in the dependent variable. The independent variables do not explain this variation and it constitutes the factors impacting the dependent variable but are not included in the model.

The error terms should have a mean of zero to ensure the model predictions are correct on average, meaning unbiased. This implies that the model’s predictions are neither systematically overpredicted nor underpredicted.

Picture a model with a mean error of -10. The negative mean error implies the model consistently overpredicts the actual values. Such systematic error is known as bias and signifies an inadequate model. A non-zero mean error or expected value suggests that part of the error is predictable and should be incorporated into the model. The error term should include only the random error. Fortunately, including a constant term (intercept) in the regression model ensures a zero-mean error.

This assumption can be examined by looking at the Residual vs Fitted plot. The graph should show residual randomness to satisfy the assumption.

Independence of Error Terms

The residual of one observation should be independent of the residual of another observation. It implies that a correlation between the residuals violates this assumption. When the residuals correlate, it suggests that the estimated coefficients are biased due to an overlooked pattern in the data. This can lead to misleading model results and interpretation.

Time series data often violates this assumption, where errors of previous time points influence errors of subsequent time points.

The Durbin-Watson test is a statistical test for detecting the presence of autocorrelation in the residuals. The value ranges from 0 to 4. A value less than 1.5 signals a negative serial correlation, and a value greater than 2.5 signals a positive serial correlation. A value between 1.5 and 2.5 indicates no autocorrelation.

For positive serial correlation, transform the dependent or independent variable using lags. Researchers can use distributed lag models that incorporate the current and lagged values of the dependent and independent variables respectively. For negative serial correlation, check for an over differenced variable. Adding relevant variables to capture this information can help eliminate the presence of autocorrelation. 

Homogeneity of Variance (Homoscedasticity) 

Homoscedasticity refers to residuals having constant variance across all observations or the levels of the independent variables. Simply, it means that the errors should be evenly distributed along the range of the predictor variable(s). This indicates that the model’s predictions are reliable across the levels of the predictor(s).

Conversely, heteroscedasticity describes changing error variance at the independent variable’s levels leading to inaccurate coefficient estimates and unreliable significance tests.

A scatterplot of the model’s fitted values vs. the fitted values residuals can be used to check for heteroscedasticity. Heteroscedasticity is observed when the spread of the residuals increases in one direction. An even spread of the residuals around the horizontal axis indicates constant variance. 

The Breusch-Pagan Test for Heteroscedasticity

The Breusch-Pagan test is an alternative way to test whether or not heteroscedasticity exists in a regression model.

Below we define the null and alternative hypotheses for the test:

• Null Hypothesis (H₀): Homoscedasticity is present (the residuals are distributed with equal variance).
• Alternative Hypothesis (H_A): Homoscedasticity is absent (Heteroscedasticity, the residuals are not distributed with equal variance).

If the test results show a p-value less than a specified significance level such as α = .05, then the null hypothesis is rejected and the study concludes that heteroscedasticity is present in the regression model.

Since the error term is expected to be random, it should not correlate with any other variable except the dependent variable as the error term always explains some variability in the dependent variable.

The Breusch-Pagan test assumes that when heteroscedasticity is present, the variance of the error term is correlated to the independent variables in the model. Thus, homoscedasticity implies the independent variable(s) are uncorrelated with the error term. This is so because extraneous variables with zero correlation are unbiased even when not controlled in a model.

Alternatively, when there is a correlation, OLS estimation may attribute some variance explained by the error to the independent variable which biases the coefficient estimate. The strength of the correlation is directly proportional to the extent of bias. Consequently, strong correlations lead to greater bias whereas weak correlations may have a negligible bias.

The following steps aid to perform the Breusch-Pagan test:

1. Fit the linear regression model and save the residuals.
2. Square the residuals and save them as a new variable i.e. sum of squared residuals
3. Regress the sum of squared residuals on the independent variables from the initial model.
4. Test the significance of the coefficients in the sum of squared residuals regression model.

For p-values of the coefficients that are below 0.05 significance level, the test suggests the presence of heteroscedasticity in the initial model.

Note: Breusch-Pagan test is sensitive to the presence of multicollinearity in the model. Therefore, it is best to check for multicollinearity and address it before performing the test.

Other Tests for Heteroscedasticity

Other tests that examine heteroscedasticity include the White test and the Goldfeld-Quandt test. The White test is a general test that detects more complex patterns of heteroscedasticity. The Goldfeld-Quandt test, commonly used in econometrics, divides the data into two groups based on the values of an independent variable and checks whether the variances of the residuals in the two groups are equal.

Deviations from this assumption are observed when the dependent and independent variables occur concurrently, there is omitted variable bias, or the independent variables have measurement error.

Solutions to prevent violations of this assumption include using random assignment, limiting the study population to specific groups, or controlling for a confounding variable in the model to incorporate its influence on the dependent variable.

Address heteroscedasticity by transforming the independent or dependent variables using logarithmic or square root functions to stabilize the variance.

Alternatively, use weighted least squares (WLS) which assigns weights to observations based on their variance, giving more weight to observations with lower variance. 

It is important to note that formal tests are sensitive to non-normality, hence visual inspection is usually adopted as an informative way to detect heteroscedasticity.

No Multicollinearity

Multicollinearity is checked when there is more than one independent variable in a linear model. The independent variables should not be highly correlated with one another. When the independent variables are correlated, multicollinearity exists, which makes it difficult for OLS estimation to differentiate or accurately interpret the influence of correlated variables. 

The variance inflation factor (VIF) is used to detect multicollinearity. Statistical software computes a VIF value for each independent variable when running the linear regression. The VIF values range from one and have no upper limit. A VIF value greater than 10 signals multicollinearity, greater than five indicates strong correlation, less than five shows moderate correlation, and approximately one signals no correlation.

Multicollinearity may lead to very sensitive regression coefficients that change wildly to minimal model modifications. Also, multicollinearity reduces the accuracy of the calculated coefficients which weakens the power of the regression model resulting to questionable p-values. It is thus critical to address multicollinearity issues in the model.