Ensemble Modeling with R

Ensemble methods are advanced techniques often used to solve complex machine learning problems. In simple terms, an ensemble method is a process where different and independent models (also referred to as the "weak learners") are combined to produce an outcome. The hypothesis is that combining multiple models can produce better results by decreasing generalization error.

Three of the most popular methods for ensemble modeling are bagging, boosting, and stacking. In this guide, you will learn how to implement these techniques with R.

In this guide, we will use a fictitious dataset of loan applicants containing 600 observations and 10 variables, as described below:

1. Marital_status: Whether the applicant is married ("Yes") or not ("No")

2. Is_graduate: Whether the applicant is a graduate ("Yes") or not ("No")

3. Income: Annual Income of the applicant (in USD)

4. Loan_amount: Loan amount (in USD) for which the application was submitted

5. Credit_score: Whether the applicant's credit score is good ("Good") or not ("Bad")

6. Approval_status: Whether the loan application was approved ("Yes") or not ("No")

7. Age: The applicant's age in years

8. Sex: Whether the applicant is a male ("M") or a female ("F")

9. Investment: Total investment in stocks and mutual funds (in USD) as declared by the applicant

10. Purpose: Purpose of applying for the loan

Let's start by loading the required libraries and the data.

Output:

The output shows that the dataset has five numerical variables (labeled as int) and five character variables (labeled as chr). We will convert these into factor variables using the line of code below.

Output:

The goal of ensemble modeling is to improve performance over a baseline model by combining multiple models. So, we will set the baseline performance measure by starting with one algorithm. In our case, we will build a logistic regression algorithm.

We will build our model on the training set and evaluate its performance on the test set. This is called the holdout-validation method for evaluating model performance.

The first line of code below sets the random seed for reproducibility of results. The second line loads the caTools package that will be used for data partitioning, while the third to fifth lines create the training and test datasets. The train set contains 70 percent of the data (420 observations of 10 variables) and the test set contains the remaining 30 percent (180 observations of 10 variables).

Output:

To fit the logistic regression model, the first step is to instantiate the algorithm, which is done using the first line of code below. The second line uses the trained algorithm to make predictions on the test data. The third line generates the confusion matrix, while the fourth line prints the accuracy of the test data.

Output:

We see that the accuracy of the single model is 88 percent. We will now build various ensemble models and see if they improves performance.

Bagging, or bootstrap aggregation, is an ensemble method that involves training the same algorithm many times by using different subsets sampled from the training data. The final output prediction is then averaged across the predictions of all the sub-models. The two most popular bagging ensemble techniques are Bagged Decision Trees and Random Forest.

This method performs best with algorithms that have high variance, for example, decision trees. We start by setting the seed in the first line of code. The second line specifies the parameters used to control the model training process, while the third line trains the bagged tree algorithm. The argument method="treebag" specifies the algorithm. The fourth to sixth lines of code generate predictions on the test data, create the confusion matrix, and compute the accuracy, which comes out to be 79 percent. This is not better than the logistic regression model.

Output:

Random Forest is an extension of bagged decision trees, where the samples of the training dataset are taken with replacement. The trees are constructed with the objective of reducing the correlation between the individual decision trees.

We follow the same steps as above, with the exception that while training the algorithm, we set method="rf" to specify that a random forest model is to be built. The accuracy of the random forest ensemble is 92 percent, a significant improvement over the models built earlier.

Output:

In boosting, multiple models are trained sequentially and each model learns from the errors of its predecessors. In this guide, we will implement a gradient boosting algorithm.

The boosting algorithm focuses on classification problems and aims to convert a set of weak classifiers into a strong one.

We follow the same steps as above, with exception that while training the algorithm, we set method="gbm" to specify that a gradient boosting model is to be built. The accuracy of the model is 78.9 percent, which is lower than the baseline accuracy.

Output:

In this approach, the predictions of multiple caret models are combined using the caretEnsemble package. Then, given the list of models, the caretStack() function is used to specify a higher-order model to learn how to best combine the predictions of sub-models together. The first line of code sets the seed, while the second line specifies the control parameters to be used while modeling. The third line uses the list of algorithms to use, while the fourth line uses the caretList() function to train the model. The sixth line prints the summary of the result. The output shows that the SVM algorithm creates the most accurate model, with a mean accuracy of 92.7 percent.

Output:

Let’s combine the predictions of the classifiers using a simple logistic regression model. The output shows that the accuracy is 91.7 percent.

Output:

In this guide, you have learned about ensemble modeling with R. The performance of the models implemented in the guide is summarized below:

1. Logistic Regression: Accuracy of 87.8 percent

2. Bagged Decision Trees: Accuracy of 78.9 percent

3. Random Forest: Accuracy of 91.7 percent

4. Stochastic Gradient Boosting`: Accuracy of 78.9 percent

5. Stacking: Accuracy of 91.7 percent

To learn more about data science using R, please refer to the following guides:

1. Interpreting Data Using Descriptive Statistics with R

2. Interpreting Data Using Statistical Models with R

3. Time Series Forecasting Using R

4. Hypothesis Testing - Interpreting Data with Statistical Models

5. Machine Learning with Text Data Using R

6. Visualization of Text Data Using Word Cloud in R

7. Exploring Data Visually with R

8. Coping with Missing, Invalid and Duplicate Data in R

9. Reshaping Data with R

10. Working with Data Types in R

11. Splitting and Combining Data with R

12. Linear, Lasso, and Ridge Regression with R