链接：https://pan.baidu.com/s/1NfRQJUvxjETnaB4RFqpazg 

提取码：c9rq 

 

1.1 Introduction

Over the last couple of decades, multiple classifier systems, also called ensemble

systems have enjoyed growing attention within the computational intelligence and

machine learning community. This attention has been well deserved, as ensemble

systems have proven themselves to be very effective and extremely versatile in

a broad spectrum of problem domains and real-world applications. Originally

developed to reduce the variance—thereby improving the accuracy—of an auto￾mated decision-making system, ensemble systems have since been successfully

used to address a variety of machine learning problems, such as feature selection,

confidence estimation, missing feature, incremental learning, error correction, class￾imbalanced data, learning concept drift from nonstationary distributions, among

others. This chapter provides an overview of ensemble systems, their properties,

and how they can be applied to such a wide spectrum of applications.

Truth be told, machine learning and computational intelligence researchers have

been rather late in discovering the ensemble-based systems, and the benefits offered

by such systems in decision making. While there is now a significant body of

knowledge and literature on ensemble systems as a result of a couple of decades

of intensive research, ensemble-based decision making has in fact been around

and part of our daily lives perhaps as long as the civilized communities existed.

You see, ensemble-based decision making is nothing new to us; as humans, we

use such systems in our daily lives so often that it is perhaps second nature to us.

Examples are many: the essence of democracy where a group of people vote to

make a decision, whether to choose an elected official or to decide on a new law,

is in fact based on ensemble-based decision making. The judicial system in many

countries, whether based on a jury of peers or a panel of judges, is also based on

R. Polikar ()

Rowan University, Glassboro, NJ 08028, USA

e-mail: polikar@rowan.edu

C. Zhang and Y. Ma (eds.), Ensemble Machine Learning: Methods and Applications,

DOI 10.1007/978-1-4419-9326-7 1, © Springer Science+Business Media, LLC 2012

1

2 R. Polikar

ensemble-based decision making. Perhaps more practically, whenever we are faced

with making a decision that has some important consequence, we often seek the

opinions of different “experts” to help us make that decision; consulting with several

doctors before agreeing to a major medical operation, reading user reviews before

purchasing an item, calling references before hiring a potential job applicant, even

peer review of this article prior to publication, are all examples of ensemble-based

decision making. In the context of this discussion, we will loosely use the terms

expert, classifier, hypothesis, and decision interchangeably.

While the original goal for using ensemble systems is in fact similar to the reason

we use such mechanisms in our daily lives—that is, to improve our confidence that

we are making the right decision, by weighing various opinions, and combining

them through some thought process to reach a final decision—there are many

other machine-learning specific applications of ensemble systems. These include

confidence estimation, feature selection, addressing missing features, incremental

learning from sequential data, data fusion of heterogeneous data types, learning nonstationary environments, and addressing imbalanced data problems, among others.

In this chapter, we first provide a background on ensemble systems, including

statistical and computational reasons for using them. Next, we discuss the three pillars of the ensemble systems: diversity, training ensemble members, and combining

ensemble members. After an overview of commonly used ensemble-based algorithms, we then look at various aforementioned applications of ensemble systems as

we try to answer the question “what else can ensemble systems do for you?”

1.1.1 Statistical and Computational Justifications

for Ensemble Systems

The premise of using ensemble-based decision systems in our daily lives is

fundamentally not different from their use in computational intelligence. We consult

with others before making a decision often because of the variability in the past

record and accuracy of any of the individual decision makers. If in fact there were

such an expert, or perhaps an oracle, whose predictions were always true, we would

never need any other decision maker, and there would never be a need for ensemblebased systems. Alas, no such oracle exists; every decision maker has an imperfect

past record. In other words, the accuracy of each decision maker’s decision has

a nonzero variability. Now, note that any classification error is composed of two

components that we can control: bias, the accuracy of the classifier; and variance,

the precision of the classifier when trained on different training sets. Often, these

two components have a trade-off relationship: classifiers with low bias tend to have

high variance and vice versa. On the other hand, we also know that averaging has

a smoothing (variance-reducing) effect. Hence, the goal of ensemble systems is to

create several classifiers with relatively fixed (or similar) bias and then combining

their outputs, say by averaging, to reduce the variance.