DETECTION AND HANDLING OF OUTLIERS

Abstract

Abstract : Data sometimes include observations that are clearly separated from the bulk of the data. If we can determine the source that produces such anomalous results, we might be able to give an explanation for those values, such results may have arisen from a reading, recording, or calculating error in data, an instrument may have not been adjusted properly. If we are sure that a mistake had happened while collecting or recording the data, then the solution is simply to discard those anomalous values, however in many cases the sample outliers may contain important information and thus data should be investigated carefully before deciding whether to include or to exclude any anomalous values. This dissertation studies sources, nature, effects associated problems, ways of detection, accommodation and transformation of outliers present in any sample data, also the effect of outliers in regression. Generally speaking the aim is to address the following three major questions: 1- Do outliers exist in a sample data or not? 2- If yes, do we have to include or to exclude them? 3- If we have to include them, how to handle them? And how to minimize their effect on the results and on the conclusions. In chapter 1, we present a background of outliers together with several definitions of outliers as well as their sources. Also we discuss why outliers cause problems and how to detect them. Chapter 2 lists down all definitions and notations used in this dissertation. Chapter 3 presents methods for labeling of outliers, such as the z-score, the modified Z-score and the box plots method, as well as methods for outliers identification such as ESD single and many outlier procedures, Dixon type test, box plot rule, the L, test, Ek-statistic, sample kurtosis and the W- statistics. Chapter 4 discusses ways for accommodation of outliers in normally distributed samples. In accommodation of outliers we present the trimmed mean, the winsorized mean, the scale estimators and the confidence intervals. We also go beyond the normally distributed samples to handle data that follow, and we presented ways for transforming data to resemble a normal distribution such as the logarithm transformation and the square root method. In chapter 5, the effect of outliers on the fitted regression models was discussed and ways for identifying outliers from graphs such as scatter plots and residual plots and numerically were presented. All above methods were applied on sets of hypothetical data and the effect of outliers in each case was highlighted. Finally, we conclude that exploring of outliers is important and fundamental for any data analysis in all areas of study and that nearly all scientists and statisticians should be aware about outliers and shall pay particular attention to it, as it distort the results of any experiment and accordingly the conclusions which may be derived from it.