Controversies Around the Role of Statistical Tests in Experimental Research

Despite widespread use of significance testing in empirical research, its interpretation and researchers' excessive confidence in its results have been criticized for years. In this article, the logic of statistical testing in the Fisher and Neyman-Pearson approaches are described, some common misinterpretations of basic concepts behind statistical tests are reviewed, and the philosophical and psychological issues that can contribute to these misinterpretations are analyzed. Some frequent criticisms against statistical tests are revisited, with the conclusion that most of them refer not to the tests themselves but to the misuse of tests on the part of researchers. In accordance with Levin (1998a), statistical tests should be transformed into a more intelligent process that helps researchers in their work. Possible ways in which statistical education might contribute to the better understanding and application of statistical inference are suggested.     

Panel模型中的精确检验和置信区间

利用广义p-值和广义置信区间的概念,研究了Panel模型中未知参数的检验和置信区间问题.对于回归系数,分别考虑了单个情形和多个线性无关情形下的检验和置信区间问题,得到了精确检验和置信区间.对于方差分量,研究了其任意线性组合的检验和置信区间问题,建立了精确检验和置信区间.基于广义p-值和广义置信区间,获取精确检验和置信区间的方法具有计算方便、易应用于小样本问题的特点.最后,分别从理论和数值上研究了这些精确检验和置信区间的统计性质.     

The relative performance of bivariate causality tests in small samples

Causality tests have been applied to establish directional effects and to reduce the set of potential predictors. For the latter type of application only bivariate tests can be used. In this study we compare bivariate causality tests. Although the problem addressed in general and could benefit researchers from different fields, most attention is given to marketing applications. Even though there are many alternative tests, applications in marketing have almost exclusively been based on the Haugh—Pierce test. We compare five bivariate tests in a specific marketing application. The empirical results indicate that conclusions about causality may depend strongly on the test used. To provide generalizable insights about the relative performances of alternative tests we conduct a simulation study with data characteristics that cover the range of conditions encountered by researchers who have applied causality tests in marketing. We find that the Granger—Wald test has the highest power but also the greatest upward bias in alpha (the probability of a type-I error). If causality testing is done for the purpose of selecting a good subset of the available predictors, this combination of high power and high alpha may be attractive. For researchers desiring a simple test with a substantial amount of power and little upward bias in alpha we recommend the Granger—Sargent test. Interestingly, neither of these Granger tests has been used in marketing.     

假设检验的相对稳定性

郑祖康^[1]提出了数理统计学中一个有意义的新问题－假设检验的稳定性。但是,由于稳定性定义的条件太强,从而通常检验统计量的稳定性并不好。本文根据假设检验的实际意义,定义了假设检验的相对稳定性,讨论了一类常用检验统计量的相对稳定性的一些性质,作为特例,得到似然比检验的相对稳定性的性质。     

Cryptographic pseudo-random sequence from the spatial chaotic map

A scheme for pseudo-random binary sequence generation based on the spatial chaotic map is proposed. In order to face the challenge of using the proposed PRBS in cryptography, the proposed PRBS is subjected to statistical tests which are the well-known FIPS-140-1 in the area of cryptography, and correlation properties of the proposed sequences are investigated. The proposed PRBS successfully passes all these tests. Results of statistical testing of the sequences are found encouraging. The results of statistical tests suggest strong candidature for cryptographic applications.     

Formal Hypothesis Tests for Additive Structure in Random Forests

While statistical learning methods have proved powerful tools for predictive modeling, the black-box nature of the models they produce can severely limit their interpretability and the ability to conduct formal inference. However, the natural structure of ensemble learners like bagged trees and random forests has been shown to admit desirable asymptotic properties when base learners are built with proper subsamples. In this work, we demonstrate that by defining an appropriate grid structure on the covariate space, we may carry out formal hypothesis tests for both variable importance and underlying additive model structure. To our knowledge, these tests represent the first statistical tools for investigating the underlying regression structure in a context such as random forests. We develop notions of total and partial additivity and further demonstrate that testing can be carried out at no additional computational cost by estimating the variance within the process of constructing the ensemble. Furthermore, we propose a novel extension of these testing procedures using random projections to allow for computationally efficient testing procedures that retain high power even when the grid size is much larger than that of the training set.     

An Approximate Distribution for the Maximum Order Complexity

In this paper we give an approximate probability distribution for the maximum order complexity of a random binary sequence. This enables the development of statistical tests based on maximum order complexity for the testing of a binary sequence generator. These tests are analogous to those based on linear complexity.     

The use of analysis of variance procedures in biological studies

The analysis of variance (ANOVA) is widely used in biological studies, yet there remains considerable confusion among researchers about the interpretation of hypotheses being tested. Ambiguities arise when statistical designs are unbalanced, and in particular when not all combinations of design factors are represented in the data. This paper clarifies the relationship among hypothesis testing, statistical modelling and computing procedures in ANOVA for unbalanced data. A simple two-factor fixed effects design is used to illustrate three common parametrizations for ANOVA models, and some associations among these parametrizations are developed. Biologically meaningful hypotheses for main effects and interactions are given in terms of each parametrization, and procedures for testing the hypotheses are described. The standard statistical computing procedures in ANOVA are given along with their corresponding hypotheses. Throughout the development unbalanced designs are assumed and attention is given to problems that arise with missing cells.     

Comparative analysis of the power of goodness-of-fit tests for near competing hypotheses. II. Verification of complex hypotheses

In this part of the article, we use the statistical modeling methods to analyze the power of a series of goodness-of-fit tests for complex hypotheses. We estimate the power of tests relative to some near competing hypotheses. The combination of results enables us to order the tests with respect to their power for testing both simple and complex hypotheses.     

Undertaking data analysis of student outcomes as professional development for teachers

The study reports on collaborations with practitioners to examine the results of students’ performances on high stakes tests as a means to strengthen practitioners’ knowledge of probability and statistics and to empower their conduct of investigations on student performance. Four issues are summarized: the development of their statistical reasoning, their understanding of the meaning of and relationships among the concepts of validity, reliability and fairness as applied to testing, their introduction to the history of testing and its relationship to science, society and cultural inequality, and their reports of independent inquiries. Data on performance on pre- and post-tests demonstrate growth in teacher reasoning and in their professionalism in raising important issues about testing     

Business failure prediction using the UTADIS multicriteria analysis method

Business failure prediction is one of the most essential problems in the field of financial management. The research on developing quantitative business failure prediction models has been focused on building discriminant models to distinguish among failed and non-failed firms. Several researchers in this field have proposed multivariate statistical discrimination techniques. This paper explores the applicability of multicriteria analysis to predict business failure. Four preference disaggregation methods, namely the UTADIS method and three of its variants, are compared to three well-known multivariate statistical and econometric techniques, namely discriminant analysis, logit and probit analyses. A basic (learning) sample and a holdout (testing) sample are used to perform the comparison. Through this comparison, the relative performance of all the aforementioned methods is investigated regarding their discriminating and predicting ability.     

Some Approximate Statistical Tests

This paper enumerates and discusses a number of quick, short-cut statistical tests that are available for two common problems. The first problem is testing the difference between the means of samples drawn from two populations whilst the second is testing a sequence of observations to see whether there is a trend present.     

The fuzzy decision problem: An approach to the problem of testing statistical hypotheses with fuzzy information

This paper is devoted to the problems of testing statistical hypotheses about an experiment, when the available information from its sampling is `vague'. When the information supplied by the experimental sampling is exact, the problems of testing statistical hypotheses about the experiment can be regarded as a particular statistical decision problem. In addition, decision procedures may be used in problems of testing hypotheses.In a similar manner, the problem of testing statistical hypotheses about an experiment when the available sample information is vague, is approached in this paper as a particular fuzzy decision problem (as defined by Tanaka, Okuda and Asai). This approach assumes that the previous information about the experiment can be expressed by means of certain conditional probabilistic information, whereas the present information about it can be expressed by means of fuzzy information. The preceding framework allows us to extend the notion of risk function and some nonfuzzy decision procedures to the fuzzy case, and particularize them to the problem of testing.Finally, several illustrative examples are presented.     

罕见变异关联分析中的统计方法研究进展

二代测序数据的持续增多以及全基因组关联分析存在着只关注常见变异对表型影响的缺陷,使得研究人员开始考虑罕见变异对表型表达的影响。近年来,涉及罕见变异关联分析的统计方法研究成为一个活跃的领域,大批统计检验方法被相继提出。然而,大多数方法没有得到全面详细的比较和分析。因此,缺乏对现有方法的评价以及对它们如何使用的指导。本文对该领域的一些具有代表性的方法做一全面的综述,并介绍一些最新的研究进展。     

Statistical inference on mixing proportion

In this paper, the interval estimation and hypothesis testing of the mixing proportion in mixture distributions are considered. A statistical inferential method is proposed which is inspired by the generalized p-values and generalized pivotal quantity. In some situations, the true levels of the tests given in the paper are equal to nominal levels, and the true coverage of the interval estimation or confidence bounds is also equal to nominal one. In other situations, under mild conditions, the tests are consistent and the coverage of the interval estimations or the confidence bounds is asymptotically equal to nominal coverage. Meanwhile, some simulations are performed which show that our method is satisfactory.     

Model selection in Neural Networks: Some difficulties

This paper considers two related issues regarding feedforward Neural Networks (NNs). The first involves the question of whether the network weights corresponding to the best fitting network are unique. Our empirical tests suggest an answer in the negative, whether using standard Backpropagation algorithm or our preferred direct (non-gradient-based) search procedure. We also offer a theoretical analysis which suggests that there will almost inevitably be functional relationships between network weights. The second issue concerns the use of standard statistical approaches to testing the significance of weights or groups of weights. Treating feedforward NNs as an interesting way to carry out nonlinear regression suggests that statistical tests should be employed. According to our results, however, statistical tests can in practice be indeterminate. It is rather difficult to choose either the number of hidden layers or the number of nodes on this basis.     

Statistical inference in non-nested econometric models

The purpose of this paper is to discuss some procedures that are available for testing non-nested (or separate) hypotheses in the statistics and econometrics literature. Since many of these techniques may also be exploited in other disciplines, it is hoped that an elaboration of the principal theoretical findings may make them more readily accessible to researchers in other disciplines. Several simple examples are used to illustrate the concepts of nested and non-nested hypotheses and, within the latter category, “global” and “partial” non-nested hypotheses. Two alternative methods of testing non-nested hypotheses are discussed and contrasted: the first of these is Cox's modification of the likelihood-ratio statistic, and the second is Atkinson's comprehensive model approach. A major emphasis is placed on the role of the Cox principle of hypothesis testing, which enables a broad range of hypotheses to be tested within the same framework. The problem associated with the application of the comprehensive model approach to composite non-nested hypotheses is also highlighted; Roy's union-intersection principle is presented as a viable method of dealing with this problem. Simulation results concerning the finite-sample properties of various tests are discussed, together with an analysis of some attempts to correct the poor size of the Cox and related tests.     

Nonparametric Methods for Checking the Validity of Prior Order Information

A large number of statistical procedures have been proposed in the literature to explicitly utilize available information about the ordering of treatment effects at increasing treatment levels. These procedures are generally more efficient than those ignoring the order information. However, when the assumed order information is incorrect, order restricted procedures are inferior and, strictly speaking, invalid. Just as any statistical model needs to be validated by data, order information to be used in a statistical analysis should also be justified by data first. A common statistical format for checking the validity of order information is to test the null hypothesis of the ordering representing the order information. Parametric tests for ordered null hypotheses have been extensively studied in the literature. These tests are not suitable for data with nonnormal or unknown underlying distributions. The objective of this study is to develop a general distribution-free testing theory for ordered null hypotheses based on rank order statistics and score generating functions. Sufficient and necessary conditions for the consistency of the proposed general tests are rigorously established.     

Comparing several population means: a parametric bootstrap method, and its comparison with usual ANOVA F test as well as ANOM

This paper deals with testing the equality of several homoscedastic normal population means. We introduce a newly developed computational approach test (CAT), which is essentially a parametric bootstrap method, and discuss its merits and demerits. In the process of studying the CAT’s usefulness, we compare it with the traditional one-way ANOVA’s F test as well as the analysis of means (ANOM) method. Further, the model robustness of the above three methods have been studied under the ‘t-model’. The motivation behind the proposed CAT is to provide the applied researchers a statistical tool to carry out a comparison of several population means, in a parametric setup, without worrying about the sampling distribution of the inherent test statistic. The CAT can be used to test the equality of several means when the populations are assumed to be heteroscedastic t-distributions.     

Practical powerful wavelet packet tests for second-order stationarity

Methods designed for second-order stationary time series can be misleading when applied to nonstationary series, often resulting in inaccurate models and poor forecasts. Hence, testing time series stationarity is important especially with the advent of the ‘data revolution’ and the recent explosion in the number of nonstationary time series analysis tools. Most existing stationarity tests rely on a single basis. We propose new tests that use nondecimated basis libraries which permit discovery of a wider range of nonstationary behaviours, with greater power whilst preserving acceptable statistical size. Our tests work with a wide range of time series including those whose marginal distributions possess heavy tails. We provide freeware R software that implements our tests and a range of graphical tools to identify the location and duration of nonstationarities. Theoretical and simulated power calculations show the superiority of our wavelet packet approach in a number of important situations and, hence, we suggest that the new tests are useful additions to the analyst's toolbox.