Decisions based partly or solely on predictions from probabilistic models may be sensitive to model misspecification. Statisticians are taught from an early stage that “all models are wrong, but some are useful”; however, little formal guidance exists on how to assess the impact of model approximation on decision making, or how to proceed when optimal actions appear sensitive to model fidelity. This article presents an overview of recent developments across different disciplines to address this. We review diagnostic techniques, including graphical approaches and summary statistics, to help highlight decisions made through minimised expected loss that are sensitive to model misspecification. We then consider formal methods for decision making under model misspecification by quantifying stability of optimal actions to perturbations to the model within a neighbourhood of model space. This neighbourhood is defined in either one of two ways. First, in a strong sense via an information (Kullback–Leibler) divergence around the approximating model. Second, using a Bayesian nonparametric model (prior) centred on the approximating model, in order to “average out” over possible misspecifications. This is presented in the context of recent work in the robust control, macroeconomics and financial mathematics literature. We adopt a Bayesian approach throughout although the presentation is agnostic to this position.

Watson and Holmes propose ways of investigating robustness of statistical decisions by examining certain neighbourhoods around a posterior distribution. This may partly amount to ad hoc modelling of extra uncertainty. Instead of creating neighbourhoods around the posterior a posteriori, we argue that it might be more fruitful to model a layer of extra uncertainty first, in the model building process, and then allow the data to determine how big the resulting neighbourhoods ought to be. We develop and briefly illustrate a general strategy along such lines.

We elaborate on Watson and Holmes’ observation that misspecification is contextual: a model that is wrong can still be adequate in one prediction context, yet grossly inadequate in another. One can incorporate such phenomena by adopting a generalized posterior, in which the likelihood is multiplied by an exponentiated loss. We argue that Watson and Holmes’ characterization of such generalized posteriors does not really explain their good practical performance, and we provide an alternative explanation which suggests a further extension of the method.

How may we develop methods of analysis which address the consequences of the mismatch between the formal structural requirements of Bayesian analysis and the actual assessments that are carried out in practice? A paper by Watson and Holmes provides an overview of methods developed to address such issues and makes suggestions as to how such analyses might be carried out. This article adds commentary on the principles and practices which should guide us in such problems.

This note discusses Watson and Holmes [Statist. Sci. (2016) 31 465–489] and their proposals towards more robust Bayesian decisions. While we acknowledge and commend the authors for setting new and all-encompassing principles of Bayesian robustness, and while we appreciate the strong anchoring of these within a decision-theoretic framework, we remain uncertain as to what extent such principles can be applied outside binary decisions. We also wonder at the ultimate relevance of Kullback–Leibler neighbourhoods into characterising robustness and we instead favour extensions along nonparametric axes.

The work described here was motivated by the need to address a long standing problem in engineering, namely, the tracking of reliability growth. An archetypal scenario is the performance of software as it evolves over time. Computer scientists are challenged by the task of when to release software. The same is also true for complex engineered systems like aircraft, automobiles and ballistic missiles. Tracking problems also arise in actuarial science, biostatistics, cancer research and mathematical finance.

A natural approach for addressing such problems is via the control theory methods of filtering, smoothing and prediction. But to invoke such methods, one needs a proper philosophical foundation, and this has been lacking. The first three sections of this paper endeavour to fill this gap. A consequence is the point of view proposed here, namely, that reliability not be interpreted as a probability. Rather, reliability should be conceptualized as a dynamically evolving propensity in the sense of Pierce and Popper. Whereas propensity is to be taken as an undefined primitive, it manifests as a chance (or frequency) in the sense of de Finetti. The idea of looking at reliability as a propensity also appears in the philosophical writings of Kolmogorov. Furthermore, survivability which quantifies ones uncertainty about a propensity should be the metric of performance that needs to be tracked. The first part of this paper is thus a proposal for a paradigm shift in the manner in which one conceptualizes reliability, and by extension, survival analysis. This message is also germane to other areas of applied probability and statistics, like queueing, inventory and time series analysis.

The second part of this paper is technical. Its purpose is to show how the philosophical material of the first part can be incorporated into a framework that leads to a methodological package. To do so, we focus on the problem which motivated the first part, and develop a mathematical model for describing the evolution of an item’s propensity to survive. The item could be a component, a system or a biological entity. The proposed model is based on the notion of competing risks. Models like this also appear in biostatistics under the label of cure models. Whereas the competing risks scenario is instructive, it is not the only way to describe the phenomenon of growth; its use here is illustrative. All the same, one of its virtues is that it paves the path towards a contribution to the state of the art of filtering by considering the case of censored observations. Even though censoring is the hallmark of survival analysis, it could also arise in time series analysis and control theory, making the development here of a broader and more general appeal.

Singpurwalla presents an insightful proposal on foundations of reliability [Statist. Sci. 31 (2016) 521–540], suggesting to consider reliability not as a probability but as a propensity, in particular as the unobservable parameter in De Finetti’s famous representation theorem. One specific issue considered is reliability growth, with example scenario the performance of software as it evolves over time. We briefly discuss some related aspects, mainly based on applied research on statistical methods to support software testing and insights from our research on system reliability.

Singpurwalla addresses the important challenge of modelling a unique individual. He proposes “propensity” as an approach to describing the reliability or life time of “one of a kind”. My view is that mathematical modelling is only possible when we assume that nonunique features provide sufficient information for statistical prediction to be useful. As far as possible, we should test our assumptions. However, contrary to a popular perception of Hume, we always rely on some beliefs.

Since the early nineteenth century, the concept of objective probability has been dynamic. As we recognize this history, we can strengthen Professor Nozer Singpuwalla’s vision of reliability of survival analysis by aligning it with earlier conceptions elaborated by Laplace, Borel, Kolmogorov, Ville and Neyman. By emphasizing testing and recognizing the generality of the vision of Kolmogorov and Neyman, we gain a perspective that does not rely on exchangeability.

This rejoinder to the discussants of Filtering and Tracking Survival Propensity begins with a brief history of the statistical aspects of reliability and its impact on survival analysis and responds to the several issues raised by the discussants, some of which are conceptual and some pragmatic.

The paper gives a statistically focused selective view of chaos-based communication which uses segments of noise-like chaotic waves as carriers of messages, replacing the traditional sinusoidal radio waves. The presentation concerns joint statistical and dynamical modelling of the binary communication system known as “chaos shift-keying”, representative of the area, and leverages the statistical properties of chaos. Practically, such systems apply to both wireless and optical laser communication channels. Theoretically, the chaotic waves are generated iteratively by chaotic maps, and practically, by electronic circuits or lasers. Both single-user and multiple-user systems are covered. The focus is on likelihood-based decoding of messages, essentially estimation of binary-valued parameters and efficiency of the system is in terms of the probability of bit decoding error. The emphasis is on exact theoretical results for bit error rate, their structured approximations and engineering interpretations. Design issues, optimality of performance, interference and fading are other topics considered. The statistical aspects of chaotic synchronization are involved in the modelling of optical systems. Empirical illustrations from an experimental laser-based system are presented. The overall aim is to show the use of statistical methodology in unifying and advancing the area.

We consider the use of default priors in the Bayes methodology for seeking information concerning the true value of a parameter. By default prior, we mean the mathematical prior as initiated by Bayes [Philos. Trans. R. Soc. Lond. 53 (1763) 370–418] and pursued by Laplace [Théorie Analytique des Probabilités (1812) Courcier], Jeffreys [Theory of Probability (1961) Clarendon Press], Bernardo [J. Roy. Statist. Soc. Ser. B 41 (1979) 113–147] and many more, and then recently viewed as “potentially dangerous” [Science 340 (2013) 1177–1178] and “potentially useful” [Science 341 (2013) 1452]. We do not mean, however, the genuine prior [Science 340 (2013) 1177–1178] that has an empirical reference and would invoke standard frequency modelling. And we do not mean the subjective or opinion prior that an individual might have and would be viewed as specific to that individual. A mathematical prior has no referenced frequency information, but on occasion is known otherwise to lead to repetition properties called confidence. We investigate the presence of such supportive property, and ask can Bayes give reliability for other than the particular parameter weightings chosen for the conditional calculation. Thus, does the methodology have reproducibility? Or is it a leap of faith.

For sample-space analysis, recent higher-order likelihood methods with regular models show that third-order accuracy is widely available using profile contours [In Past, Present and Future of Statistical Science (2014) 237–252 CRC Press].

But for parameter-space analysis, accuracy is widely limited to first order. An exception arises with a scalar full parameter and the use of the scalar Jeffreys [J. Roy. Statist. Soc. Ser. B 25 (1963) 318–329]. But for vector full parameter even with a scalar interest parameter, difficulties have long been known [J. Roy. Statist. Soc. Ser. B 35 (1973) 189–233] and with parameter curvature, accuracy beyond first order can be unavailable [Statist. Sci. 26 (2011) 299–316]. We show, however, that calculations on the parameter space can give full second-order information for a chosen scalar interest parameter; these calculations, however, require a Jeffreys prior that is used fully restricted to the one-dimensional profile for that interest parameter. Such a prior is effectively data-dependent and parameter-dependent and is focally restricted to the one-dimensional contour; these priors fall outside the usual Bayes approach and yet with substantial calculations can still give less than frequency analysis.

We provide simple examples using discrete extensions of Jeffreys prior. These serve as counter-examples to general claims that Bayes can offer accuracy for statistical inference. To obtain this accuracy with Bayes, more effort is required compared to recent likelihood methods, which still remain more accurate. And with vector full parameters, accuracy beyond first order is routinely not available, as a change in parameter curvature causes Bayes and frequentist values to change in opposite direction, yet frequentist has full reproducibility.

An alternative is to view default Bayes as an exploratory technique and then ask does it do as it overtly claims? Is it reproducible as understood in contemporary science? The posterior gives a distribution for an interest parameter and, thereby, a quantile for the interest parameter; an oracle could record whether it was left or right of the true value. If the average split in evaluative repetitions is in accord with the nominal level, then the approach is providing accuracy. And if not, then what is up, other than performance specific to the parameter frequencies in the prior. No one has answers although speculative claims abound.

Age–period–cohort models have been used to examine and forecast cancer incidence and mortality for over three decades. However, the fitting and interpretation of these models requires great care because of the well-known identifiability problem that exists; given any two of age, period, and cohort, the third is determined. In this paper, we review the identifiability problem and models that have been proposed for analysis, from both frequentist and Bayesian standpoints. A number of recent analyses that use age–period–cohort models are described and critiqued before data on cancer incidence in Washington State are analyzed with various models, including a new Bayesian approach based on an identifiable parameterization.

Very long and noisy sequence data arise from biological sciences to social science including high throughput data in genomics and stock prices in econometrics. Often such data are collected in order to identify and understand shifts in trends, for example, from a bull market to a bear market in finance or from a normal number of chromosome copies to an excessive number of chromosome copies in genetics. Thus, identifying multiple change points in a long, possibly very long, sequence is an important problem. In this article, we review both classical and new multiple change-point detection strategies. Considering the long history and the extensive literature on the change-point detection, we provide an in-depth discussion on a normal mean change-point model from aspects of regression analysis, hypothesis testing, consistency and inference. In particular, we present a strategy to gather and aggregate local information for change-point detection that has become the cornerstone of several emerging methods because of its attractiveness in both computational and theoretical properties.

Chien-Fu Jeff Wu was born January 15, 1949, in Taiwan. He earned a B.Sc. in Mathematics from National Taiwan University in 1971, and a Ph.D. in Statistics from the University of California, Berkeley in 1976. He has been a faculty member at the University of Wisconsin, Madison (1977–1988), the University of Waterloo (1988–1993), the University of Michigan (1995–2003; department chair 1995–8) and currently is the Coca-Cola Chair in Engineering Statistics and Professor in the H. Milton Stewart School of Industrial and Systems Engineering at the Georgia Institute of Technology. He is known for his work on the convergence of the EM algorithm, resampling methods, nonlinear least squares, sensitivity testing and industrial statistics, including design of experiments, robust parameter design and computer experiments, and has been credited for coining the term “data science” as early as 1997.

Jeff has received several awards, including the COPSS Presidents’ Award (1987), the Shewhart Medal (2008), the R. A. Fisher Lectureship (2011) and the Deming Lecturer Award (2012). He is an elected member of Academia Sinica (2000) and the National Academy of Engineering (2004), and has received many other awards and honors including an honorary doctorate from the University of Waterloo.

Jeff has supervised 45 Ph.D. students to date, many of whom are active researchers in the statistical sciences. He has published more than 170 peer-reviewed articles and two books. He was the second Editor of Statistica Sinica (1993–96). Jeff married Susan Chang in 1979, and they have two children, Emily and Justin.

This conversation took place in Atlanta, Georgia, on April 21, 2015.

A. Samad Hedayat was born on July 11, 1937, in Jahrom, Iran. He finished his undergraduate education in bioengineering with honors from the University of Tehran in 1962 and came to the U.S. to study statistics at Cornell, completing his Ph.D. in 1969. Just a few years later, in 1974, Samad accepted a full professor position at the University of Illinois at Chicago Circle—now called University of Illinois at Chicago (UIC)—and was named UIC Distinguished Professor in 2003. He was an early leader in the Department of Mathematics, Statistics and Computer Science and he remains a driving force to this day. Samad has also made substantial contributions in terms of research and service to the field, as evidenced by his numerous honors: he is an elected member of the International Statistical Institute, a fellow of the Institute of Mathematical Statistics and the American Statistical Association and an honorary member of the Iranian Mathematical Society, among others. This conversation, which was conducted in September 2015 and May 2016, touches on Professor Hedayat’s career, and the past, present and future of statistics. In keeping with one of his great passions, it also offers an abundance of advice for students and junior faculty.