{"id":375,"date":"2017-06-09T18:12:35","date_gmt":"2017-06-09T22:12:35","guid":{"rendered":"http:\/\/hydrouncertainty.org\/?p=375"},"modified":"2019-01-15T12:27:39","modified_gmt":"2019-01-15T17:27:39","slug":"uncertainty-about-uncertainty","status":"publish","type":"post","link":"http:\/\/hydrouncertainty.org\/2017\/06\/09\/uncertainty-about-uncertainty\/","title":{"rendered":"Uncertainty about Uncertainty"},"content":{"rendered":"\n<p> I like to point to Keith Beven&#8217;s (1987) conference paper, titled \u2018Towards a new paradigm in hydrology\u2019, as a place for new hydrologists to start to develop an understanding of uncertainty in the hydrological sciences<a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftn1\">[1]<\/a>. In that paper Keith argued that \u201clittle to no success\u201d had been made against the fundamental problem of developing theories about how small-scale complexities lead to large-scale behavior in hydrological systems<a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftn2\">[2]<\/a>.<\/p>\n\n\n\n<p>The paper discussed what hydrologists might do about this situation, and in the last two paragraphs Keith made two predictions:<\/p>\n\n\n\n<ul><li>First, that hydrology of the future will require a theoretical framework that is inherently stochastic to deal with the <em>\u201cvalue of imperfect observations and qualitative knowledge in reducing predictive uncertainty.\u201d<\/em> <\/li><li>Second, that hydrologists would not actually develop this type of theoretical framework, but instead would capitalize on the (then) emerging power of desktop computing to approximate uncertainty; leading to results that <em>\u201cmay not be pretty,\u201d<\/em> but which are <em>\u201crealistic in reflecting both our understanding and lack of knowledge of hydrological systems.\u201d<\/em> <\/li><\/ul>\n\n\n\n<p>Of course, both of his predictions were essentially correct. In 2007 Jeff McDonnell wrote <em>\u201cto make continued progress in watershed hydrology \u2026 we need to \u2026 explore the set of organizing principles that might underlie heterogeneity and complexity\u201d <\/em>(McDonnell et al., 2007)<em>.<\/em> Jeff went on to describe several possibilities for \u2018moving beyond\u2019  heterogeneity and process complexity, but the point is nevertheless  clear: the problem remains unsolved. Similarly, Keith wrote last year  that <em>\u201cour perceptual model of uncertainty is now much more sophisticated \u2026 but this has not resulted in analogous progress in uncertainty quantification\u201d<\/em> (Beven, 2016).   <\/p>\n\n\n\n<p>The situation in hydrology right now is that we understand that macro-scale behaviors of watersheds are governed by small-scale heterogeneities, but we don\u2019t have a theory about how this works, and we don\u2019t have any fundamental theory that allows us to (reliably) quantify predictive uncertainties related to these processes.<\/p>\n\n\n\n<p>Instead, what we have are ad hoc strategies for obtaining numbers that seem like they might be related with uncertainty. One example of this is the recent proliferation of multi-parameterization modeling systems that allow the user to choose between a variety of options for different flux parameterizations. An example is the Structure for Unifying Multiple Modeling Alternatives; Clark et al. (2015) wrote that <em>\u201c[SUMMA] provides capabilities to evaluate different representations of spatial heterogeneity and different flux parameterizations, and therefore tackle the fundamental modeling challenge of simulating the fluxes of water and energy over a hierarchy of spatial scales.\u201d<\/em> It\u2019s unclear why predicting with a variety of different parameterizations of scale-dependent processes allows us to \u2018tackle\u2019 scale-related challenges: we still lack a fundamental theory of hydrologic scaling, and making predictions with several different parameterizations is not in any way reflective of the actual nature of our lack of knowledge about the principles and behavior of hydrologic systems.<\/p>\n\n\n\n<p>Other methods that we often use for uncertainty quantification suffer from similar problems. Bayesian methods allow us to do precisely one thing: inter-compare or average several different competing models. Bayesian methods, and indeed any methods for model inter-comparison or model averaging, are fundamentally incapable of helping us to understand the difference between our family of models (<em>i.e.,<\/em> those models that are assigned finite probability by the prior) and the real system. Gelman &amp; Shalizi (2013) give a philosophical treatment of this problem that is worth reading.<\/p>\n\n\n\n<p>It has been suggested that we might use empirical methods to develop probability distributions over different components of predictive imprecision (<em>e.g.,<\/em> Montanari and Koutsoyiannis, 2012), but this type of approach assumes stationarity not only in those aspects of the hydrological system that are captured by the model, but also stationarity in the relationship between model error and those parts of the hydrological system that are not captured by the model.<\/p>\n\n\n\n<p>The point is that uncertainty is tautologically inestimable, and so it is really no surprise that Keith\u2019s second prediction came true \u2013 there was never any real possibility to develop a rigorous theoretical basis for understanding uncertainty, scale-dependent or otherwise. More than that, the methods that we have come up with to approximate uncertainty don\u2019t actually do that at all \u2013 at least not in any way that is fundamentally or theoretically reliable. I propose the following challenge: provide a theorem that proves a bounded, asymptotic, or even consistent relationship between any quantitative estimator and real-world uncertainty under evaluable assumptions. I offer the standard wager for scientific controversy<a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftn3\">[3]<\/a>: a bottle of Yamazaki 12 year, or comparable.<\/p>\n\n\n\n<p>Until we have such a theorem, I propose that it is not useful to talk about uncertainty quantification \u2013 approximate or otherwise, \u2013 because none of our estimators are related to real uncertainty in any systematic way.<\/p>\n\n\n\n<p>Instead, I predict a new paradigm change in hydrology. I suspect that within the next 30 years, the conversation in hydrology will be about information rather than uncertainty. The reason for this is that while it is impossible to estimate uncertainty (even approximately), it is possible to obtain at least bounded estimates of information measures (Nearing and Gupta, 2017). The main project of science seems to be about comparing the information contained in observation data with the information provided by a hypothesis-driven model. Similarly, the problem of scaling under heterogeneity seems to be fundamentally about cross-scale information transfers, rather than about uncertainty. Given recent work in basic physics (<em>e.g.,<\/em> Cao et al., 2017), I suspect that it will not take three more decades for us to discover real scaling theories under this type of perspective.<\/p>\n\n\n\n<p>Of course, we will always want to know the reliability of our model predictions, and for this reason the concept of uncertainty will never go away completely. I propose, however, that the tractable and meaningful challenge is to understand the actual predictive precision implied by our hydrological theory and hypotheses. At present, we do not do this. Current practice is to build models that are over-precise, and then append \u2018uncertainty\u2019 distributions to their predictions. This method of dealing with a lack of complete knowledge stems fundamentally from our Newtonian heritage. Essentially all process-based hydrological models are expressed as PDEs, which must admit solution in order to make a prediction.<\/p>\n\n\n\n<p>There are two problems with building dynamical models as PDEs. First, such models make <em>ontological<\/em> predictions (predictions about what will happen), whereas what we actually want are <em>epistemological<\/em> predictions (predictions about what we can know about what will happen). The uncertainty probabilities that we append to our models are the latter, and they are what we actually need for both hypothesis testing and decision support. But these probabilities are not the product of actually solving our model equations. Even if our model is a stochastic PDE, the random walk component is simply an ad hoc appendage on the drift function. Sampling model inputs or different model structures does not actually tell us anything about our lack of knowledge associated with any of those model structures. Models built as PDEs simply do not solve for anything that represents what we can know from our physical theory and hypotheses.<\/p>\n\n\n\n<p>The second problem is that a PDE only provides a prediction if it can be solved. This requires that we prescribe values (or distributions) over all parameters contained in our hypothetical parameterizations, some of which are impossible to measure and otherwise difficult to estimate. It would be exciting to have a method for constructing models that allows us to assign values to only those parameters that we feel we actually have some information about.<\/p>\n\n\n\n<p>But there is, in principle (although I have no example of such), a way to build  this type of model. Instead of expressing conservation principles using  differential equations, we could express them as symmetry constraints on probability distributions. To do this, we might specify a Bayesian network such that  each node is a random variable representing a particular scale-dependent  quantity at a particular time and location within the modeled system; conservation  laws could then be used to effectively rule out large portions of the joint  space of values over these random variables. By imposing conservation laws and  other physical principles as constraints on joint probability  distributions, our models would fundamentally solve for what we can know about the future or unobserved behavior of a  dynamical system conditional on whatever information (theories, hypotheses, data) are used to build the model. In principle, anything that we do know, or  wish to hypothesize, could be imposed as a constraint on the joint distribution  over a family of random variables representing different aspects of  system behavior.<\/p>\n\n\n\n<p>Although such a strategy would not allow us to measure epistemic uncertainty (uncertainty is always and still inestimable), at least it would allow  us to know what information we actually have about the behavior of hydrologic systems. This would be a very different way of approaching model  building than appending uncertainty distributions to PDE solutions, and  would allow us to actually quantify the information content of our scientific hypotheses  and models.<\/p>\n\n\n\n<p>So perhaps my predictions about paradigm change will not come to pass. I am, after all, essentially arguing against two of the most fundamental concepts in our science: that we should not use Bayesian methods to evaluate models, and that we should not use differential equations to build models. I  do suspect that I am right about both of these things, in the sense that  our science (indeed, any science of complex systems) would accelerate by  abandoning these ideas in favor of information-centric philosophies and methods,  but perhaps it will take longer than 30 years to demonstrate that such a substantial change is necessary.\u00a0 <br><\/p>\n\n\n\n<h2>References:<\/h2>\n\n\n\n<ol><li>Beven, K. &#8216;Towards a new paradigm in hydrology&#8217;. <em>Water for the Future: Hydrology in Perspective.\u00a0<\/em>, Rome, Italy: IAHS Publication.<\/li><li>Beven, K. J. (2016) &#8216;Facets of uncertainty: Epistemic error, non-stationarity, likelihood, hypothesis testing, and communication&#8217;, <em>Hydrological Sciences Journal,<\/em> (9), pp. 1652-1665.<\/li><li>Cao, C., Carroll, S. M. and Michalakis, S. (2017) &#8216;Space from Hilbert space: Recovering geometry from bulk entanglement&#8217;, <em>Physical Review D,<\/em> 95(2), pp. 024031.<\/li><li>Clark, M. P., Nijssen, B., Lundquist, J. D., Kavetski, D., Rupp, D. E., Woods, R. A., Freer, J. E., Gutmann, E. D., Wood, A. W. and Gochis, D. J. (2015) &#8216;A unified approach for process\u2010based hydrologic modeling: 2. Model implementation and case studies&#8217;, <em>Water Resources Research,<\/em> 51(4), pp. 2515-2542.<\/li><li>Dooge, J. C. I. (1986) &#8216;Looking for hydrologic laws&#8217;, <em>Water Resources Research,<\/em> 22(9S), pp. 46S-58S.<\/li><li>Gelman, A. and Shalizi, C. R. (2013) &#8216;Philosophy and the practice of Bayesian statistics&#8217;, <em>British Journal of Mathematical and Statistical Psychology,<\/em> 66(1), pp. 8-38.<\/li><li>McDonnell, J. J., Sivapalan, M., Vach\u00e9, K., Dunn, S., Grant, G., Haggerty, R., Hinz, C., Hooper, R., Kirchner, J. and Roderick, M. L. (2007) &#8216;Moving beyond heterogeneity and process complexity: A new vision for watershed hydrology&#8217;, <em>Water Resources Research,<\/em> 43(7).<\/li><li>Montanari, A. and Koutsoyiannis, D. (2012) &#8216;A blueprint for process-based modeling of uncertain hydrological systems&#8217;, <em>Water Resources Research,<\/em> 48(9), pp. n\/a-n\/a.<\/li><li>Nearing, G. S. and Gupta, H. V. (2017) &#8216;Information vs. uncertainty as a foundation for a science of environmental modeling&#8217;, <em>https:\/\/arxiv.org\/abs\/1704.07512<\/em>.<\/li><\/ol>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p><a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftnref1\">[1]<\/a>It\u2019s the kind of paper that can be enjoyed with a beer.<br><a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftnref2\">[2]<\/a>Dooge (1986) gave a somewhat more technical discussion of this same problem.<br><a href=\"https:\/\/www.blogger.com\/blogger.g?blogID=6821077921195605758#_ftnref3\">[3]<\/a><em>e.g.,https:\/\/www.quantamagazine.org\/supersymmetry-bet-settled-with-cognac-20160822<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I like to point to Keith Beven&#8217;s (1987) conference paper, titled \u2018Towards a new paradigm in hydrology\u2019, as a place for new hydrologists to start to develop an understanding of uncertainty in the hydrological sciences[1]. In that paper Keith argued<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[10],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/posts\/375"}],"collection":[{"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/comments?post=375"}],"version-history":[{"count":1,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/posts\/375\/revisions"}],"predecessor-version":[{"id":376,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/posts\/375\/revisions\/376"}],"wp:attachment":[{"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/media?parent=375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/categories?post=375"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/hydrouncertainty.org\/wp-json\/wp\/v2\/tags?post=375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}