This report documents the results of a simulation based calibration
(SBC) run for RBesT
. The calibration data will be generated
whenever relevant changes to the gMAP
function were made.
The calibration runs are performed for typical use cases of
gMAP
. These include the three likelihoods (binomial,
gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a
run with a very/less conservative prior choice for between-trial
heterogeniety parameter.
The calibration data presented here has been generated at and with
the RBesT
git version as:
## Created: 2022-07-25 15:43:29 UTC
## git hash: a9012f0d7bf708d5f2998048157da97f3921e166
## MD5: e085f208d722d6ce54f5e7cbb5bafe62
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## /Users/weberse2/work/BesT/inst/sbc/calibration.rds
## "e085f208d722d6ce54f5e7cbb5bafe62"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
Likelihood:
Hierarchical prior:
\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]
\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]
The fake data simulation function returns for binomial and Poisson
data the sum of the responses while for normal the mean summary is used.
Please refer to the sbc_tools.R
and
make_reference_rankhist.R
R programs for the implementation
details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
family | data_scenario | sd_tau | N | total_divergent | total_divergent_sim_fraction | min_ess | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail |
---|---|---|---|---|---|---|---|---|---|---|
binomial | dense | 0.5 | 10000 | 41 | 0.004 | 173 | 1.013 | 0 | 196 | 169 |
binomial | dense | 1.0 | 10000 | 65 | 0.005 | 85 | 1.024 | 0 | 200 | 171 |
binomial | sparse | 0.5 | 10000 | 0 | 0.000 | 343 | 1.013 | 0 | 153 | 163 |
binomial | sparse | 1.0 | 10000 | 7 | 0.001 | 251 | 1.018 | 0 | 189 | 130 |
gaussian | dense | 0.5 | 10000 | 87 | 0.008 | 221 | 1.023 | 0 | 62 | 173 |
gaussian | dense | 1.0 | 10000 | 93 | 0.007 | 73 | 1.027 | 0 | 149 | 168 |
gaussian | sparse | 0.5 | 10000 | 739 | 0.036 | 8 | 1.265 | 1 | 33 | 133 |
gaussian | sparse | 1.0 | 10000 | 1282 | 0.062 | 12 | 1.236 | 1 | 105 | 119 |
poisson | dense | 0.5 | 10000 | 112 | 0.009 | 126 | 1.023 | 0 | 81 | 149 |
poisson | dense | 1.0 | 10000 | 145 | 0.013 | 29 | 1.118 | 0 | 84 | 230 |
poisson | sparse | 0.5 | 10000 | 1536 | 0.048 | 6 | 1.439 | 1 | 127 | 120 |
poisson | sparse | 1.0 | 10000 | 1902 | 0.063 | 3 | 1.733 | 1 | 45 | 51 |
Note: Large Rhat is defined as exceeding 1.2.
data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
1 | dense | binomial | 0.5 | mu | 75.341 | 63 | 0.137 |
13 | dense | binomial | 1 | mu | 66.074 | 63 | 0.371 |
25 | dense | gaussian | 0.5 | mu | 65.485 | 63 | 0.391 |
37 | dense | gaussian | 1 | mu | 58.202 | 63 | 0.648 |
49 | dense | poisson | 0.5 | mu | 63.258 | 63 | 0.467 |
61 | dense | poisson | 1 | mu | 88.410 | 63 | 0.019 |
73 | sparse | binomial | 0.5 | mu | 63.322 | 63 | 0.465 |
78 | sparse | binomial | 1 | mu | 72.000 | 63 | 0.205 |
83 | sparse | gaussian | 0.5 | mu | 47.373 | 63 | 0.929 |
88 | sparse | gaussian | 1 | mu | 80.282 | 63 | 0.070 |
93 | sparse | poisson | 0.5 | mu | 56.819 | 63 | 0.695 |
98 | sparse | poisson | 1 | mu | 76.698 | 63 | 0.115 |
data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
2 | dense | binomial | 0.5 | tau | 74.074 | 63 | 0.160 |
14 | dense | binomial | 1 | tau | 69.402 | 63 | 0.271 |
26 | dense | gaussian | 0.5 | tau | 99.622 | 63 | 0.002 |
38 | dense | gaussian | 1 | tau | 64.922 | 63 | 0.410 |
50 | dense | poisson | 0.5 | tau | 74.368 | 63 | 0.155 |
62 | dense | poisson | 1 | tau | 51.238 | 63 | 0.856 |
74 | sparse | binomial | 0.5 | tau | 51.546 | 63 | 0.848 |
79 | sparse | binomial | 1 | tau | 58.227 | 63 | 0.647 |
84 | sparse | gaussian | 0.5 | tau | 50.432 | 63 | 0.874 |
89 | sparse | gaussian | 1 | tau | 54.080 | 63 | 0.781 |
94 | sparse | poisson | 0.5 | tau | 44.147 | 63 | 0.966 |
99 | sparse | poisson | 1 | tau | 62.515 | 63 | 0.494 |
data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
3 | dense | binomial | 0.5 | theta\[1\] | 79.846 | 63 | 0.075 |
4 | dense | binomial | 0.5 | theta\[10\] | 43.597 | 63 | 0.970 |
5 | dense | binomial | 0.5 | theta\[2\] | 60.954 | 63 | 0.550 |
6 | dense | binomial | 0.5 | theta\[3\] | 54.797 | 63 | 0.760 |
7 | dense | binomial | 0.5 | theta\[4\] | 57.728 | 63 | 0.664 |
8 | dense | binomial | 0.5 | theta\[5\] | 68.134 | 63 | 0.307 |
9 | dense | binomial | 0.5 | theta\[6\] | 42.816 | 63 | 0.976 |
10 | dense | binomial | 0.5 | theta\[7\] | 47.334 | 63 | 0.929 |
11 | dense | binomial | 0.5 | theta\[8\] | 55.027 | 63 | 0.753 |
12 | dense | binomial | 0.5 | theta\[9\] | 63.578 | 63 | 0.456 |
15 | dense | binomial | 1 | theta\[1\] | 85.222 | 63 | 0.033 |
16 | dense | binomial | 1 | theta\[10\] | 46.490 | 63 | 0.941 |
17 | dense | binomial | 1 | theta\[2\] | 67.264 | 63 | 0.333 |
18 | dense | binomial | 1 | theta\[3\] | 67.149 | 63 | 0.337 |
19 | dense | binomial | 1 | theta\[4\] | 49.766 | 63 | 0.887 |
20 | dense | binomial | 1 | theta\[5\] | 76.941 | 63 | 0.111 |
21 | dense | binomial | 1 | theta\[6\] | 47.782 | 63 | 0.923 |
22 | dense | binomial | 1 | theta\[7\] | 55.155 | 63 | 0.749 |
23 | dense | binomial | 1 | theta\[8\] | 53.210 | 63 | 0.806 |
24 | dense | binomial | 1 | theta\[9\] | 55.782 | 63 | 0.729 |
27 | dense | gaussian | 0.5 | theta\[1\] | 51.456 | 63 | 0.850 |
28 | dense | gaussian | 0.5 | theta\[10\] | 68.442 | 63 | 0.298 |
29 | dense | gaussian | 0.5 | theta\[2\] | 84.006 | 63 | 0.040 |
30 | dense | gaussian | 0.5 | theta\[3\] | 68.736 | 63 | 0.289 |
31 | dense | gaussian | 0.5 | theta\[4\] | 63.104 | 63 | 0.473 |
32 | dense | gaussian | 0.5 | theta\[5\] | 57.690 | 63 | 0.665 |
33 | dense | gaussian | 0.5 | theta\[6\] | 53.146 | 63 | 0.807 |
34 | dense | gaussian | 0.5 | theta\[7\] | 67.366 | 63 | 0.330 |
35 | dense | gaussian | 0.5 | theta\[8\] | 63.974 | 63 | 0.442 |
36 | dense | gaussian | 0.5 | theta\[9\] | 61.517 | 63 | 0.529 |
39 | dense | gaussian | 1 | theta\[1\] | 57.574 | 63 | 0.669 |
40 | dense | gaussian | 1 | theta\[10\] | 62.003 | 63 | 0.512 |
41 | dense | gaussian | 1 | theta\[2\] | 61.786 | 63 | 0.520 |
42 | dense | gaussian | 1 | theta\[3\] | 83.008 | 63 | 0.046 |
43 | dense | gaussian | 1 | theta\[4\] | 65.254 | 63 | 0.398 |
44 | dense | gaussian | 1 | theta\[5\] | 70.733 | 63 | 0.235 |
45 | dense | gaussian | 1 | theta\[6\] | 61.402 | 63 | 0.533 |
46 | dense | gaussian | 1 | theta\[7\] | 86.822 | 63 | 0.025 |
47 | dense | gaussian | 1 | theta\[8\] | 77.888 | 63 | 0.098 |
48 | dense | gaussian | 1 | theta\[9\] | 77.862 | 63 | 0.098 |
51 | dense | poisson | 0.5 | theta\[1\] | 50.163 | 63 | 0.879 |
52 | dense | poisson | 0.5 | theta\[10\] | 59.251 | 63 | 0.611 |
53 | dense | poisson | 0.5 | theta\[2\] | 51.110 | 63 | 0.859 |
54 | dense | poisson | 0.5 | theta\[3\] | 59.661 | 63 | 0.596 |
55 | dense | poisson | 0.5 | theta\[4\] | 46.925 | 63 | 0.935 |
56 | dense | poisson | 0.5 | theta\[5\] | 52.096 | 63 | 0.835 |
57 | dense | poisson | 0.5 | theta\[6\] | 55.846 | 63 | 0.727 |
58 | dense | poisson | 0.5 | theta\[7\] | 55.872 | 63 | 0.726 |
59 | dense | poisson | 0.5 | theta\[8\] | 76.442 | 63 | 0.119 |
60 | dense | poisson | 0.5 | theta\[9\] | 55.782 | 63 | 0.729 |
63 | dense | poisson | 1 | theta\[1\] | 51.123 | 63 | 0.858 |
64 | dense | poisson | 1 | theta\[10\] | 74.752 | 63 | 0.148 |
65 | dense | poisson | 1 | theta\[2\] | 51.725 | 63 | 0.844 |
66 | dense | poisson | 1 | theta\[3\] | 49.549 | 63 | 0.892 |
67 | dense | poisson | 1 | theta\[4\] | 61.325 | 63 | 0.536 |
68 | dense | poisson | 1 | theta\[5\] | 67.392 | 63 | 0.329 |
69 | dense | poisson | 1 | theta\[6\] | 57.702 | 63 | 0.665 |
70 | dense | poisson | 1 | theta\[7\] | 60.314 | 63 | 0.573 |
71 | dense | poisson | 1 | theta\[8\] | 53.811 | 63 | 0.789 |
72 | dense | poisson | 1 | theta\[9\] | 72.371 | 63 | 0.196 |
75 | sparse | binomial | 0.5 | theta\[1\] | 46.682 | 63 | 0.938 |
76 | sparse | binomial | 0.5 | theta\[2\] | 57.536 | 63 | 0.671 |
77 | sparse | binomial | 0.5 | theta\[3\] | 69.709 | 63 | 0.262 |
80 | sparse | binomial | 1 | theta\[1\] | 60.992 | 63 | 0.548 |
81 | sparse | binomial | 1 | theta\[2\] | 56.128 | 63 | 0.718 |
82 | sparse | binomial | 1 | theta\[3\] | 66.842 | 63 | 0.347 |
85 | sparse | gaussian | 0.5 | theta\[1\] | 61.658 | 63 | 0.524 |
86 | sparse | gaussian | 0.5 | theta\[2\] | 85.184 | 63 | 0.033 |
87 | sparse | gaussian | 0.5 | theta\[3\] | 62.195 | 63 | 0.505 |
90 | sparse | gaussian | 1 | theta\[1\] | 49.382 | 63 | 0.895 |
91 | sparse | gaussian | 1 | theta\[2\] | 63.744 | 63 | 0.450 |
92 | sparse | gaussian | 1 | theta\[3\] | 57.062 | 63 | 0.687 |
95 | sparse | poisson | 0.5 | theta\[1\] | 63.117 | 63 | 0.472 |
96 | sparse | poisson | 0.5 | theta\[2\] | 59.430 | 63 | 0.604 |
97 | sparse | poisson | 0.5 | theta\[3\] | 64.858 | 63 | 0.412 |
100 | sparse | poisson | 1 | theta\[1\] | 70.259 | 63 | 0.248 |
101 | sparse | poisson | 1 | theta\[2\] | 69.491 | 63 | 0.268 |
102 | sparse | poisson | 1 | theta\[3\] | 66.829 | 63 | 0.347 |
## R version 4.1.0 (2021-05-18)
## Platform: x86_64-apple-darwin17.0 (64-bit)
## Running under: macOS Big Sur 10.16
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## attached base packages:
## [1] tools stats graphics grDevices datasets utils methods
## [8] base
##
## other attached packages:
## [1] purrr_0.3.4 rstan_2.21.2 StanHeaders_2.21.0-7
## [4] here_1.0.1 ggplot2_3.3.5 broom_0.7.9
## [7] tidyr_1.1.3 dplyr_1.0.7 assertthat_0.2.1
## [10] knitr_1.41
##
## loaded via a namespace (and not attached):
## [1] tidyselect_1.1.1 xfun_0.36 V8_3.4.2 colorspace_2.0-2
## [5] vctrs_0.3.8 generics_0.1.0 htmltools_0.5.1.1 stats4_4.1.0
## [9] loo_2.4.1 yaml_2.2.1 utf8_1.2.2 rlang_0.4.11
## [13] pkgbuild_1.2.0 pillar_1.6.2 glue_1.4.2 withr_2.4.2
## [17] DBI_1.1.1 matrixStats_0.60.1 lifecycle_1.0.0 stringr_1.4.0
## [21] munsell_0.5.0 gtable_0.3.0 codetools_0.2-18 evaluate_0.20
## [25] inline_0.3.19 callr_3.7.0 ps_1.6.0 parallel_4.1.0
## [29] curl_4.3.2 fansi_0.5.0 highr_0.9 Rcpp_1.0.7
## [33] scales_1.1.1 backports_1.2.1 RcppParallel_5.1.4 jsonlite_1.7.2
## [37] gridExtra_2.3 digest_0.6.27 stringi_1.7.3 processx_3.5.2
## [41] grid_4.1.0 rprojroot_2.0.2 cli_3.0.1 magrittr_2.0.1
## [45] tibble_3.1.3 crayon_1.4.1 pkgconfig_2.0.3 ellipsis_0.3.2
## [49] prettyunits_1.1.1 rmarkdown_2.19 R6_2.5.1 compiler_4.1.0