p_significance.lm {parameters} | R Documentation |
Compute the probability of Practical Significance (ps),
which can be conceptualized as a unidirectional equivalence test. It returns
the probability that an effect is above a given threshold corresponding to a
negligible effect in the median's direction, considering a parameter's full
confidence interval. In comparison the the equivalence_test()
function,
where the SGPV (second generation p-value) describes the proportion of the
full confidence interval that is inside the ROPE, the value returned by
p_significance()
describes the larger proportion of the full confidence
interval that is outside the ROPE. This makes p_significance()
comparable
to bayestestR::p_direction()
, however, while p_direction()
compares to
a point-null by default, p_significance()
compares to a range-null.
## S3 method for class 'lm'
p_significance(x, threshold = "default", ci = 0.95, verbose = TRUE, ...)
x |
A statistical model. |
threshold |
The threshold value that separates significant from negligible effect, which can have following possible values:
|
ci |
Confidence Interval (CI) level. Default to |
verbose |
Toggle warnings and messages. |
... |
Arguments passed to other methods, e.g. |
p_significance()
returns the proportion of the full confidence
interval range (assuming a normally distributed, equal-tailed interval) that
is outside a certain range (the negligible effect, or ROPE, see argument
threshold
). If there are values of the distribution both below and above
the ROPE, p_significance()
returns the higher probability of a value being
outside the ROPE. Typically, this value should be larger than 0.5 to indicate
practical significance. However, if the range of the negligible effect is
rather large compared to the range of the confidence interval,
p_significance()
will be less than 0.5, which indicates no clear practical
significance.
Note that the assumed interval, which is used to calculate the practical significance, is an approximation of the full interval based on the chosen confidence level. For example, if the 95% confidence interval of a coefficient ranges from -1 to 1, the underlying full (normally distributed) interval approximately ranges from -1.9 to 1.9, see also following code:
# simulate full normal distribution out <- bayestestR::distribution_normal(10000, 0, 0.5) # range of "full" distribution range(out) # range of 95% CI round(quantile(out, probs = c(0.025, 0.975)), 2)
This ensures that the practical significance always refers to the general compatible parameter space of coefficients. Therefore, the full interval is similar to a Bayesian posterior distribution of an equivalent Bayesian model, see following code:
library(bayestestR) library(brms) m <- lm(mpg ~ gear + wt + cyl + hp, data = mtcars) m2 <- brm(mpg ~ gear + wt + cyl + hp, data = mtcars) # probability of significance (ps) for frequentist model p_significance(m) # similar to ps of Bayesian models p_significance(m2) # similar to ps of simulated draws / bootstrap samples p_significance(simulate_model(m))
A data frame.
There is no standardized approach to drawing conclusions based on the available data and statistical models. A frequently chosen but also much criticized approach is to evaluate results based on their statistical significance (Amrhein et al. 2017).
A more sophisticated way would be to test whether estimated effects exceed the "smallest effect size of interest", to avoid even the smallest effects being considered relevant simply because they are statistically significant, but clinically or practically irrelevant (Lakens et al. 2018, Lakens 2024).
A rather unconventional approach, which is nevertheless advocated by various authors, is to interpret results from classical regression models in terms of probabilities, similar to the usual approach in Bayesian statistics (Greenland et al. 2022; Rafi and Greenland 2020; Schweder 2018; Schweder and Hjort 2003; Vos 2022).
The parameters package provides several options or functions to aid statistical inference. These are, for example:
equivalence_test()
, to compute the (conditional) equivalence test for
frequentist models
p_significance()
, to compute the probability of practical significance,
which can be conceptualized as a unidirectional equivalence test
p_function()
, or consonance function, to compute p-values and
compatibility (confidence) intervals for statistical models
the pd
argument (setting pd = TRUE
) in model_parameters()
includes
a column with the probability of direction, i.e. the probability that a
parameter is strictly positive or negative. See bayestestR::p_direction()
for details.
the s_value
argument (setting s_value = TRUE
) in model_parameters()
replaces the p-values with their related S-values (Rafi and Greenland 2020)
finally, it is possible to generate distributions of model coefficients by
generating bootstrap-samples (setting bootstrap = TRUE
) or simulating
draws from model coefficients using simulate_model()
. These samples
can then be treated as "posterior samples" and used in many functions from
the bayestestR package.
Most of the above shown options or functions derive from methods originally
implemented for Bayesian models (Makowski et al. 2019). However, assuming
that model assumptions are met (which means, the model fits well to the data,
the correct model is chosen that reflects the data generating process
(distributional model family) etc.), it seems appropriate to interpret
results from classical frequentist models in a "Bayesian way" (more details:
documentation in p_function()
).
There is also a plot()
-method
implemented in the see-package.
For more details, see bayestestR::p_significance()
. See also
equivalence_test()
.
data(qol_cancer)
model <- lm(QoL ~ time + age + education, data = qol_cancer)
p_significance(model)
p_significance(model, threshold = c(-0.5, 1.5))
# plot method
if (require("see", quietly = TRUE)) {
result <- p_significance(model)
plot(result)
}