Methodology Used to Estimate the Bayes Significance Level for the
Null Hypothesis of No Change between the 2008-2010 and
2016-2018 Substate Population Percentages
This discussion focuses on the methodology used to conduct statistical tests of significance for comparing National Survey on Drug Use and Health (NSDUH) substate population percentages between 2008-2010 and 2016-2018. Estimates shown in Tables 1 to 12 earlier in this document are based on small area estimation (SAE) procedures. The methodology used to compare substate population percentages associated with these estimates is the same methodology used previously to compare the nonoverlapping year substate population percentages, such as the 2008-2010 and 2014-2016 substate population percentages.1 However, because some substate regions within 21 states were defined differently for the two time periods (i.e., 2008-2010 and 2016-2018), significance test results (based on the Bayes significance levels for the null hypothesis of no change: p value) could not be produced for these regions. In this document's tables, a double hyphen ("--") is used to denote substate regions that are not defined consistently across the two time periods. Note that all of the regions in Colorado, Kansas, Rhode Island, and South Dakota changed across the two time periods, so only estimates at the state level are included in the tables for these four states.
For details about the 2016-2018 substate region definitions, see the "2016-2018 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/. For details about the 2008-2010 substate region definitions, see the "2008-2010 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/. In eight states, the names of some substate regions changed across the two time periods, but the definitions remained the same. States that have such comparable regions with different names are listed in Exhibit A.1 at the end of this document. The exhibit includes information regarding only these eight states; however, all of the 2016-2018 substate regions for each of these states are listed, although aggregate regions are included only if an aggregate region is comparable with a substate region with a different name.
Comparisons of substate population percentages between 2008-2010 and 2016-2018 were made only for individuals aged 12 or older, except for the underage alcohol use outcome in Table 5 (where comparisons were made for individuals aged 12 to 20) and for the four mental health outcomes (serious mental illness, any mental illness, serious thoughts of suicide, and major depressive episode) in Tables 9 through 12, respectively (where comparisons were made for adults aged 18 or older).
To determine whether differences between population percentages of a substate region over two time periods were statistically significant, let 
and 
denote the 2008-2010 and 2016-2018 prevalence rates, respectively, for substate region-s. The difference between and is defined in terms of the log-odds ratio 
as opposed to the simple difference because the posterior distribution of the is closer to Gaussian than the posterior distribution of the simple difference, 
. The is defined as follows:
. D
In this equation, the ln denotes the natural logarithm. The p value given in each of the tables in this document (i.e., Tables 1 to 12) is computed to test the null hypothesis of no difference; that is, 
or equivalently 
. An estimate of is given by
, D
where the 
are the 2008-2010 substate estimates presented in the "2008-2010 NSDUH Substate Region Estimates: Excel Tables" at https://www.samhsa.gov/data/, and the 
are the 2016-2018 substate estimates presented in the "2016-2018 NSDUH Substate Region Estimates: Excel Tables" at https://www.samhsa.gov/data/.
Let 
and 
, then an estimate of the posterior variance of is given by 
, where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation, as follows:
. D
Note that 
and 
used here to calculate 
are the same posterior variances used in calculating the 2008-2010 Bayesian confidence intervals (CIs) and the 2016-2018 CIs, respectively. Thus, can be calculated using the published Bayesian CIs, 
and 
, as follows:
where 
and 
.
The correlation between and was obtained by simultaneously modeling the 2008-2010 and 2016-2018 NSDUH data using a method similar to modeling the correlations between the 2008-2010 and 2014-2016 substate estimates.2
To calculate the p value for testing the null hypothesis of no difference (), it was assumed that the posterior distribution of is normal with 
and 
. The Bayesian p value or significance level for the null hypothesis of no difference is 
, where Z is a standard normal random variate, 
, and 
denotes the absolute value of z. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin,3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is, 
. With the posterior distribution of approximately normal, 
is given by the above expression.
