Methodology Used to Estimate the Bayes Significance Level for the Null Hypothesis of
No Change between the 2008-2010 and 2014-2016 Population Percentages
This discussion focuses on the methodology used to conduct statistical tests of significance for comparing National Survey on Drug Use and Health (NSDUH) substate population percentages between 2008-2010 and 2014-2016. Population percentages shown in Tables 1 to 12 earlier in this document are based on small area estimation (SAE) procedures, and the methodology used to compare these population percentages is the same methodology used to compare the state small area estimates.1 However, because some substate regions within 21 states were defined differently for the two time periods (i.e., 2008-2010 and 2014-2016), significance test results (based on the Bayes significance levels for the null hypothesis of no change: p value) could not be produced for these regions. In this document's tables, a double hyphen ("--") is used to denote substate regions that are not defined consistently across the two time periods. Note that all of the regions in Colorado, Kansas, Rhode Island, and South Dakota changed across the two time periods, so only estimates at the state level are included in the tables for these four states.
For details about the 2014-2016 substate region definitions, see the "2014-2016 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/. For details about the 2008-2010 substate region definitions, see the "2008-2010 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/. In eight states, the names of some substate regions changed across the two time periods, but the definitions remained the same. States that have such comparable regions with different names are listed in Exhibit A.1 at the end of this document. The exhibit includes information regarding only these eight states; however, all of the 2014-2016 substate regions for each of these states are listed, although aggregate regions are included only if an aggregate region is comparable with a substate region with a different name.
Comparisons of substate population percentages between 2008-2010 and 2014-2016 were made only for individuals aged 12 or older, except for the underage alcohol use outcome in Table 5 (where comparisons were made for individuals aged 12 to 20) and for the four mental health outcomes (serious mental illness [SMI], any mental illness [AMI], serious thoughts of suicide, and major depressive episode [MDE]) in Tables 9 through 12, respectively (where comparisons were made for adults aged 18 or older).
To determine whether differences between substate prevalence rates over two time periods were statistically significant, let 
and 
denote the 2008-2010 and 2014-2016 prevalence rates, respectively, for substate-s. The change between and is defined in terms of the log-odds ratio 
as opposed to the simple difference because the posterior distribution of the is closer to Gaussian than the posterior distribution of the simple difference, 
. The is defined as follows:
, D
where ln denotes the natural logarithm.
The p value given in each of the tables in this document (i.e., Tables 1 to 12) is computed to test the null hypothesis of no change; that is, 
or equivalently 
. An estimate of is given by
, D
where the 
are the 2008-2010 substate estimates presented in the above tables and in the "2008-2010 NSDUH Substate Region Estimates: Excel Tables" at https://www.samhsa.gov/data/ and the 
are the 2014-2016 substate estimates presented in the above tables and in the "2014-2016 NSDUH Substate Region Estimates: Excel Tables" at the same web address.
To compute the posterior variance of 
, that is, 
, let 
and 
, then 
, where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation, as follows:
. D
Note that the 
and 
used here to calculate are the same variances used in calculating the 2008-2010 Bayesian confidence intervals (CIs) and the 2014-2016 Bayesian CIs, respectively. Thus, the posterior variances can be calculated using the published Bayesian CIs, 
and 
, as follows:
where 
and 
.
The correlation between and was obtained by simultaneously modeling the 2008-2010 and the 2014-2016 NSDUH data using a method similar to obtaining the correlation for producing state-level estimates of the variance of . For this simultaneous model, four age groups (12 to 17, 18 to 25, 26 to 34, and 35 or older) by two time periods (2008-2010 and 2014-2016), that is, eight subpopulation-specific models, were fitted, each with its own set of fixed and random effects. 2
To calculate the p value for testing the null hypothesis of no difference 
, it was assumed that the posterior distribution of 
is normal with 
and 
. With , the Bayes p value or significance level for the null hypothesis of no difference is 
, where Z is a standard normal random variate, 
, and 
denotes the absolute value of z. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is, 
. With the posterior distribution of approximately normal, 
is given by the above expression.
| State | 2014-2016 Substate Region Name | 2008-2010 Substate Region Name |
|---|---|---|
| Arizona | Central | Maricopa |
| North | -- | |
| South A | Pima | |
| South B | -- | |
| -- | Rural North | |
| -- | Rural South | |
| Florida | Circuit 1 | Circuit 1 |
| Circuit 2 | -- | |
| -- | Circuit 2 plus Madison and Taylor | |
| Circuits 3 and 8 | -- | |
| Circuit 4 | Circuit 4 | |
| Circuit 5 | Circuit 5 | |
| Circuit 6 | Circuit 6 | |
| Circuit 7 | Circuit 7 | |
| -- | Circuit 8 plus Columbia, Dixie, Hamilton, Lafayette, and Suwannee | |
| Circuit 9 | Circuit 9 | |
| Circuit 10 | Circuit 10 | |
| Circuit 12 | Circuit 12 | |
| Circuit 13 (Hillsborough) | Circuit 13 (Hillsborough) | |
| Circuit 14 | Circuit 14 | |
| Circuit 15 (Palm Beach) | Circuit 15 (Palm Beach) | |
| Circuit 17 (Broward) | Broward (Circuit 17) | |
| Circuit 18 | Circuit 18 | |
| Circuit 19 | Circuit 19 | |
| Circuit 20 | Circuit 20 | |
| Region F – Southern (Circuits 11 and 16) | South (Circuits 11 and 16) | |
| Kentucky | Adanta, Cumberland River, and Lifeskills | Adanta, Cumberland River, and Lifeskills |
| Bluegrass, Comprehend, and North Key | Bluegrass, Comprehend, and North Key | |
| Centerstone | Seven Counties | |
| Communicare and River Valley | Communicare and River Valley | |
| Four Rivers and Pennyroyal | Four Rivers and Pennyroyal | |
| Kentucky River, Mountain, and Pathways | Kentucky River, Mountain, and Pathways | |
| Michigan | Region 1 | Pathway and Western |
| Region 2 | -- | |
| Region 3 | -- | |
| Region 4 | -- | |
| Region 5 | -- | |
| Region 6 | -- | |
| Region 7 | -- | |
| Region 8 | Oakland | |
| Region 9 | Macomb | |
| Region 10 | -- | |
| -- | Detroit City | |
| -- | Genesee | |
| -- | Kalamazoo | |
| -- | Kent | |
| -- | Lakeshore | |
| -- | Mid South | |
| -- | Northern | |
| -- | Riverhaven | |
| -- | Saginaw | |
| -- | Southeast | |
| -- | St. Clair | |
| -- | Washtenaw | |
| Nevada1 | Clark – Region 1 | Clark |
| Region 31 | Rural | |
| Capital District | -- | |
| Rural Frontier | -- | |
| Washoe – Region 2 | Washoe | |
| New York2 | Region 1: Long Island | Region 5 |
| Region 2: New York City2 | Region A | |
| Region 2A: Bronx | Region 1 | |
| Region 2B: Kings | -- | |
| Region 2C: New York | Region 3 | |
| Region 2D: Queens | Region 4 | |
| Region 2E: Richmond | -- | |
| Region 3: Mid-Hudson | -- | |
| Region 4: Capital Region | -- | |
| Region 5: Mohawk Valley | -- | |
| Region 6: North Country | -- | |
| Region 7: Tug Hill Seaway | -- | |
| Region 8: Central | -- | |
| Region 9: Southern Tier | -- | |
| Region 10: Finger Lakes | -- | |
| Region 11: Western | -- | |
| -- | Region 2 | |
| -- | Region 6 | |
| -- | Region 7 | |
| -- | Region 8 | |
| -- | Region 9 | |
| -- | Region 10 | |
| -- | Region 11 | |
| -- | Region 12 | |
| -- | Region 13 | |
| -- | Region 14 | |
| -- | Region 15 | |
| North Carolina | Alliance Behavioral Healthcare 1 | -- |
| Alliance Behavioral Healthcare 2 | -- | |
| Cardinal Innovations Healthcare Solutions 1 | -- | |
| Cardinal Innovations Healthcare Solutions 2 | -- | |
| Cardinal Innovations Healthcare Solutions 3 | Mecklenburg | |
| CenterPoint Human Services | CenterPoint | |
| Eastpointe | Eastpoint | |
| Partners Behavioral Health Management | Pathways | |
| Sandhills Center 1 | Sandhills | |
| Sandhills Center 2 | Guilford | |
| Smoky Mountain Center 1 | Smoky Mountain | |
| Smoky Mountain Center 2 | Western Highland | |
| Trillium Health Resources 1 | East Carolina | |
| Trillium Health Resources 2 | ECCS | |
| -- | Durham | |
| -- | PBH | |
| Washington | Greater Columbia and North Central | -- |
| King | North 2 | |
| North Sound | North 1 | |
| Pierce | -- | |
| Salish | -- | |
| Spokane | -- | |
| SW WA and Great Rivers | -- | |
| Thurston-Mason | -- | |
| -- | East 1 | |
| -- | East 2 | |
| -- | West 1 | |
| -- | West 2 | |
| -- The 2008-2010 or 2014-2016 substate region was not comparable across the two time periods. NOTE: All of the 2014-2016 substate regions are listed for the eight states shown in this exhibit; therefore, substate region names that changed as well as those that did not are provided. 1 In Nevada, an aggregate region for 2014-2016 was comparable with a substate region in 2008-2010. Thus, the aggregate region was included in this exhibit. 2 In New York, an aggregate region for 2014-2016 was comparable with an aggregate region with a different name in 2008-2010. Thus, the aggregate region was included in this exhibit. |
||
Endnotes
1 For details, see the "Comparison of 2002-2003 and 2013-2014 NSDUH State Prevalence Estimates" at https://www.samhsa.gov/data/.
2 For details on this methodology, see the link to the document noted in footnote 1.
3 Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys (Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics). New York, NY: John Wiley & Sons.
Long Descriptions for Equations
Long Description: Equation 1 The log-odds ratio, lor sub s, is defined as the natural logarithm of the ratio of two quantities. The numerator of the ratio is pi 2 sub s divided by 1 minus pi 2 sub s. The denominator of the ratio is pi 1 sub s divided by 1 minus pi 1 sub s, where pi 1 sub s is the 2008-2010 prevalence rate and pi 2 sub s is the 2014-2016 prevalence rate for substate area s.
Long description end. Click here to return to Equation 1.
Long Description: Equation 2 The log-odds ratio estimate, lor hat sub s, is defined as the natural logarithm of the ratio of two quantities. The numerator of the ratio is pi 2 sub s divided by 1 minus pi 2 sub s. The denominator of the ratio is pi 1 sub s divided by 1 minus pi 1 sub s, where pi 1 sub s is the 2008-2010 prevalence rate and pi 2 sub s is the 2014-2016 prevalence rate for substate area s.
Long description end. Click here to return to Equation 2.
Long Description: Equation 3 The covariance between the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat is equal to the correlation between the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat multiplied by the square root of the product of the variances of the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat.
Long description end. Click here to return to Equation 3.
Long Description: Equation 4 Variance v of the natural logarithm of Theta sub i is equal to the square of quantity q. Quantity q is calculated as the difference between capital U sub i and capital L sub i divided by 2 times 1.96, where i takes values 1 and 2.
Long description end. Click here to return to Equation 4.
