Methodology Used to Estimate the Bayes Significance Level for the Null Hypothesis of
No Change between the 2002-2004 and 2012-2014 Population Percentages
This discussion focuses on the methodology used to conduct statistical tests of significance for comparing National Survey on Drug Use and Health (NSDUH) substate population percentages between 2002-2004 and 2012-2014. Population percentages shown in Tables 1 to 22 earlier in this document are based on small area estimation (SAE) procedures, and the methodology used to compare these population percentages is the same methodology used to compare the state small area estimates.1 However, because some substate regions within 24 states were defined differently for the two time periods (i.e., 2002-2004 and 2012-2014), significance test results (based on the Bayes significance levels for the null hypothesis of no change: p value) could not be produced for these regions. In this document's tables, a double hyphen ("--") is used to denote substate regions that are not defined consistently across the two time periods. Note that all of the regions in Georgia, North Carolina, and South Dakota changed across the two time periods, so only estimates at the state level are included in the tables for these three states.
For details about the 2012-2014 substate region definitions, see the "2012-2014 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/. For details about the 2002-2004 substate region definitions, see Section D of the "Substate Estimates from the 2002-2004 National Surveys on Drug Use and Health," which is archived at the same web address. In eight states, the names of some substate regions changed across the two time periods, but the definitions remained the same. States that have such comparable regions with different names are listed in Exhibit A.1 at the end of this document. The exhibit includes information regarding only these eight states; however, all of the 2012-2014 substate regions for each of these states are listed, although aggregate regions are included only if an aggregate region is comparable with a substate region with a different name.
Comparisons of substate population percentages between 2002-2004 and 2012-2014 were made only for individuals aged 12 or older, except for the underage alcohol use outcome in Table 11 and the underage binge alcohol use outcome in Table 12 (where comparisons were made for individuals aged 12 to 20).
To determine whether differences between substate prevalence rates over two time periods were statistically significant, let 
and 
denote the 2002-2004 and 2012-2014 prevalence rates, respectively, for substate-s. The change between and is defined in terms of the log-odds ratio 
as opposed to the simple difference because the posterior distribution of the is closer to Gaussian than the posterior distribution of the simple difference, 
. The is defined as follows:
, D
where ln denotes the natural logarithm.
The p value given in each of the tables in this document (i.e., Tables 1 to 22) is computed to test the null hypothesis of no change; that is, 
or equivalently 
. An estimate of is given by
, D
where the 
are the 2002-2004 substate estimates presented in the above tables and in Section C of the "Substate Estimates from the 2002-2004 National Surveys on Drug Use and Health" archived at https://www.samhsa.gov/data/ and the 
are the 2012-2014 substate estimates presented in the above tables and in the "2012-2014 NSDUH Substate Region Estimates: Excel Tables" at the same web address.
To compute the posterior variance of 
, that is, 
let 
and 
, then 
, where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation, as follows:
. D
Note that the 
and 
used here to calculate are the same variances used in calculating the 2002-2004 Bayesian confidence intervals (CIs) and the 2012-2014 Bayesian CIs, respectively. Thus, the posterior variances can be calculated using the published Bayesian CIs, 
and 
, as follows:
where 
and 
.
The correlation between and was obtained by simultaneously modeling the 2002-2004 and the 2012-2014 NSDUH data using a method similar to obtaining the correlation for producing state-level estimates of the variance of . For this simultaneous model, four age groups (12 to 17, 18 to 25, 26 to 34, and 35 or older) by two time periods (2002-2004 and 2012-2014), that is, eight subpopulation-specific models, were fitted, each with its own set of fixed and random effects.2
To calculate the p value for testing the null hypothesis of no difference 
it was assumed that the posterior distribution of 
is normal with 
and 
. With , the Bayes p value or significance level for the null hypothesis of no difference is 
, where Z is a standard normal random variate, 
, and 
denotes the absolute value of z. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is, 
. With the posterior distribution of approximately normal, 
is given by the above expression.
| State | 2012-2014 Substate Region Name | 2002-2004 Substate Region Name |
|---|---|---|
| -- The 2002-2004 or 2012-2014 substate region was not comparable across the two time periods. NOTE: All of the 2012-2014 substate regions are listed for the eight states shown in this table; therefore, substate region names that changed as well as those that did not are provided. 1 In Nevada, an aggregate region for 2012-2014 was comparable with a substate region in 2002-2004. Thus, the aggregate region was included in this exhibit. |
||
| Arkansas | Catchment Area 1 | -- |
| Catchment Area 2 | Areas 2 and 3 | |
| Catchment Area 3 | Areas 4 and 7 | |
| Catchment Area 4 | -- | |
| Catchment Area 5 | -- | |
| Catchment Area 6 | -- | |
| Catchment Area 7 | -- | |
| Catchment Area 8 | Area 9 | |
| -- | Areas 1 and 5 | |
| -- | Area 6 | |
| -- | Areas 8 and 12 | |
| -- | Areas 10, 11, and 13 | |
| Florida | Broward (Circuit 17) | District 10 (Broward) |
| Circuit 1 | District 1 | |
| Circuit 2 plus Madison and Taylor | -- | |
| Circuit 4 | -- | |
| Circuit 5 | District 13 | |
| Circuit 6 | -- | |
| Circuit 7 | -- | |
| Circuit 8 plus Columbia, Dixie, Hamilton, Lafayette, and Suwannee | -- | |
| Circuit 9 | -- | |
| Circuit 10 | District 14 | |
| Circuit 12 | -- | |
| Circuit 13 (Hillsborough) | -- | |
| Circuit 14 | -- | |
| Circuit 15 (Palm Beach) | District 9 (Palm Beach) | |
| Circuit 18 | -- | |
| Circuit 19 | District 15 | |
| Circuit 20 | District 8 | |
| South (Circuits 11 and 16) | District 11 | |
| -- | District 2 | |
| -- | District 3 | |
| -- | District 4 | |
| -- | District 7 | |
| -- | District 12 | |
| -- | Suncoast | |
| Maine | Aroostook | -- |
| Central | -- | |
| Cumberland | Region 1 (Cumberland) | |
| Downeast | -- | |
| Midcoast | Region 5 | |
| Penquis | -- | |
| Western | -- | |
| York | Region 2 (York) | |
| -- | Region 3 | |
| -- | Region 4 | |
| -- | Region 6 (Penobscot) | |
| -- | Region 7 | |
| Michigan | Region 1 | -- |
| Region 2 | -- | |
| Region 3 | -- | |
| Region 4 | -- | |
| Region 5 | -- | |
| Region 6 | -- | |
| Region 7 | -- | |
| Region 8 | Oakland | |
| Region 9 | Macomb | |
| Region 10 | -- | |
| -- | Bay | |
| -- | Detroit City | |
| -- | Genesee | |
| -- | Kalamazoo | |
| -- | Kent | |
| -- | Lakeshore | |
| -- | Mid South | |
| -- | Northern | |
| -- | Pathways | |
| -- | Saginaw | |
| -- | Southeast | |
| -- | St. Clair | |
| -- | Washtenaw | |
| -- | Western | |
| Nevada | Clark - Region 1 | Clark |
| Region 31 | Rural | |
| Capital District | -- | |
| Rural Frontier | -- | |
| Washoe - Region 2 | Washoe | |
| Pennsylvania | Region 1 (Allegheny) | Region 1 (Allegheny) |
| Regions 3, 8, 9, and 51 | -- | |
| Regions 4, 11, 37, and 49 | Regions 4, 11, 42, and 48 | |
| Regions 5, 18, 23, 24, and 46 | Regions 5, 20, 26, 28, and 40 | |
| Regions 6, 12, 16, 31, 35, 45, and 47 | -- | |
| Regions 7, 13, 20, and 23 | Regions 7, 13, 22, and 37 | |
| Regions 10, 15, 27, 32, 43, and 44 | -- | |
| Regions 17 and 21 | Regions 19 and 23 | |
| Regions 19, 26, 28, and 42 | Regions 21, 30, 32, and 49 | |
| Regions 22, 38, 40, 41, and 48 | Regions 17, 24, 27, 43, and 47 | |
| Regions 29 and 34 | Regions 33 and 38 | |
| Regions 30 and 50 | -- | |
| Regions 36 (Philadelphia) | Region 18 (Philadelphia) | |
| -- | Regions 2, 3, 8, and 9 | |
| -- | Regions 6, 12, 16, 35, 39, 41, 44, and 45 | |
| -- | Regions 10, 14, 15, 25, 31, 36, and 46 | |
| -- | Regions 29 and 34 | |
| Washington | Greater Columbia and North Central | -- |
| King | Region 4 (King) | |
| North Sound | Region 3 | |
| Peninsula | -- | |
| Pierce | -- | |
| Spokane | -- | |
| SW WA and Timberlands | -- | |
| Thurston-Mason | -- | |
| -- | Region 1 | |
| -- | Region 2 | |
| -- | Region 5 | |
| -- | Region 6 | |
| West Virginia | Region I | Northern A and B |
| Region II | -- | |
| Region III | South Central I | |
| Region IV | -- | |
| Region V | -- | |
| Region VI | -- | |
| -- | Eastern Highland | |
| -- | Northern C and D | |
| -- | South Central II | |
| -- | South Central III | |
| -- | Southern I and III | |
| -- | Southern II | |
Endnotes
1 For details, see the "Comparison of 2002-2003 and 2013-2014 NSDUH State Prevalence Estimates" at https://www.samhsa.gov/data/.
2 For details on this methodology, see the link to the document noted in footnote 1.
3 Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys (Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics). New York, NY: John Wiley & Sons.
Long Descriptions for Equations
Long Description: Equation 1 The log-odds ratio, lor sub s, is defined as the natural logarithm of the ratio of two quantities. The numerator of the ratio is pi 2 sub s divided by 1 minus pi 2 sub s. The denominator of the ratio is pi 1 sub s divided by 1 minus pi 1 sub s, where pi 1 sub s is the 2002-2004 prevalence rate and pi 2 sub s is the 2012-2014 prevalence rate for substate area s.
Long description end. Click here to return to Equation 1.
Long Description: Equation 2 The estimate of the log-odds ratio, lor hat sub s, is defined as the natural logarithm of the ratio of two quantities. The numerator of the ratio is p 2 sub s divided by 1 minus p 2 sub s. The denominator of the ratio is p 1 sub s divided by 1 minus p 1 sub s, where p 1 sub s is the 2002-2004 small area estimate and p 2 sub s is the 2012-2014 small area estimate for substate area s.
Long description end. Click here to return to Equation 2.
Long Description: Equation 3 The covariance between the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat is equal to the correlation between the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat multiplied by the square root of the product of the variances of the natural logarithm of Theta 1 hat and the natural logarithm of Theta 2 hat.
Long description end. Click here to return to Equation 3.
Long Description: Equation 4 Variance v of the natural logarithm of Theta sub i is equal to the square of quantity q. Quantity q is calculated as the difference between capital U sub i and capital L sub i divided by 2 times 1.96, where i takes values 1 and 2.
Long description end. Click here to return to Equation 4.
