|Year : 2019 | Volume
| Issue : 1 | Page : 48-50
Survey research methods: A guide for creating post-stratification weights to correct for sample bias
Kenneth D Royal
Department of Clinical Sciences, North Carolina State University, Raleigh, North Carolina, USA
|Date of Web Publication||30-May-2019|
Dr. Kenneth D Royal
Department of Clinical Sciences, College of Veterinary Medicine, North Carolina State University, Raleigh, North Carolina
Source of Support: None, Conflict of Interest: None
Nonrepresentative data pose one of the greatest validity threats in survey research. Samples that are underrepresented and/or overrepresented based on demographic subgroups can introduce bias that distorts both the accuracy and the inferences made about the results. This article discusses the concept of poststratification weighting, a post hoc statistical procedure used to correct for sampling bias in survey research studies. Procedural steps for calculating poststratification weights are presented, and an example involving a simulated cohort of students in a medical school is provided for demonstration purposes. SPSS statistical software coding is presented to help researchers get started with their own calculations of poststratification weights.
Keywords: Assessment, bias, evaluation, health surveys, medical education, statistics, survey research, surveys
|How to cite this article:|
Royal KD. Survey research methods: A guide for creating post-stratification weights to correct for sample bias. Educ Health Prof 2019;2:48-50
|How to cite this URL:|
Royal KD. Survey research methods: A guide for creating post-stratification weights to correct for sample bias. Educ Health Prof [serial online] 2019 [cited 2020 Jul 13];2:48-50. Available from: http://www.ehpjournal.com/text.asp?2019/2/1/48/259389
| Introduction|| |
In medical and health professions education, most surveys are administered in the context of a census study in which all members of a population (e.g., a student cohort) are surveyed. With exception to surveys that require participation, it is typical for only some individuals to complete the survey. When surveys fail to achieve a 100% participation rate, response bias becomes a concern. In social research, various subpopulations often respond to survey items differently according to factors such as race, gender, and other demographic characteristics. As a result, underrepresentation or overrepresentation from members of various subpopulation groups can introduce bias into survey results. The consequence is that statistical software will simply analyze the data given, thus providing greater weight to those individuals who were overrepresented and lesser weight to those individuals who were underrepresented. This results in a validity threat, as both the accuracy and the inferences made about the results are distorted by the sampling bias. Given this reality, it is critical that survey researchers make every effort to produce accurate, nonbiased estimates that characterize the views, attitudes, beliefs, etc., of both the entire population and its major subpopulation groups.
Typically, survey researchers attempt to minimize response bias by obtaining representative samples. In short, representative samples help ensure that one's findings may be generalizable to the population from which the sample was drawn. A major advantage of census studies in the context of medical and health professions education is that population parameters, such as demographic characteristics and other auxiliary statistics, typically are known. Thus, with the use of a chi-squared test, researchers can determine if the participants that completed the survey proportionally resemble the population of interest based on key characteristics. If chi-squared tests confirm that the sample resembles the population, then the researcher may proceed with the analysis and subsequent reporting of results. However, if chi-squared tests indicate that the sample is disproportionate, then the researcher should take some action to correct for this bias. One option is to obtain additional data from members of subpopulation groups that are underrepresented in the data. However, given the relatively small size of most populations, stratified sampling techniques typically are only marginally helpful in this context. Thus, researchers often are forced to consider other alternatives.
One robust alternative to correct for sample distributions that do not perfectly resemble population distributions is to apply poststratification weights. In short, poststratification weighting involves taking sample data and aligning the representation of various subpopulation groups to match that of the known population. As the name implies, poststratification weights are calculated after all data are collected. When the procedure is performed correctly, extant data are statistically adjusted to reflect population parameters, making results both more accurate and generalizable across the population of interest. Thus, the aim of this article is to provide an overview of poststratification weighting and demonstrate how this procedure can be leveraged to obtain more accurate results in many, if not most, medical and health professions education survey contexts.
| Procedural Steps for Creating Poststratification Weights|| |
First, let us consider the procedural steps necessary for calculating weight values.
- Step 1: Create a table to assemble your variables
- Step 2: Populate your values for Population (N) and Sample (n), where appropriate
- Step 3: Calculate a total count for the Population (N) and Sample (n) columns
- Step 4: Populate the Proportion of Population column by dividing the value for each Combined Variable in the Population (N) column by its column Total
- Step 5: Populate the Proportion of Sample column by dividing the value for each Combined Variable in the Sample (n) column by its column Total
- Step 6: Calculate Weight by dividing the value in each cell of the Proportion of Population column by the value in each cell of the Proportion of Sample column.
Next, let us apply these steps using an illustrative example.
An illustrative example
Suppose a 1st-year medical school cohort consists of 100 students. Of those 100 students, 50 identified as male and 50 identified as female. With respect to race/ethnicity, 70 students self-reported as White, 20 as Black, and 10 as Other. These data serve as the auxiliary statistics for this exercise.
First, we need to produce a crosstab contingency table [Table 1] to establish counts for each combination of race and gender variables (see Combined Variables column). Known student cohort values are entered into the Population (N) column. For this exercise, let us assume that 57 students completed the survey. Thus, next, we need to identify which 57 students of the 100 in the population completed the survey and similarly provide counts for each combination of race and gender in the Sample (n) column. Let us also assume that our sample participants responded in a disproportionate manner (thus justifying the need for poststratification weights), with females responding in greater numbers than males and Black students responding in greater numbers than White or Other students. Simulated values are provided in the Sample (n) column. Proportional values are then created for both the population parameters (e.g., 35 White males divided by 100 students in the total population is 0.350) and the sample's statistics (e.g., 15 White males divided by 57 students in the sample of participants is 0.263) by dividing each value by its respective total count. Proportion of Population values are then divided by Proportion of Sample values (e.g., 0.350 divided by 0.263 equals 1.330) to determine the Weight. A visual inspection of the weights provides a quality assurance check confirming that the values are correct. Next, let us identify how to apply weights using IBM SPSS Statistics for Windows, Version 25.0. (IBM Corp., Armonk, NY, USA).
Applying weights in a statistical software package
After weights are calculated, the weights need to be applied to the data. This process will vary depending on the statistical software program used, but the essence of the process is generally the same. For convenience, SPSS syntax is provided in this example. Suppose the coding schema in the dataset for Race is 1 = White, 2 = Black, and 3 = Other and for Gender is 1 = Male and 2 = Female, the following coding schema would create a new variable (named Weight).
If (Race = 1 and Gender = 1) Weight = 1.330.
If (Race = 2 and Gender = 1) Weight = 1.140.
If (Race = 3 and Gender = 1) Weight = 2.850.
If (Race = 1 and Gender = 2) Weight = 0.798.
If (Race = 2 and Gender = 2) Weight = 0.713.
If (Race = 3 and Gender = 2) Weight = 0.950.
Finally, we would access the weighting function in the software program to ensure that weights are activated and the analyses are conducted using these weights. In SPSS, we would go to Data, select Weight Cases, select Weight Cases By, select the name of the weighting variable (Weight), and then, click OK. This will activate the weights, and all outputs will be weighted accordingly. Once the statistical analysis is performed, the output should be inspected again to ensure that the weighting was successful.
| Concluding Remarks|| |
Poststratification weights offer an effective approach for correcting bias from overrepresented and underrepresented samples. The technique can also help discern the degree to which bias exists should a researcher choose to compare weighted versus unweighted results. The weighting process is relatively straightforward and can be applied to many survey research studies conducted in the field of medical and health professions education.
As noted previously, poststratification weights cannot be accurately calculated unless auxiliary statistics are available. Ideally, auxiliary statistics will consist of exact population parameters, as inexact estimates of a population will result in some measurement error that will be retained even after the weighting process.
There are a number of ways to produce poststratification weights. The method presented in this article is only one approach and was selected because it is a method that most medical and health professions education researchers can perform without having to consult a statistician or psychometrician for assistance. Persons with familiarity with other statistical software programs (e.g., SAS, STATA, and R) can similarly perform these functions. In fact, many programs have macros and other special features that can automate the process. Readers who are more comfortable in performing statistical analyses with other software programs are encouraged to consult the “Help” function within the software and/or perform an online search for tutorials on how to calculate weights using other programs.
There are some additional considerations that survey researchers should take into account. First, it is a good practice to report both weighted and unweighted values as part of the presentation of results. While many consider only the weighted values to be of importance, reporting unweighted values will provide transparency to readers. In addition, it is important to note that calculating weights typically results in an increase in the size of standard errors associated with the estimates. Therefore, for studies in which statistical precision is paramount, researchers should use a statistical procedure that adjusts standard errors based on the unweighted N, as opposed to the weighted N. Perhaps, the biggest problem with poststratification weights is that additional bias may result for subgroups that are not taken into account as part of the weighting process. Therefore, researchers should report weighted data only for those variables that were adjusted and refrain from speculating on how other subpopulations responded. Finally, it should be noted that the example presented in this study is a rather rudimentary example of poststratification weights. Studies involving multivariate data can quickly become increasingly complicated; thus, researchers should consult comprehensive texts by Valliant et al., Bethlehem and Biffignandi, and Biemer and Christ  for additional guidance on how to use poststratification and other types of statistical weights in these contexts.
Financial support and sponsorship
Conflicts of interest
Dr. Royal is the editor-in-chief of Education in the Health Professions. All peer-review activities relating to this manuscript were independently performed by other members of the editorial board.
| References|| |
Royal KD. Four tenets of modern validity theory for medical education assessment and evaluation. Adv Med Educ Pract 2017;8:567-70.
Valliant R, Dever JA, Kreuter F. Practical Tools for Designing and Weighting Survey Samples. New York: Springer; 2013.
Bethlehem J, Biffignandi S. Wiley Handbooks in Survey Methodology: Handbook of Web Surveys. Hoboken, US: Wiley; 2011.
Biemer PP, Christ LL. Weighting survey data. In: de Leeuw ED, Hox J, Dillman D, editors. International handbook of survey methodology. New York, NY: Routledge; 2008.