Brief introduction to survey data analysis ideas

• simple or one-stage cluster sample select a sample of clusters from a list of all of the clusters and then select all of the units from the selected clusters
  • e.g. sample streets from a list of streets and then take all houses in the sampled streets

• Multistage cluster sampling
  • employs the clustering idea at several levels
    • • e.g. sample schools from a list of schools and, for each selected school, sample classes from the list of classes in that school and then either take all or a sample of students from each of the selected classes
    • The 1st stage of clustering is the first level at which sampling occurs (schools in the Example)
    • The 2nd stage of clustering is the 2nd level at which sampling occurs (classes in the Example)
    • and so on

Primary Sampling Units (psu)

• The psu’s are the entities selected at the first level at which sampling is performed
  • If no cluster sampling involved (srs or stratified sampling alone)
    • the psu is the unit/person selected into the sample
• Where cluster sampling is involved:
  • the psu are the entities selected at the 1st stage of sampling
    • • e.g. If we sample schools, then classes within schools and then, finally, students within classes, then the psu’s are the schools
• Practical aside
  • To make corrections that are adequate for most purposes, computer programs only need to know about the first stage of cluster sampling. (As of SAS 9.1, it only asks for and uses the first stage information)

• In particularly common for cluster samples to be taken from within every stratum of the population
  • e.g. we could take a cluster sample of schools from every region of the country (here regions are the strata and schools are the clusters)

What is the most important new skill to be learned for analysing survey data?

• Was stratified sampling used? If so, what were the strata?
• Was cluster sampling used? If so, what were the clusters?
• What were the selection probabilities?
  • • most programs want “sampling weights” which are the inverse (1 over) probabilities of selection

• The “precision” of an estimate
  • relates to how variable estimates of this type would be if we repeatedly kept taking samples
  • It’s an idea we try to capture using the standard error as our measure
    • • Smaller std error = more precise estimate.
      • larger std error = less precise estimate.In most survey applications, a confidence interval is approx.
        • CI for true value: estimate ± 2×se

so higher precision translates into narrower confidence interval

• The “accuracy” of an estimate
  • incorporates any bias as well as the sampling variability
  • It’s an idea we try to capture using the Mean Squared Error (MSE) as our measure
    • • • • MSE = se² + bias² = variance + bias²
  • High accuracy translates to low Mean Squared Error

• Remember back to “normal statistics”
  • Our measure of precision for a sample mean was se(x_bar) = s / sqrt(n)
  • Our measure of precision for a sample proportion was
    • • • • se(p_hat) = sqrt{p_hat x (1-p_hat)/n}
  • As the sample size increases our estimates tend to get more precise scaling as 1/sqrt(n)
    • • e.g., to double the precision (i.e. halve the std error) we need to take 4 times as large a sample, etc
• In general, we get greater precision (smaller std errors, narrower CIs) if we take larger samples.
• Standard errors tend to decrease roughly proportionately to 1 / sqrt(sample size)
  • • • (if we are talking about the standard error of an estimate that relates only to a subgroup, e.g. the mean income for Europeans, then the relevant “sample size” is the sample size for that subgroup)

• It common in surveys, for units in the population to be selected for the sample with different probabilities
  • It is particularly common for units in different strata to be sampled at different rates (cf. Maori and European above)
  • It is critically important, however, that no unit in the population has a zero probability of being selected

Why use unequal probabilities of selection?

• Often used in the context of stratified sampling
  • Particularly if reporting is also to be done at the level of the strata themselves, as well as for the whole population, likely to want to ensure large enough sample sizes in each stratum to ensure sufficiently precise stratum-level estimates (e.g. income levels accurately estimated for each of the ethnicity)
    • • Thus units in small strata will have higher probabilities of selection than units in large strata
      • e.g. take equal numbers of Maori, Pacifica and Other
  • Often makes good practical sense to include more of “the big important units”
    • • e.g. economic surveys often survey all of the very big companies
• Can sometimes increases precision of estimates

What goes wrong if we ignore it?

• If, when doing the analysis, we ignore the fact that the data has been sampled using unequal probabilities of selection we can get the wrong answers for almost everything
• If the probabilities of selection are not all the same, then we have to give the program:
  • either the selection probabilities
  • or, “the weights” = 1/selection probabilities -- this is what we’ll usually be doing
    • The basic idea of weighting is that, “the weight to be assigned for a unit in the sample is the number of units in the population that the sampled unit represents”
      • e.g. if we select n_j =100 people at random from stratum j which has N_j =1,000 people in it, then each of the 100 people selected “represents” 10 people
        • more generally, each of the one of the n_j people selected represents people
    • or, we can supply other information from which the program can work these things out
      • • e.g. we might give the program the numbers of people in the population in each stratum

Reminder: What is the most important new skill to be learned for analysing survey data?

• Was stratified sampling used? If so, what were the strata?
• Was cluster sampling used? If so, what were the clusters?
• What were the selection probabilities? (many programs want “weights” which are 1 over the probability of selection)
• Should we be making finite population corrections? (see later)

• It is also very important to be able to describe the sampling structures to software. This second skill is a much easier skill to learn, however.

• We call studies addressing this “descriptive” studies
• They want to summarise the way it was in a particular population during a particular time span
  • And present estimates such quantities as population means, totals, counts, proportions, ratios
    • often broken out by region, age, sex, calendar year, …
• These are real, finite populations
  • If we had data from a complete census (rather than sampling) we would just calculate these summaries and there would be no uncertainty
    • So no need for standard errors, confidence intervals etc. We’d know exactly what the population summary values were
  • But usually we sample. The uncertainty is caused by the randomness in the selection of who gets into the sample that we use to calculate our estimates
    • Our confidence intervals are interval estimates of the quantity that we would have obtained if we’d been able to calculate the summary for the whole finite population
  • No other area of statistics really cares about these types of inference

• Many researchers are most interested in “How does it work?”, or “Why is it like that”? or Predicting what will happen if we did it again or in the future
  • We call these studies with these aims, “analytic”
    • We are interested in the nature of relationships between variables, in differences, regressions, … – all the standard STATS 20x stuff and beyond
  • If we had data on a whole finite population we would analyse the data thinking in terms of the population itself having been generated by some random process and we would be interested in patterns in the way this random data was generated
    • Even with complete data from the whole finite population, there would still be uncertainty

• Analysis needs to allow for uncertainties in the random process that generated the finite population data plus additional uncertainties generated by sampling from that finite population

• So with analytic studies …
  • Even with complete data from the whole finite population, there would still be uncertainty
  • Analysis needs to allow for uncertainties in the random process that generated the finite population data plus additional uncertainties generated by sampling from that finite population
    • We can do all of the STATS 20x analyses, the linear and logistic regressions …
    • All that is new here is that we use special programs designed for survey data and the program needs to be told how the sampling was done
    • Apart from that it is pretty much business as usual

• Drawing a unit out of a hat, measuring it, putting it back in the hat, and then measuring it again on some subsequent draw seldom makes any practical sense

• The usual standard errors of estimates of characteristics of the finite population are too big if the sample size n makes up a substantial fraction of the population size N
• To compensate we use co-called finite population corrections (fpc)
• Roughly, actual std error of an estimate = Usual std error x sqrt(1-f)
  • where f, is the so-called “sampling fraction” (= sample size/popⁿ size)

• Generally, we use finite population corrections for descriptive studies
  • • (interested in describing the way the population is)
  • but not analytic studies
    • (interested in the processes that produced a population like that)
• We will address the issues around fpc’s in more detail in a later Lecture
  • including why the standard errors under sampling from a finite population should be smaller than the standard ones,