Over a lifetime of being a statistician there are a few things I have observed.
Sample size
Sampling technique
Bayes Theorem
Sample size
I have found over many decades of doing ‘truly random’ surveys on a wide variety of subjects that a sample of 25 can be just as accurate as a good survey of 400 or more. The key element is that the survey must be in-person, open-ended. People cannot fit their opinions into a fixed questionnaire statement; they require a more complex environment to respond. A good listener with an open-ended series of multiple questions can get a survey response that matches the final outcome of voting or buying behaviors.
This is why a 12 person jury is trusted by people whose lives depend on it.
Sampling technique
Kinsey used an interesting survey technique. He would interview a small population, like a PTA group, a prison population or a Rotary club if he could get 100% participation.
For a California Public Utilities Commission study of a large scale fraud (the fraud was carried out by a utility on people who didn't speak English), I did two types of surveys with the same questionnaire. One survey was of a standard 400-person random sample of Latinos and 400 Chinese-Americans (the subjects of the fraud). A second one was of several 100% surveys of activist groups in each population. The survey instrument had more than 25 questions. The entire survey work for each study was done by two separate commercial survey research firms.
The results: on each of the more than 25 questions, the responses were identical or nearly identical regardless of the sampling method.
Kinsey was a pioneer in many ways.
Bayes Theorem
Bayes Theorem is two centuries old and very simple. Multiple probability estimates can be multiplied together. For example: 50% of all men in Chicago wear a wrist watch. Fourteen percent of all men are left handed. Using Bayes Theorem .50 x.14 is .07 so 7% of all men are left-handed and wear a wrist watch. No dispute unless something else is going on.
Well, too often, something else is going on. A meta study across many nations, of adults, with large samples, tested whether using zinc treatments to reduce cold symptoms was effective. Results: no.
I looked at individual studies and found that nearly all were zinc gluconate and one was zinc acetate. They were not the same. The zinc acetate in one study was effective; the others for zinc gluconate weren’t. The overall results were wrong.
When using Bayes be careful that the subjects are in the same population.