Karl
Popper is the man who defined science for my generation and probably
several more. His book The Logic of Scientific Discovery was published
before I was born (1934) and translated by him into English in the late
1950s. I read it in the early 1960's.
Popper defined science as
the cumulative testing of falsifiable hypotheses. So all scientific
statements must be made in a falsifiable form. Yes: "Ice transforms to
water at 0 degrees C". No: "Global climate is changing".
Any experiment with humans must be double blind with a control group. Double blind means the person doing the experiment can not know which subjects are in the control group and which aren't. The control group must be as identical as possible to the test group.
I have spent 50 years designing research experiments. It is always difficult and requires imagination. I rarely see good research done anywhere.
I want to report one of my most unusual experiments. This is from an earlier blog: In 1985 I debated Dr. Peter Sherrill, who later became one of my closest friends, over the issue of classic random sample versus a Kinsey 100% group sample. The debate was public, in a California Public Utilities Commission meeting room. I was on one end of a ten-foot table representing Public Advocates and Peter was at the other end representing Field Research and Pacific Bell.
Two experts on survey research brought together to resolve a thorny issue of public concern.
In 1984, some Mexican labor activists reported to Robert Gnaizda (Public Advocates) that farm workers in Salinas had monthly phone bills of $32, with every feature from call forwarding to three-way calling. None of these farm workers could speak English.
Gnaizda phoned around the state and found that the same phenomenon was occurring in Chinese and Vietnamese communities.
What was
happening was later proved in utility hearings. Pacific Bell sales
people, who had strong prize incentives for signing up customers, would
sign up a customer for the maximum phone package of $32 anytime the
sales person reached a person who didn’t speak English. Don’t speak
English … you just bought $32 a month of technical phone services.
The public debate was between Dr. Sherrill (photo on left) and me over the issue of measuring the extent of the marketing abuse damage. How many people had been signed up for services without knowing what was going on?
Dr. Sherrill made the argument that classic random phone sampling with multi-lingual interviewers would find the correct answer. I argued that the most reliable method would be a 100% in-person sample of ethnic groups. The survey money would be paid to the ethnic organizations for 100% turnout and participation of their members. This is the Kinsey sampling method.
We were both persuasive or maybe we both weren’t. Pacific Bell was ordered to do both kinds of sampling to find how many people had been cheated and how much money had been falsely taken from non-English speakers.
Dr. Sherrill (Peter) and I designed and agreed upon the questionnaire.
The day arrived when
the computer printouts from both surveys were ready. Peter and I sat
down in a Corey Canapary & Galanis Research office (they had done my
ethnic group survey) near Union Square. We looked at the responses to
the first question. Then the second question, then all the questions on
the first page, then we rapidly went through the entire printout.
The answers were the same from the two distinctly different survey methods. I don’t mean the same in a statistical sense, I mean the same in the sense of being one or two points different on every single question.
Peter and I were stunned. As professionals we reverted to the term “robust data” meaning the truth will out regardless of the method. But we were still stunned at the nearly identical outcomes.
This research in 1985 was probably the most careful, thorough and reliable proof that the Kinsey research method was excellent. Superb.
Peter and my research results were never publish in a survey research journal.
Any experiment with humans must be double blind with a control group. Double blind means the person doing the experiment can not know which subjects are in the control group and which aren't. The control group must be as identical as possible to the test group.
I have spent 50 years designing research experiments. It is always difficult and requires imagination. I rarely see good research done anywhere.
I want to report one of my most unusual experiments. This is from an earlier blog: In 1985 I debated Dr. Peter Sherrill, who later became one of my closest friends, over the issue of classic random sample versus a Kinsey 100% group sample. The debate was public, in a California Public Utilities Commission meeting room. I was on one end of a ten-foot table representing Public Advocates and Peter was at the other end representing Field Research and Pacific Bell.
Two experts on survey research brought together to resolve a thorny issue of public concern.
In 1984, some Mexican labor activists reported to Robert Gnaizda (Public Advocates) that farm workers in Salinas had monthly phone bills of $32, with every feature from call forwarding to three-way calling. None of these farm workers could speak English.
Gnaizda phoned around the state and found that the same phenomenon was occurring in Chinese and Vietnamese communities.
The public debate was between Dr. Sherrill (photo on left) and me over the issue of measuring the extent of the marketing abuse damage. How many people had been signed up for services without knowing what was going on?
Dr. Sherrill made the argument that classic random phone sampling with multi-lingual interviewers would find the correct answer. I argued that the most reliable method would be a 100% in-person sample of ethnic groups. The survey money would be paid to the ethnic organizations for 100% turnout and participation of their members. This is the Kinsey sampling method.
We were both persuasive or maybe we both weren’t. Pacific Bell was ordered to do both kinds of sampling to find how many people had been cheated and how much money had been falsely taken from non-English speakers.
Dr. Sherrill (Peter) and I designed and agreed upon the questionnaire.
The answers were the same from the two distinctly different survey methods. I don’t mean the same in a statistical sense, I mean the same in the sense of being one or two points different on every single question.
Peter and I were stunned. As professionals we reverted to the term “robust data” meaning the truth will out regardless of the method. But we were still stunned at the nearly identical outcomes.
This research in 1985 was probably the most careful, thorough and reliable proof that the Kinsey research method was excellent. Superb.
Peter and my research results were never publish in a survey research journal.
Oh yes, Pacific Bell had to pay $60 million in refunds and $16 million in penalties.
-----------------------------
Top photo Popper, bottom photo Kinsey.