Term
| For each "world-shaking" issue, what is the first (perhaps most important) task? |
|
Definition
| Deciding whether an issue is "significant." |
|
|
Term
| For each "world-shaking" issue, what is the second task? |
|
Definition
| To set priorities among the issues to decide which should get worked on first. |
|
|
Term
| For each "world-shaking" issue, what is need for both the first and second task? |
|
Definition
| The determination of significance and priority setting both require data. |
|
|
Term
For each "world-shaking" issue, what is the next task after the first two?
What does this require? |
|
Definition
Next task is to set a goal.
Specifying a method of measuring progress toward attaining the goal. |
|
|
Term
| For each "world-shaking" issue, what w things does the task after the first two help? |
|
Definition
| Setting a goal is helpful in deciding when to declare victory. Also helps decide what policies to consider. |
|
|
Term
| For each "world-shaking" issue, what tends to happen without the task after the first two? |
|
Definition
| Without a goal, programs tend to drag on indefinitely. |
|
|
Term
| What is a drawback of insisting on good data. |
|
Definition
| Available data is often incomplete, biased, or nearly irrelevant. Data can be misused, misinterpreted, or the quest for it can be distracting. |
|
|
Term
What does the data never do?
What happens with it? |
|
Definition
Never "speaks for itself."
Someone decides what is relevant and how its presented to decision makers. |
|
|
Term
What exacerbates the limitations of the data?
Why?
How is this transmitted? |
|
Definition
The need for quick answers.
Public wants quick action when they're awaken to an issue.
This urgency is transmitted to policymakers, then to the "data-miners." (i.e. you want it bad, you get it bad). |
|
|
Term
When the time horizon for getting the facts together is a few days, what is the best source of information?
Why? |
|
Definition
The telephone network.
One can quickly discover who are experts and quickly yield large amount of info. |
|
|
Term
What is helpful to access the telephone network.
How? |
|
Definition
To be known, by name, to experts.
Being in prestigious job gets their attention, not always their candor. |
|
|
Term
What ist 1st step for accessing phone network?
What else must be determined? |
|
Definition
Learning the views of the experts.
Legitimate views and how much confidence to place in each view. |
|
|
Term
When does congress usually become interested in long-term study?
Why? |
|
Definition
When they do not want to make a decision.
Research is a plausible way of ducking an issue. |
|
|
Term
| What do scientists and policy makers inherently disagree on? |
|
Definition
| The importance of time deadlines |
|
|
Term
| What criticisms about external validity do scientists tend to dismiss? |
|
Definition
|
|
Term
| What is almost entire focus for review of PhD students? |
|
Definition
| Interval validity with essentially no attention to external validity. |
|
|
Term
| What is done by "wildpersons?" |
|
Definition
| First investigations of an idea done with existing data and crude tools (i.e. with innovative external validity). |
|
|
Term
What are the results of "wildpersons"?
What do they spark? |
|
Definition
Only "suggestive" and almost certainly controversial.
More careful studies that scientists will view as conclusive. |
|
|
Term
What do most scientists not want to do that often leads to important discoveries?
Why?
What do some scientists prefer to work out? |
|
Definition
Take 1st steps that lead important discoveries.
Result is often not supported by later, more careful work.
Properties of new distributions. |
|
|
Term
| How do papers with large step toward external validity differ in evaluation from those with tiny step? |
|
Definition
| Exploratory work that seeks large step (i.e. wildpaper) does not depend as much on internal validity. |
|
|
Term
| What is often exchanged for greater external validity? |
|
Definition
|
|
Term
What is crazy, from scientific or policy viewpoint, about classifying how "satisfactory" studies are?
What should be done? |
|
Definition
Tendency to classify studies as "satisfactory" or not and then discarding all studies that are not satisfactory.
Look at all studies then assign some weight to the results. |
|
|
Term
It is only when users do what that collection methods improve?
What does this conclude? |
|
Definition
Constantly complain about data quality.
Without a client, the quality of the data will be low and generally worthless. |
|
|
Term
What is major challenge in designing study to test hypothesis?
What does it depend on? |
|
Definition
Sufficient sample size for desired confidence.
Reliability of study for accuracy and time required to observe. |
|
|
Term
What is second level of experimental design study?
What can be done for this? |
|
Definition
To ask whether a study can be done as initially planned?
A preliminary study. |
|
|
Term
| What's the 3 purposes of a large clinical trial? |
|
Definition
(1) resolve any final doubt
(2) convince the public
(3) estimate expected effects with greater precision |
|
|
Term
No study is ever what?
Why? |
|
Definition
Completely satisfactory.
Each fails either internal or external validity test, often both. |
|
|
Term
| What is needed to address the relevant question of "what inference can be drawn from a flawed study?" |
|
Definition
| Knowing what are good estimates of the parameters of interest and estimates of their variability. |
|
|
Term
| With inherent preference for internal validity, what bias do scientists have? |
|
Definition
| Smaller samples and more complete collection of data on each subject. |
|
|
Term
| What is disadvantage of study with smaller sample size? |
|
Definition
| Insufficient power to test the hypothesis |
|
|
Term
| How does a decrease in the expected frequency affect the required sample size? |
|
Definition
| Increases required sample size |
|
|
Term
| Within a study, what becomes more likely with a lower frequency of value? |
|
Definition
| that observed association is a spurious correlation (i.e. random occurrence) |
|
|
Term
What is assumed best test for potentially carcinogenic chemicals (i.e. assumed to be most accurate)?
What is result reminiscent of? |
|
Definition
rodent bioassay
Alice in Wonderland |
|
|
Term
What can produce estimates at a small fraction of the cost of doing an experiment?
What do these require? |
|
Definition
Focus on structure and clever usage of existing data.
Focus on causal modelling to find possible sources of bias. |
|
|
Term
| What new branch of stat does paper cry need for? This is gov decision makers greatest need from stat advisers. |
|
Definition
| Interpreting flawed studies |
|
|
Term
What is highest-priority need for adviser?
Why? |
|
Definition
Interpret existing studies (i.e. show what inferences can be drawn from a flawed study).
All studies have flaws. |
|
|
Term
| What is the meaning of interpreting a study? |
|
Definition
| Estimating its mean, median, or other central tendency measure and its qualitative and quantitative uncertainty |
|
|
Term
What occupation is of greatest need?
For what purpose? |
|
Definition
statistics adviser
bring informed, skeptical perspective to interpretations of the data (i.e. what data means and what are incorrect inferences) |
|
|
Term
| Discuss why is causality so difficult to prove? (What problems do you have proving it - and in proving anything?) (10) |
|
Definition
(1) Causality is difficult to prove because there are other effects that may be responsible for the changes.
(2) The studies usually establish a correlation but not causality
(3) there is no control as it is usually based on observations and not experimentation
(4) there may exist internal or external external validity problems
(5) it is difficult to ascertain proximity, precedence, and proportionality
(6) some of the model assumptions may affect the interpretation |
|
|
Term
| What are some rules or conditions we can apply to help confirm if we have causality or not? Identify three of these helpful conditions. (9) |
|
Definition
Proximity - the factors need to be related by theory or reason
Intertemporal precedence - if A causes B, A must come before B
Proportionality - a change in A should cause a similarly proportionate change in B |
|
|
Term
| What do we mean by random error (what is it)? (5) |
|
Definition
| This is any experimental error that occurs at random. It equals the difference between the estimated value and the true value which results from uncontrolled factors. |
|
|
Term
Does increasing the sampling rate always reduce random error? (1)
Why? (5)
What else could we do to reduce random eorr? (5) |
|
Definition
No
The measurement tool used will still have the same accuracy (e.g. unaccounted factors)
(1) change the experimental design
(2) increase the seensitivity and precision of the measurement test. |
|
|
Term
| What is systematic error? (5) |
|
Definition
| Error that consistently occurs due to flawed measurement tool or experimental design. It is consistent in either over- or underestimation. |
|
|
Term
| Explain why sensitivity does not reduce systematic error? (5) |
|
Definition
| the error is not random and consistent therefore by increasing sensitivitiy the problem is not dispelled because the tool is still the same. |
|
|
Term
| Under what conditions might you use a telephone network? (3) |
|
Definition
|
|
Term
Your assistant use the telephone network and phoned his colleagues to answer an important question. What 2 questions would you ask your assistant who did the telephoning to find out the interval validity of the telephone network process he used? (6)
How would this increase validity? (5)
How would this increase validity? (5) |
|
Definition
(1) Did they obtain a random sample of the experts? and how did they decide who to call?
(2) Did they maintain the same contions for each call (i.e. provide same introduction, questions, and time)?
By obtaining an exhaustive list and then randomly selecting from this, sample selection bias is reduced and experts from both sides have equal an opportunity to participate.
Ensures that respondents have the same opportunity to express their views. |
|
|
Term
| What do we gain when we use randomized experiments instead of non-randomized experiements? I.e. what sources of invalidity do we overcome when we randomize? (10) |
|
Definition
When random sampling, we gain a better reflection of our study's population. We gain an unbiased process.
We overcome invalidity from bias. |
|
|
Term
| What is the difference between internal and external validity? I.e. contrast them. (10) |
|
Definition
|
|
Term
| When you look at a study that claims to prove causality, what do you investigate about the study (what questions do you ask) to determine whether causality was actually proved? (10) |
|
Definition
(1) what experimental deisgn was used? was it done randomly?
(2) is the study reproducible under similar conditions? (
3) is there a theory undelrying the results?
(4) can the results be generalizable?
(5) Are there lurking variables?
(6) Is there a control group for comparison? |
|
|
Term
Define Granger Causality
What are its weaknesses? |
|
Definition
The Granger causality test is a technique for determining whether one time series is useful in forecasting another.
If both X and Y are driven by common 3rd process with different lags, one may accept the alternative hypothesis of Granger causality. Yet manipulation of one of the variables would not change the other. I.e. Granger Causality may produce misleading results when the true relationship involves three or more variables. |
|
|
Term
| What is internal validity? |
|
Definition
| Validity of whether the science we are using is good. It is also concerned with whether we have used the best approach in collecting data and estimating the parmeters. |
|
|
Term
| What is external validity? |
|
Definition
| This deals with whether the data/measurement provides insight into the actual problem at hand. |
|
|
Term
A valid measurement is one that measures what it claims (or purports) to measure. However, there my be errors in measurement.
In this case of random error, what might we do to reduce random error in a measurement? |
|
Definition
(1) Improve the ability of instrument to measure
(2) Incrase the sensitivity of the measuring device
(3) Use experimental design to reduce measuring error |
|
|
Term
I am interested in the effects of bankruptcy (financial collapse) on farmers. I obtain a list of farmers who went bankrupt in Indiana last year. From these farmers I obtain financial statements for both before and after bankruptcy. I analyze these before and after statements to determine the effects of bankruptcy.
What is the experimental design used in this study?
What improved experimental design could you have used that would be better?
How would the new design make your study better? |
|
Definition
|
|
Term
There are a number of major ways to test an hypothesis: a. Introspection, theory, simulation,
b. Anecdotes, case studies
c. Non-expermintal observations - e.g. surveys
d. Non-randomized experiments, and
e. Randomized experiments
What do we gain when we use randomized experiments instead of non-randomized experiments? I.e. what sources of invalidity do we overcome when we randomize an experiment? |
|
Definition
|
|
Term
Lave believes that the highest priority need is the interpretation of existing studies - in essence, discovering what inferences can be drawn from previous (maybe flawed) studies. How would you try to gain useful information (high external validity from an existing study?
What are the key characteristics of the existing study that you would have to examine? |
|
Definition
Key Characteristics
(1) method of data collection or measurement of variables
(2) sampling procedure and sampling methodology
(3) analytical approaches used
(4) the variables used and their interaction
(5) error sources and limitations of those studies so as to avoid them in your result
In order to gain useful information you have to eview vast majority of the literature that is relevant to what you are doing. This helps to know the gaps and other flaws and how you have to go about it. |
|
|
Term
If one needs to answer a question in a few days without time to do research, Lester Lave ("in "Does the Surgeon General Need a Statistical Advisor") suggests that an experienced person can do well answering a question by telephoning knowledgeable people.
What are two threats to validity from this telephone survey process; how might they be overcome or reduced? |
|
Definition
Selection bias; establish networks with the respondents and have their background characteristics so that you can randomly select who to interview. Also you have to select experts from both sides of the issue.
No control ove the interview wrt time, questions; introduction; (1) provide same background information to each respondent (2) ask same questions to all respondents (3) allow same time for each respondent to answer each questoin |
|
|
Term
What is false precision?
Why do we want to avoid false precision? |
|
Definition
This is when data is presented in a manner that implies better precision than is actually the case.
[] |
|
|
Term
| What do we mean when we say a measure is reliable? |
|
Definition
| It is one that yields consistent results when repeated under conditions taken to be constant. |
|
|
Term
| In measurement there is always ome random variation. if one wants to make a masure more reliable, one can always increase sensitivity. At what point of increasing sensitivity does it become useless to do so? |
|
Definition
| Top of Amir's head answer: when the magnitude of the economic benefit from improved measurement no longer exceeds the magnitude of its economic cost. |
|
|
