Introduction to Experimental Design

Motivation

Whether for just a summer or for the duration of an entire Ph.D project, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform quantitative experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use these attempts as wisely as if they were a limited number of wishes you have been granted. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

The essential starting point in all this is a very clear idea about the question to be answered. Without this initial investment of time and thought, it is easy to carry out an experiment which cannot answer the question either because the practical work itself has a flaw or because the results cannot be statistically analysed. This is perhaps the hardest lesson to learn.

–David Heath, An Introduction to Experimental Design and Statistics for Biology, 1995

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the realm of possible outcomes.
     • Example: microbial evolution experiments, in vitro selection.
   • Want to record as much descriptive information as possible, because important factors are unknown.
   • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
     • Example: reconstructing strains with combinations of known evolved alleles to determine which ones cause a phenotype
   • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.

• • Consist of different manipulative treatments that contrast the factor being tested for causality.
    • Treatments that correspond to baseline, known, natural states of the system labelled controls.
  • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Experimental error - unavoidable imprecision in measurements of the same sample.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days). Can be reduced by making measurements to be compared at the same time and with a randomized design in physical space.
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested. Can be reduced by increasing sample size.

Calculate the observations that you expect under the null hypothesis (H₀) that the factor is not important. Compare this probability to the alternative hypothesis (H₁) that the factor is explanatory. If the probability under the alternative hypothesis is greater (by a statistically significant margin) then reject the null hypothesis.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.
• Beware of unintended correlations in time. Measure everything that you want to compare at the same time (in one experimental block). If this is not possible (for example, if some assays fail and you want to redo those), you should include internal controls that were measured both times in order to detect and account for systematic biases between the two experimental blocks of measurements.
• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Making Measurements

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay? Can you achieve the expected result?
• How reliably is my assay working?
  • Does it give the same values from day to day?
• Am I measuring what I think I'm measuring?

Common Tricks

• If there is no way to manage a pure +/- treatment for the presence of some compound or factor, try spiking in a known, additional amount of what you are testing instead.

Further Information

• Lenski's Ten Commandments of Statistical Analysis
• Design of Experiments (Wikipedia)
• Handbook of Biological Statistics
• The Really Easy Statistics Site

Biblography

1. Heath, D. (1995) An Introduction to Experimental Design and Statistics for Biology. UCL Press, London.
2. Montgomery, D.C. (2001) Design and Analysis of Experiments. John Wiley & Sons, New York.

Introduction to Experimental Design

Motivation

Whether for just a summer or for the duration of an entire Ph.D project, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform quantitative experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use these attempts as wisely as if they were a limited number of wishes you have been granted. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the realm of possible outcomes.
     • Example: microbial evolution experiments, in vitro selection.
   • Want to record as much descriptive information as possible, because important factors are unknown.
   • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
     • Example: reconstructing strains with combinations of known evolved alleles to determine which ones cause a phenotype
   • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.
   • Research hypothesis generation - Brainstorm* – Make a list of as many explanations as you can.
   • Consist of different manipulative treatments that contrast the factor being tested for causality.
     • Treatments that correspond to baseline, known, natural states of the system labelled controls.
   • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Experimental error - unavoidable imprecision in measurements of the same sample.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days). Can be reduced by making measurements to be compared at the same time and with a randomized design in physical space.
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested. Can be reduced by increasing sample size.

Calculate the observations that you expect under the null hypothesis (H₀) that the factor is not important. Compare this probability to the alternative hypothesis (H₁) that the factor is explanatory. If the probability under the alternative hypothesis is greater (by a statistically significant margin) then reject the null hypothesis.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.
• Beware of unintended correlations in time. Measure everything that you want to compare at the same time (in one experimental block). If this is not possible (for example, if some assays fail and you want to redo those), you should include internal controls that were measured both times in order to detect and account for systematic biases between the two experimental blocks of measurements.
• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Making Measurements

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay? Can you achieve the expected result?
• How reliably is my assay working?
  • Does it give the same values from day to day?
• Am I measuring what I think I'm measuring?

Common Tricks

• If there is no way to manage a pure +/- treatment for the presence of some compound or factor, try spiking in a known, additional amount of what you are testing instead.

Further Information

• Design of Experiments (Wikipedia)
• Handbook of Biological Statistics
• The Really Easy Statistics Site

Biblography

1. Heath, D. (1995) An Introduction to Experimental Design and Statistics for Biology. UCL Press, London.
2. Montgomery, D.C. (2001) Design and Analysis of Experiments. John Wiley & Sons, New York.

Introduction to Experimental Design

Motivation

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"

• • • Example: microbial evolution experiments, in vitro selection.
  • Want to record as much descriptive information as possible, because important factors are unknown.
  • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!

1. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.

• • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.
  • Research hypothesis generation - Brainstorm* – Make a list of as many explanations as you can.
  • Consist of different manipulative treatments that contrast the factor being tested for causality.
    • Treatments that correspond to baseline, known, natural states of the system labelled controls.
  • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.
• Beware of unintended correlations in time. Measure everything that you want to compare at the same time (in one experimental block). If this is not possible (for example, if some assays fail and you want to redo those), you should include internal controls that were measured both times in order to detect and account for systematic biases between the two experimental blocks of measurements.
• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Making Measurements

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay? Can you achieve the expected result?
• How reliably is my assay working?
  • Does it give the same values from day to day?
• Am I measuring what I think I'm measuring?

Common Tricks

Further Information

• Handbook of Biological Statistics
• The Really Easy Statistics Site

1. Heath, D. (1995) An Introduction to Experimental Design and Statistics for Biology. UCL Press, London.
2. Montgomery, D.C. (2001) Design and Analysis of Experiments. John Wiley & Sons, New York.

Introduction to Experimental Design

Motivation

Whether for just a summer or for an entire Ph.D, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use them wisely. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

"The essential starting point in all this is a very clear idea about the question to be answered. Without this initial investment of time and thought, it is easy to carry out an experiment which cannot answer the question either because the practical work itself has a flaw or because the results cannot be statistically analysed. This is perhaps the hardest lesson to learn." [1]

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the ream of possible outcomes.
     • Example: microbial evolution experiments, in vitro selection.
   • Want to record as much descriptive information as possible, because important factors are unknown.
   • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
   • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.
   • Research hypothesis generation - Brainstorm* – Make a list of as many explanations as you can.
   • Consist of different manipulative treatments that contrast the factor being tested for causality.
     • Treatments that correspond to baseline, known, natural states of the system labelled controls.
   • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Experimental error - unavoidable imprecision in measurements of the same sample. Can be reduced by increasing sample size.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days).
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested.

Calculate the observations that you expect under the null hypothesis (H₀) that the factor is not important. Compare this probability to the alternative hypothesis (H₁) that the factor is explanatory. If the probability under the alternative hypothesis is greater (by a statistically significant margin) then reject the null hypothesis.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.
–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.
• Beware of unintended correlations in time. Measure everything that you want to compare at the same time (in one experimental block). If this is not possible (for example, if some assays fail and you want to redo those), you should include internal controls that were measured both times in order to detect and account for systematic biases between the two experimental blocks of measurements.
• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Making Measurements

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay? Can you achieve the expected result?
• How reliably is my assay working?
  • Does it give the same values from day to day?
• Am I measuring what I think I'm measuring?

Common Tricks

• If you can not have a pure +/- treatment for the presence of some compound, try spiking in a known amount of what you are testing.

Further Information

• http://en.wikipedia.org/wiki/Design_of_experiments

References

1. Heath, D. (1995) An Introduction to Experimental Design and Statistics for Biology. UCL Press, London.
2. Montgomery, D.C. (2001) Design and Analysis of Experiments. John Wiley & Sons, New York.

Internal page with more information...

Introduction to Experimental Design

Motivation

Whether for just a summer or for an entire Ph.D, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use them wisely. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

"The essential starting point in all this is a very clear idea about the question to be answered. Without this initial investment of time and thought, it is easy to carry out an experiment which cannot answer the question either because the practical work itself has a flaw or because the results cannot be statistically analysed. This is perhaps the hardest lesson to learn." [1]

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the ream of possible outcomes.
     • Example: microbial evolution experiments, in vitro selection.
   • Want to record as much descriptive information as possible, because important factors are unknown.
   • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
   • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.
   • Research hypothesis generation - Brainstorm* – Make a list of as many explanations as you can.
   • Consist of different manipulative treatments that contrast the factor being tested for causality.
     • Treatments that correspond to baseline, known, natural states of the system labelled controls.
   • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Experimental error - unavoidable imprecision in measurements of the same sample. Can be reduced by increasing sample size.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days).
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested.

Calculate the observations that you expect under the null hypothesis (H₀) that the factor is not important. Compare this probability to the alternative hypothesis (H₁) that the factor is explanatory. If the probability under the alternative hypothesis is greater (by a statistically significant margin) then reject the null hypothesis.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.
–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.
• Beware of unintended correlations in time. Measure everything that you want to compare at the same time (in one experimental block). If this is not possible (for example, if some assays fail and you want to redo those), you should include internal controls that were measured both times in order to detect and account for systematic biases between the two experimental blocks of measurements.
• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Making Measurements

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay? Can you achieve the expected result?
• How reliably is my assay working?
  • Does it give the same values from day to day?
• Am I measuring what I think I'm measuring?

Common Tricks

Further Information

• http://en.wikipedia.org/wiki/Design_of_experiments

References

Introduction to Experimental Design

Motivation

Whether for just a summer or for an entire Ph.D, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use them wisely. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

"The essential starting point in all this is a very clear idea about the question to be answered. Without this initial investment of time and thought, it is easy to carry out an experiment which cannot answer the question either because the practical work itself has a flaw or because the results cannot be statistically analysed. This is perhaps the hardest lesson to learn." [1]

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the ream of possible outcomes.
     • Example: microbial evolution experiments, in vitro selection.
   • Want to record as much descriptive information as possible, because important factors are unknown.
   • Limitation: Cannot prove that a factor correlated with a given outcome causes that outcome!
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
   • Based on prior knowledge, you can enumerate or mathematically model the possible outcomes.
   • Research hypothesis generation - Brainstorm* – Make a list of as many explanations as you can.
   • Consist of different manipulative treatments that contrast the factor being tested for causality.
     • Treatments that correspond to baseline, known, natural states of the system labelled controls.
   • Limitations: May oversimplify the problem when there are interactions between factors and must be conducted under unnatural, altered, or laboratory conditions that make extrapolating to nature difficult.

Statistical Design of Experiments

Before conducting an experiment, you should already know what you are going to measure to be sure that you will be able to detect the salient differences. A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population in an experiment. For example, the red/white colonies plated on agar to measure the fraction of a population that are Ara– and Ara+ are a sample of a few hundred cells from a population of millions of cells.

There will always be variation in measurements due to:

• Experimental error - unavoidable imprecision in measurements of the same sample. Can be reduced by increasing sample size.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days).
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested.

Calculate the observations that you expect under the null hypothesis (H₀) that the factor is not important. Compare this probability to the alternative hypothesis (H₁) that the factor is explanatory. If the probability under the alternative hypothesis is greater (by a statistically significant margin) then reject the null hypothesis.

Before starting an experiment you should perform a statistical power analysis to be sure that you will be able to detect significant differences with your experimental error, number of replicates, etc.

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.
–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

Other important considerations:

• Keep the statistical analysis as simple as possible. The more complex your analysis, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and the more likely you are to misinterpret the results.
• Test one factor at a time. If possible change only one variable between treatments at a time.
• Beware of pseudo-replication. Be sure that your replicates are entirely statistically independent. Three measurements of the same sample have a very different type and amount of error than three measurements of different samples.

• Fully randomize your design. To account for variation in time and space, use a random number generator to make the placement and order of samples random with respect to your treatments. If this is too cumbersome, alternate systematically between samples in the different treatments. Examples of issues include, putting all flasks for one treatment on one side of the incubator or processing them first before other samples.

Further Information

• http://en.wikipedia.org/wiki/Design_of_experiments

References

1. Heath. Introduction to Experimental Design. Chapter 1: Why biologists need sampling, experimental design, and statistics.
2. Montgomery. Design and Analysis of Experiments

Internal page with more information...

Introduction to Experimental Design

Motivation

Whether for just a summer or for an entire Ph.D, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use them wisely. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the ream of possible outcomes.

1. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay?
• Am I measuring what I think I'm measuring?

• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days).
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested.

Further Information

• http://en.wikipedia.org/wiki/Design_of_experiments

References

1. Heath. Introduction to Experimental Design. Chapter 1: Why biologists need sampling, experimental design, and statistics.
2. Montgomery. Design and Analysis of Experiments

Internal page with more information...

Introduction to Experimental Design

Motivation

Whether for just a summer or for an entire Ph.D, working on a scientific problem is a process of trying to explain and predict the way that the natural world operates. You will typically need to use or develop new ways of measuring something about the world or perform experiments to test a hypothesis. Because the amount of time that you have to reach an answer is finite, the whole process is a bit like a game of twenty questions. There are only so many experiments that you can do before the end of summer or before it's time to graduate. Use them wisely. The more effectively that you learn how to pose the questions (and debug protocols) the more progress you will make toward scientific discovery. This page is meant to provide vocabulary, ideas, and questions related to experimental design.

Types of Experiments

1. Observational Experiments
   • Description – "What is found?"
   • Appropriate when you have no prior information about the ream of possible outcomes.
   • Example: microbial evolution experiments, in vitro selection.
2. Manipulative Experiments
   • Explanation – "Why and how is it found?"
   • Appropriate when you are trying to determine the factors which cause a certain outcome.
   • Use a mathematical model or statistical framework to enumerate all of the possible expected outcomes.

Controls

• Is my assay working?
  • Sanity check: Are the values I'm getting realistic? Do they agree with what is known from the literature?
  • Positive control: Do you have a strain or sample that should give an expected result in the assay?
• Am I measuring what I think I'm measuring?

Statistical Design of Experiments

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.
–Ronald Fisher, Presidential Address to the First Indian Statistical Congress, 1938.

A sample is the subset of a population that is examined. Generally one uses measurements of the sample to infer the properties of the population.

There will always be variation in measuring a characteristic of the natural world due to:

• Experimental error - unavoidable imprecision in measurements of the same sample.
• Biological variation - differences between individuals (cultures, strains) making up the sample which are nominally the same.
• Variation in space and time - variation in measurements due to unplanned changes in environmental conditions (Ex: performing measurements on different days).
• Sampling error - uncertainty in making inferences about the population because of chance variation in the makeup of the sample that was actually tested.

• Keep the statistical analysis as simple as possible. The more complex it is, the more difficult it will be to explain to others exactly what you did, the more likely you might make mistakes in calculations, and.
• Test one factor at a time. If possible change only one variable at a time.

• Replication
• Randomization
  • Examples: You always put treatment 1 flasks on the left side of the incubator and plate them first. You find that these
  • Solution: When in doubt, use a fully randomized treatment.

Further Information

• http://en.wikipedia.org/wiki/Design_of_experiments

References

1. Heath. Introduction to Experimental Design. Chapter 1: Why biologists need sampling, experimental design, and statistics.
2. Montgomery. Design and Analysis of Experiments

Internal page with more information...