Stat 263/363: Experimental Design

Outline and syllabus Course notes.
For Stanford people: the canvas page will have the HWs and more references.

Overview

Correlation is not causality. You've probably heard that before in any number of regression classes. If you want to infer causality from data, then the best way is to use randomized experiments. Maybe it is the only way to be sure.
In experimental design we look at how to choose the data that we will gather. In addition to being able to make causal conclusions, we also look at how to maximize the statistical efficiency of the generated data set.
Experimental design as a subject is about 100 years old. The methods in this course date back to agricultural field trials. Since then the ideas have seen use in medicine, manufacturing, quality control, computer aided design and electronic commerce. Each new field takes the previous methods and then starts adapting them. Possibly the first clinical trial was that of James Lind in 1747 showing that citrus is effective against scurvy. (It was not immediately adopted.)

There will be some problem sets, a midterm on Thursday October 22 and a project. The project will involve designing, carrying out and analyzing a real experiment. This can be from your every day life: cooking, hobbies, exercise routines, etc. There are ordinarily about 4 problem sets. As I announced in class, due to the pandemic it seems better to have a larger number of smaller problem sets.

This Mark Rober video (might serve an ad) describes an experiment to study which animals (snake vs turtle vs tarantula) are more likely to be run over by vehicles. The results are interesting. It is also funny.

Goals

Learn the main/classical methods of experimental design so that when it comes time to gather data you can work out the right choice.

See some of the research frontier in DoE: A/B testing, computer experiments, design for high dimensional regression.

Do a designed statistical experiment from conception to execution to analysis.

Topics

see page 2 of the course announcement. I'm expecting and hoping for two guest lectures to displace two of the post-midterm topics.

Instructor scribed lecture notes from Autumn 2020/21

Here is the full set of notes. Any updates to these notes will be in this full set, between now and the next time the class is taught. The scribed chapters below may not be as up to date.

Sep 15. Introduction History of design. Potential outcomes.
Sep 17. A/B testing Applications to web companies.
Sep 22. Bandits Especially Thompson sampling.
Sep 24. Pairing and blocking Prior Stat 305A ANOVA notes One way analysis.
Sep 29. ANOVA Prior MC notes ANOVA Includes functional ANOVA.
Oct 01. \(2^k\) factorials Motivations and notation.
Oct 06. \(2^{k-p}_R\) fractional factorials Aliasing and data analysis.
Oct 08. ANCOVA and crossovers Before after comparisons.
Oct 13. Split-plots and nesting Also cluster randomized trials.
Oct 15. Taguchi methods Robust design.
Oct 20. Catchup review And DOE analyses.
Oct 22. No class. Midterm.
Oct 27. Response surfaces And optimal design.
Oct 29. Supersaturated designs Hadamard and random balance.
Nov 3,5. Computer experiments Design and analysis.
Nov 10,12,17. Guest lectures and hybrids Networks|Complex clinical trials|Partially randomized data.
Nov 19. Overview And final comments.

Classes

Online Tue & Thu 2:30 to 3:50

Lectures at PhD level, homework at MS level.

Grading basis

3 units and letter grade or CR/NC.

Instructor

Art Owen

Sequoia Hall 130

My userid is owen at the address stanford.edu

Office hour: Weds 1:30 - 2:30

TAs

On zoom Hours in canvas

Texts

Some links below. More may be in canvas.

Evaluation

HW 50%. Midterm 25%. Final project 25%.

Supplementary readings

These articles should be readable for Stanford users.

Stefan Wager's causal inference notes (especially lecture 1) and Paul Holland's article on Neyman-Rubin causality.
The Lanarkshire milk experiment Less nourished children were more likely to be in the treatment groups. (And other issues.) Student gives a careful analysis.
Kohavi, Tang and Xu's Trustworthy online controlled experiments book. May also be online in SU library.
Kohavi, Henne and Sommerfield's article on A/B tests for e-commerce.
Agrawal and Goyal's article on Thompson sampling
Bubeck and Cesa-Bianchi's monograph on bandit problems.
Wikipedia entry on Hadamard matrices for Plackett-Burmann designs
S. Georgiou's review of supersaturated designs.
Krahmen and Ward's review of design for compressed sensing
Dean Eckles' thesis including a network experiment using Facebook data

More details

Be sure to give Axess a working email address:

I expect to send a small number of important emails about problem sets and the homework there. Most other announcements will be made in class. If you email me about the class, be sure to have stat 363 or stat 263 in your subject line. Otherwise, your email won't show when I search for course related emails.

Late penalties apply:

We will count days late on each problem set. Each day late is penalized by 10% of the homework value. Homework more than 3 days late will ordinarily get 0. Upload to gradescope within canvas. For sickness, interviews and other events, up to 3 late days (4 in pandemic years) total are forgiven at the end of the quarter. (Work late enough to get zero does not get redeemed though.)