5 Donor’s Voice experiments: description
Collaborators on the experiment planning: David Reinstein, Kiki Koutmeridou; also, to a lesser extent, Elizabeth Keenan, Ayelet Gneezy, Hengchen Dai, Enrico Rubaltelli, Stephan Dickert, and Peter Ayton
5.1 Plans from prereg OSF—link:
What is the impact of including ‘information about the per-dollar impact’ of a charity (in terms of services provided) on the average donation (equivalently, total amount raised) and the donation incidence rate?
We are running this subject to the final say of the charity. We have proposed that the Treatment emails (but not the control emails) will include a sentence/fragment such as the following in both a captioned photo in the email, and the email text:
“Last year, we were able to provide [general provision of an outcome here relevant to the charity] to a [recipient unit] with just $[small amount of money].”
- Average gift amount (including zeroes)
- Incidence/number of people making a gift, [and]
- Incidence of gifts of exactly $10.
In particular, we will focus on Fisher’s exact test (for incidence) and the standard rank sum and t-tests for the donation amounts.
Power calculations
Response rates in previous such emails were extremely low: approximately 1 per 3,000 emails. Our power calculations suggest that we have .29 power to detect a 50% effect, and 0.90 power to detect approximately a 100% (doubling) on incidence…
Because of this limited power, we will ask the charity to run this trial a second time with an equivalent-sized sample.
Stopping rule
We aim to continue this treatment in future charity appeals until we can statistically bound (with 95% confidence) the impact of the treatment on both incidence and average donation within a margin of 1/3 of the incidence and average donation in the control condition.
5.2 Context
Charity: A large US religiously-associated international poverty relief charity
Timing: Emails sent out at the same time withing each trial
First trial: 21-Nov-2018 ‘Thanksgiving email’ Second trial: Nov 2019 (also Thanksgiving email)
Sample size and composition:
First trial:
- Charity’s standard email list (previous donors with emails)
- Approx 182,600 emails sent out, 91.3k in each condition
Second trial: - 79,754 emails sent out, exactly half in each condition
5.4 Setup (second trial)
(Very similar to first trial, but ‘impact information content differed’)
Control:
Treatment text (otherwise as in control):
5.5 Treatment assignment/randomization
From our OSF wiki and OSF prereg
The email will randomly (without blocking/stratification) vary whether or not these emails present a statement about the charity was ‘able to provide with just $10’. …
The charity fundraising partner has informed us that they expect to send out roughly 330,000 emails