Field experiments on labour market discrimination: do sizes of randomly assigned groups matter? In traditional field experiments on racial discrimination, two identical resumes differing only by name or ID photo are sent to randomly selected firms. A statistical test is then run on the number of interview offers received by each resume to determine if any difference in the number of interview offers received between the resumes holds true in the population.
I am curious whether, if in the event that each resume is sent to a different number of companies (e.g. one is sent to 300 firms and the other 310), results that show a statistically significant difference in the number of interview offers received on one resume compared to the other will be invalidated. My intuition is that there may be a tendency for bias to be introduced as the resume that is assigned to a greater number of firms has a higher chance of being selected. Would appreciate comments on whether this is accurate.
No, this should not be a major issue. We will very carefully take into account the different sample sizes. Allow me to continue your example:
Assume &quot;Resume A&quot; is the the treatment resume and &quot;Resume B&quot; is the control resume, where the treatment has a less advantaged ethnic name (Jamal, Beyonce) and the control resume contains a anglicized name (James, Sophia). One should be careful about gender here as well.
We then compare the mean acceptance rate between the two resumes.
Suppose $N_a = 300$ and $N_b = 310$.
The accepted resumes are: $Accepted_a = 30$ and $Accepted_b = 62$.
For shortcuts, the mean rates of acceptance are $r_a = 30/300 = 0.1$ and $r_b = 62/310 = 0.2$. Notice the mean rate of acceptance normalizes each group, so size is partially accounted for here.
The variances are then:
$\sigma_a^2 = (1-r_a)*r_a = 0.090$
$\sigma_b^2 = (1-r_b)*r_b = 0.160$
And more importantly, the variance of the means of each distribution are then $\sigma_a^2/N_a$ and $\sigma_b^2/N_b$, respectively.  We are comparing the means of the two distributions with a simple mean comparison test. Notice this step explicitly uses the sample size to account for expected variation between the two distributions.
So we now compare the two to see if there is a difference, under the null hypotheses there is no difference. (In statistics, you nearly always always assume no difference/no relevance/no effect as the baseline.)
$t=\frac{r_a-r_b}{\sqrt{\sigma_a^2/N_a+\sigma_b^2/N_b}} = 3.572$
The reasoning can be very explicitly laid out as follows:

The difference between the two samples is 3.572 standard deviations
away from 0.
Under the assumptions we have made above, it seems very
unlikely that this would occur by chance if we repeated the process
(looking this up on the t-table as a one-tailed test has a p-value of less than 0.001). Bayesians will point out there are some other silent assumptions one may have made, so language is particularly important at this step, but this generally passes the intuition.
Therefore, I reject the idea that these two samples came from the same distribution. It seems implausible.
So, I am left to conclude that there is a difference between the two samples. It seems that indeed people are more accepting of the Anglicized names rather than the ethnic names.