Mock Midterm Exam — Solutions

We know \(P(\text{AID} \mid \text{KM}) = 0.28\) and \(P(\text{KM}) = 0.35\).

By the definition of conditional probability:

\[P(\text{KM} \cap \text{AID}) = P(\text{AID} \mid \text{KM}) \cdot P(\text{KM}) = 0.28 \times 0.35 = 0.098\]

Using Bayes' theorem:

\[P(\text{KM} \mid \text{AID}) = \frac{P(\text{AID} \mid \text{KM}) \cdot P(\text{KM})}{P(\text{AID})} = \frac{0.28 \times 0.35}{0.40} = \frac{0.098}{0.40} = 0.245\]

Since Customer B is in the same domain as Customer A, we can use the Customer A results as an informative prior. The 85/100 performance on Customer A translates to a prior:

\[\theta \sim \text{Beta}(85, 15)\]

Now we update with Customer B's data (6 successes, 4 failures):

\[\theta \mid \text{data}_B \sim \text{Beta}(85 + 6,\; 15 + 4) = \text{Beta}(91, 19)\]

The posterior mean is:

\[\mathbb{E}[\theta] = \frac{91}{91 + 19} = \frac{91}{110} \approx 0.827\]

So rather than reporting the naive 6/10 = 60% from Customer B's small test alone, we report a posterior estimate of about 82.7%. The strong prior from the same domain pulls the estimate up, reflecting that 10 data points are too few to override what we already know about the model's performance in online game sales.

We should also report uncertainty. A 95% credible interval is the interval \([L, U]\) such that 95% of the posterior probability lies between \(L\) and \(U\). For a \(\text{Beta}(\alpha, \beta)\) posterior, this is computed as the interval between the 0.025 and 0.975 quantiles of that distribution — meaning we believe with 95% posterior probability that the true parameter falls within this range.

The 95% credible interval for \(\text{Beta}(91, 19)\) is approximately [0.75, 0.89].

If Customer B is in a different domain (music vs games), we should not use the Customer A results as a strong prior, since the model's accuracy in game sales may not transfer to music sales.

Instead, we should use a weakly informative or uninformative prior, for example \(\text{Beta}(1, 1)\) (uniform):

\[\theta \sim \text{Beta}(1, 1)\]

Updating with Customer B's data:

\[\theta \mid \text{data}_B \sim \text{Beta}(1 + 6,\; 1 + 4) = \text{Beta}(7, 5)\]

The posterior mean is:

\[\mathbb{E}[\theta] = \frac{7}{12} \approx 0.583\]

Now the estimate (≈ 58.3%) is much lower and much more uncertain. The 95% credible interval for \(\text{Beta}(7, 5)\) is approximately [0.31, 0.83] — very wide, reflecting that we have only 10 observations and no relevant prior information.

Key insight: The choice of prior matters. Using domain knowledge appropriately (same domain → informative prior, different domain → weak prior) is a critical modeling decision that directly affects the accuracy estimate and its uncertainty.

Both teams have roughly equally reliable test estimates — Team A's are slightly more reliable (200 test examples vs 180).

The reliability of an accuracy estimate depends on the absolute size of the test set, not the total dataset size. The standard error of an estimated proportion \(\hat{p}\) is:

\[\text{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{n_{\text{test}}}}\]

Only \(n_{\text{test}}\) appears in this formula — nothing about the training set or the total dataset. Team A tests on 200 examples and Team B on 180, so their standard errors are very similar.

With a \(\text{Beta}(1,1)\) prior (uniform) and Binomial data, the posterior is:

\[\theta \mid \text{data} \sim \text{Beta}(1 + k,\; 1 + n - k)\]

Team A (120 correct out of 200):

\[\theta_A \sim \text{Beta}(121, 81)\]

Team B (175 correct out of 180):

\[\theta_B \sim \text{Beta}(176, 6)\]

The variance of a \(\text{Beta}(\alpha, \beta)\) distribution is:

\[\text{Var} = \frac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\]

Team A:

\[\text{Var}_A = \frac{121 \times 81}{202^2 \times 203} = \frac{9801}{8{,}284{,}812} \approx 0.001183\]

Team B:

\[\text{Var}_B = \frac{176 \times 6}{182^2 \times 183} = \frac{1056}{6{,}064{,}668} \approx 0.000174\]

Team A's posterior has the wider 95% credible interval.

The Beta distribution's variance depends on how far the posterior mean is from 0.5. Team A's posterior mean is \(\frac{121}{202} \approx 0.60\), close to the high-uncertainty zone. Team B's posterior mean is \(\frac{176}{182} \approx 0.967\), far toward 1, where the distribution is tightly concentrated. So even though both teams have similar test set sizes, Team A's estimate of accuracy is considerably less precise.

(a) We compute \(E[V]\) directly:

\[E[V] = (-5)(0.10) + 0(0.20) + 5(0.35) + 10(0.30) + 20(0.05) = -0.50 + 0 + 1.75 + 3.00 + 1.00 = 5.25\]

(b) Method 1 — from the definition: \(\text{Var}(V) = \sum_i p_i (v_i - \mu)^2\)

\[\text{Var}(V) = 0.10(-5 - 5.25)^2 + 0.20(0 - 5.25)^2 + 0.35(5 - 5.25)^2 + 0.30(10 - 5.25)^2 + 0.05(20 - 5.25)^2\]

\[= 10.5063 + 5.5125 + 0.0219 + 6.7688 + 10.8781 = 33.6875\]

Method 2 — via \(\text{Var}(V) = E[V^2] - (E[V])^2\):

\[E[V^2] = (-5)^2(0.10) + 0^2(0.20) + 5^2(0.35) + 10^2(0.30) + 20^2(0.05) = 2.50 + 0 + 8.75 + 30.00 + 20.00 = 61.25\]

\[\text{Var}(V) = E[V^2] - (E[V])^2 = 61.25 - (5.25)^2 = 61.25 - 27.5625 = 33.6875\]

(c) Let \(S = V_1 + V_2 + \cdots + V_{50}\). Since the runs are IID with \(n = 50\), by the CLT:

\[S \approx \mathcal{N}(\mu_S,\, \sigma^2_S) \quad \text{where} \quad \mu_S = 50 \times 5.25 = 262.50,\quad \sigma^2_S = 50 \times 33.6875 = 1684.375\]

So \(S \approx \mathcal{N}(262.50,\; 1684.375)\).

(d) Since \(S \sim \mathcal{N}(262.50,\; 1684.375)\), we can directly compute:

\[P(S \geq 350) = 1 - P(S < 350) \approx 0.0165\]

Assumption: \(X\) and \(Y\) are independent.

A linear combination of independent normal random variables is also normal. We compute the mean and variance:

\[E[2X + Y] = 2E[X] + E[Y] = 2(1) + 1 = 3\]

\[\text{Var}(2X + Y) = 4\cdot\text{Var}(X) + \text{Var}(Y) = 4(2) + 2 = 10\]

Therefore: \(2X + Y \sim \mathcal{N}(3,\; 10)\).

"At least 50 runs are needed" means that after 49 runs the cumulative value has not yet exceeded 200, i.e. \(S_{49} \leq 200\).

From Exercise 4, each run has:

\[E[V] = 5.25 \qquad \text{Var}(V) = 33.6875\]

Let \(S_{49} = V_1 + \cdots + V_{49}\). By the CLT:

\[E[S_{49}] = 49 \times 5.25 = 257.25 \qquad \text{Var}(S_{49}) = 49 \times 33.6875 = 1650.6875\]

Since \(S_{49} \sim \mathcal{N}(257.25,\; 1650.6875)\), we can directly compute:

\[P(S_{49} \leq 200) \approx 0.0794\]

There is approximately a 7.9% chance that at least 50 runs are needed. This is quite low because the expected cumulative value after 49 runs (257.25) is well above the 200 target.

(a) Let \(S_A = A_1 + \cdots + A_{10}\). By the CLT:

\[E[S_A] = 10 \times 20 = 200 \qquad \text{Var}(S_A) = 10 \times 5 = 50\]

So \(S_A \sim \mathcal{N}(200,\; 50)\), and we directly compute \(P(S_A < 200) = 0.5\).

This makes intuitive sense: the budget is exactly at the mean, so by symmetry of the normal distribution, there is a 50% chance of staying under budget.

(b) Let \(S_B = B_1 + \cdots + B_{10}\). By the CLT:

\[E[S_B] = 10 \times 22 = 220 \qquad \text{Var}(S_B) = 10 \times 10 = 100\]

So \(S_B \sim \mathcal{N}(220,\; 100)\), and we directly compute \(P(S_B < 200) \approx 0.0228\).

(c) Consider \(D = S_A - S_B\). Assuming independence between models:

\[E[D] = 200 - 220 = -20 \qquad \text{Var}(D) = 50 + 100 = 150\]

So \(D \sim \mathcal{N}(-20,\; 150)\), and we directly compute \(P(D < 0) \approx 0.9488\).

There is approximately a 94.9% chance that Model A costs less than Model B. Although Model A is cheaper on average (20 vs 22 cents per run), the high probability also reflects Model A's lower variance — it is both cheaper and more predictable.