29 Mar, 2016

Resources


Notes

Homework

Midterms


Some Terms

Linear models
$\Rightarrow$ linear in “parameters”. That is
\[y = f(X)\beta + \epsilon\]

In linear models, least Squares = BLUE = MLE, for normal errors.

BLUE
Best Linear Unbiased Estimator. $\hat\beta = (X’X)^-X’y$ are linear in $y$.
ANCOVA
Analysis of Covariance. Mix of Regressions and ANOVA.

R: options(contrasts=c("contr.sum","contr.poly"))

contr.sum = the constraint that $\sum \alpha_i = 0$

Default in R for lm(y~x) is to set the first level parameter $\alpha_i= 0$

lm(y ~ x - 1) => no intercept

Estimate of $y$ is the same, though estimate of $\hat\alpha$ may be different.

LSE
$\hat\beta = Q(\beta) = \underset{\beta}{\text{argmin}} (y-X\beta)^T(y-X\beta)$
  • find the gradient
  • set the gradient = 0

Derivatives of Vectors

  • $\frac{\partial b^Ta}{\partial b} = \frac{\partial a^Tb}{\partial b} = a$
  • $\frac{\partial b^TAb}{\partial b} = (A+A^T)b$

Review: Vector Space, Subspace

Teach Span and collumn space using simple 2 by 2 examples.

Span
The set of all linear combinations of $x_1,…,x_r \in S$ is called the space spanned by $x_1,…,x_r$. If $M$ is a subspace of $S$ and $M$ equals the space spanned by $x_1,…,x_r$ then $\bc{x_1,…,x_r}$ is called a spanning set for $M$.
column space
In terms of $y=X\beta$. The column space of $X$ is the set of $y$’s such that there exists a $\beta$ such that $y=X\beta$.

See lecture 2 notes on p. 21 / 64 for the application.

Linearly Dependent
Let $x_1,…,x_r$ be vectors in $S$. If there exists scalars $\alpha_1,…,\alpha_r$ not all 0, such that $\sum \alpha_i =0_r$ then $x_1,…,x_r$ are linearly dependent. Otherwise, linearly independent.
Basis
If $M$ is a subspace of $S$ and if $\bc{x_1,…,x_r}\in S$ is a linearly independent spanning set for $M$, then $\bc{x_1,…,x_r}$ is called a basis for $M$. (check)

Rank:

Singular: