Just as orthogonal groups preserve the magnitude of real vectors, the unitary groups preserve the magnitude of complex vectors. The “rotation” group in two complex dimensions is known as $\SU(2)$. If $v\in\CC^2$, its magnitude is defined by \begin{equation} |v|^2 = v^\dagger v \end{equation} where $\dagger$ denotes the conjugate transpose, that is \begin{equation} v^\dagger = \bar{v}^T . \end{equation} The magnitude of a complex vector is still a non-negative real number, just as for real vectors.

So if we want to preserve the magnitude of $v$, we must have \begin{equation} (Mv)^\dagger (Mv) = (v^\dagger M^\dagger)(M v) = v^\dagger v \end{equation} or simply \begin{equation} M^\dagger M = 1 . \label{unitary} \end{equation} Matrices satisfying (\ref{unitary}) are called unitary matrices; the “U” in $\SU(2)$ stands for unitary. As before, the “S” stands for special, and refers to the additional condition that \begin{equation} |M| = 1 . \label{special} \end{equation} Thus, we have \begin{equation} \SU(2) = \{M\in\CC^{2\times2}:M^\dagger M=1, |M|=1\} . \end{equation}

It's not that difficult to find the most general matrix satisfying (\ref{unitary}) and (\ref{special}), but let's first ask how many degrees of freedom there are. A $2\times2$ complex matrix has 8 real degrees of freedom, the matrix equation (\ref{unitary}) imposes 4 constraints, and the determinant condition (\ref{special}) imposes one more constraint. Thus, we expect 3 parameters in our general element—just as for $\SO(3)$.

Let's start with some examples. Our old friend $M(\phi)\in\SO(2)$ is in $\SU(2)$, but we give it a new name, writing \begin{equation} S_y(\alpha) = \begin{pmatrix} \cos\alpha & -\sin\alpha \\ \sin\alpha & \cos\alpha \end{pmatrix} . \end{equation} Other examples of unitary matrices are \begin{equation} S_x(\alpha) = \begin{pmatrix} \cos\alpha & -i\sin\alpha \\ -i\sin\alpha & \cos\alpha \end{pmatrix} , \qquad S_z(\alpha) = \begin{pmatrix} e^{-i\alpha} & 0 \\ 0 & e^{i\alpha} \end{pmatrix} . \end{equation}

We know what to do next: Differentiate! So consider \begin{align} s_y &= \dot{S}_y = S_y'(0) = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix} ,\\ s_x &= \dot{S}_x = S_x'(0) = \begin{pmatrix} 0 & -i \\ -i & 0 \end{pmatrix} ,\\ \quad s_z &= \dot{S}_z = S_z'(0) = \begin{pmatrix} -i & 0 \\ 0 & i \end{pmatrix} . \end{align} These matrices are linearly independent, and therefore span the tangent space at the identity.

So what? Well, let's compute the commutators. The result is \begin{equation} [s_x,s_y] = 2s_z , \qquad [s_y,s_z] = 2s_x , \qquad [s_z,s_x] = 2s_y . \end{equation} In other words, the commutators resulting from $\SU(2)$ are the same 1) as those resulting from $\SO(3)$. So the infinitesimal versions of these two groups are the same. So they're the same group—at least locally.

We can realize this identification explicitly by considering matrices of the form \begin{equation} X = \begin{pmatrix}z& x-iy\\ x+iy& -z\end{pmatrix} . \end{equation} The matrix $X$ is tracefree, that is \begin{equation} \tr(X) = 0 , \end{equation} and Hermitian, that is, \begin{equation} X^\dagger = X . \end{equation} We can act on $X$ with $\SU(2)$ via the action \begin{equation} X \longmapsto MXM^\dagger \end{equation} which preserves both of these conditions. Furthermore, we have \begin{equation} |X| = -(x^2+y^2+z^2) , \end{equation} and, since $|M|=1$, \begin{equation} |MXM^\dagger| = |X| \end{equation} Thus, $M$ can be identified with an element of $\SO(3)$!

However, that's not the whole story. Since there are two $M$s in our action, we can not tell the actions of $\pm M$ apart. Thus, $\SU(2)$ is the double cover of $\SO(3)$. These two groups have different global structure, but their local structure is the same.

Every orthogonal group $\SO(n)$ admits such a double cover, but the double cover is not usually a unitary group. The double cover of $\SO(n)$ is called $\Spin(n)$; we have shown that \begin{equation} \Spin(3) \cong \SU(2) . \end{equation}

But our interest is in the local structure, and $\SU(2)$ is the same as $\SO(3)$ locally.

You may recognize the $s_m$ as being $i$ times the corresponding Pauli matrices, that is, \begin{equation} s_m = -i\sigma_m . \end{equation} What are the properties of the Pauli matrices? They are Hermitian, they are tracefree, and they square to the identity, that is \begin{align} \sigma_m^\dagger &= \sigma_m ,\\ \tr(\sigma_m) &= 0 ,\\ \sigma_m^2 &= 1 . \end{align} Similarly, the $s_m$ are anti-Hermitian, tracefree, and square to minus the identity, that is \begin{align} s_m^\dagger &= -s_m ,\\ \tr(s_m) &= 0 ,\\ s_m^2 &= -1 . \end{align}

More generally, if $M(\alpha)$ is any family of elements of $\SU(2)$ such that $M(0)=1$, then $M(\alpha)^\dagger M(\alpha)=1$ by (\ref{unitary}). Differentiating this equation, we obtain \begin{equation} M'(\alpha)^\dagger M(\alpha) + M(\alpha)^\dagger M'(\alpha) = 0 \end{equation} and evaluating this result at $\alpha=0$ yields \begin{equation} A^\dagger + A = 0 \end{equation} where we have written $A$ for $\dot{M}=M'(0)$. This argument is true quite generally: If the elements of a Lie group are unitary matrices, then the elements of the corresponding Lie algebra are anti-Hermtian. Thus, we can describe the Lie algebra $\su(2)$ of $\SU(2)$ as \begin{equation} \su(2) = \{A\in\CC^{2\times2}:A^\dagger+A=0, \tr(A)=0\} . \end{equation} The same argument shows that infinitesimal orthogonal matrices are antisymmetric, as you can verify in the case of $\SO(3)$ and $\so(3)$. Thus, \begin{align} \SO(3) &= \{M\in\RR^{3\times3}:M^T M=1, |M|=1\} ,\\ \so(3) &= \{A\in\RR^{3\times3}:A^T+A=0, \tr(A)=0\} \end{align} and we have shown that \begin{equation} \su(2) \cong \so(3) \end{equation} not merely as vector spaces, but as Lie algebras (since commutators are preserved).