Pontryagin Duality provides a way to define the Fourier transform of a \(L^1(G)\) function on any locally compact abelian group \(G\). If we remove the abelian hypothesis, it breaks. What goes wrong?
First let’s see the Fourier transform on a locally compact abelian group \(G\). Take a function \(f:G\to \mathbb R\). It’s Fourier transform will be a function on the Pontryagin dual group \(\widehat G = \Hom(G, \mathbb R^\times)\), the group of characters of \(G\), and is defined \[\hat f(\chi) = \int_G f(x)\overline{\chi(x)}d\mu(x)\] where \(\mu\) is the Haar measure on \(G\). Then the inverse Fourier transform gives us \[f(g) = \int_{\widehat G} \hat f(\chi) \chi(g)d\mu(x)\] or something, where we’ve used Pontryagin duality to identify \(G\) with its double dual. What this says is the \(f(g)\) can be recovered from the values of \(\chi(g)\) for all characters \(\chi\).
Now let’s suppose \(G\) is non-abelian, and we’ll study its characters. Fun fact: if \(\chi\) is a character, then it is trivial on the commutator subgroup \([G,G]\). Recall that \([G,G]\) consists of all elements of the form \([g,h] = g^{-1}h^{-1}gh\) for \(g,h\in G\). Then since the target of any character \(\chi\) is abelian, \(\chi([g,h]) = 1\). This means the characters of \(G\) are in one-to-one correspondence with the characters of \(G/[G,G]\). In particular, any function \(f\in L^2(G)\) which is not trivial on the commutator subgroup won’t be recoverable from its Fourier transform above.
The fix, then, is to consider higher dimensional representations of \(G\). This leads you to matrix coefficients, representaitons \(G\to \GL(V)\) for higher dimensional vector spaces \(V\).