Colour-permuting automorphisms of complete Cayley graphs

Let $G$ be a (finite or infinite) group, and let $K_G = \mathrm{Cay}(G;G \smallsetminus \{1\} )$ be the complete graph with vertex set $G$, considered as a Cayley graph of $G$. Being a Cayley graph, it has a natural edge-colouring by sets of the form $\{s, s^{-1}\}$ for $s \in G$. We prove that every colour-permuting automorphism of $K_G$ is an affine map, unless $G \cong Q_8 \times B$, where $Q_8$ is the quaternion group of order $8$, and $B$ is an abelian group, such that $b^2$ is trivial for all $b \in B$.
We also prove (without any restriction on $G$) that every colour-permuting automorphism of $K_G$ is the composition of a group automorphism and a colour-preserving graph automorphism. This was conjectured by D. P. Byrne, M. J. Donner, and T. Q. Sibley in 2013.