\[a^{2} - 5ab - b^{2} + a^{2} + b^{2} =\]
\[= 2a^{2} - 5ab.\]
\[a^{2} - 5ab - b^{2} - \left( a^{2} + b^{2} \right) =\]
\[= a^{2} - 5ab - b^{2} - a^{2} - b^{2} =\]
\[= - 5ab - 2b^{2}.\]