Perhaps you can allow color weights or allow the user to choose the preferred color? The tool is for the user so let them decide, and have some kind of default.
An example of weighting, let Red have weight = 80, Green = 20, both in the palette. Now, which color should Blue be swapped with? -Red, since it has higher weight. Manual weighting might become tedious with many colors, which is why there should be a default or an alternative.
An example of weighting, let Red have weight = 80, Green = 20, both in the palette. Now, which color should Blue be swapped with? -Red, since it has higher weight. Manual weighting might become tedious with many colors, which is why there should be a default or an alternative.