Opposition leader Sussan Ley won't say if she supports a NSW council's contentious plan to remove Aboriginal and Torres Strait Islander flags, but has said "we should unite under one flag".