To keep one's distance

to stand aloof; to refrain from familiarity.

See also: Distance