To turn the back on one

to forsake or neglect him.

See also: Back