Mary Esther Brunson was born in Andalusia, Alabama in 1915. She was the 12th child of 12. She grew up at the Brunson Hotel and went to school in Andalusia. It was here as a child that she had as her best friend, her niece, Carolyn Brunson who was barely her junior. They grew up together. Mary was run over by a car when she was about 7. The old car ran across her pelvis leaving her bruised and not seriously hurt. This early childhood trauma left her with a fear of driving and she chose to never drive a car.

In 1930, Minnie Jane Seale Brunson died. Mary then age 15 left her best friend Carolyn and moved to Texas where she came to live with her sister Sue Barber.