The Young's home had originally been built in 1856 for Laura Smith, who was the daughter of a plantation owner and a slave mother. The anonymous Mississippi plantation owner built the house so that Laura's freed slave mother could raise their children away from the degradation of slavery in the Deep South. Soon after the Smith's moved into the home, the house became a station on the Underground Railroad. Other reports also claim that the house had been an inn for stage traffic from Cincinnati to the Ohio state capital.

Charles Young purchased the Young House with his mother Arminta after his father's death in June 1894. Young and his mother called their new home "Youngsholm." Young dated his fiancé, Ada Mills, who was from a middle-class Oakland, California family while he was out West. He married Ada in the winter of 1904 in Oakland. They eventually moved to “Youngsholm" with his mother where they raised their two children, Charles "Tonton" Noel - born on Christmas day in 1906, and their daughter, Marie "Kikik" Amelie, born in March 1909.

Young was a wonderful family man, a very loving husband and father. Young's close friend and regimental Chaplain - Major O.J. W. Scott said, speaking of Colonel Young that, "He was to his family, a faithful son, devoted husband, a loving father, and that his favorite sentiment was that from the Song of Solomon... “ Love is strong as death; jealousy is cruel as the grave; the coals thereof are coals of fire, which hath a most vehement flame. Many waters cannot quince love, neither can the floods drown it.” Scott went on to say that Young often taught that it does not pay to hate anyone, and That thoughts are forces that attract like forces to themselves. Scott also said that he and Young had discussed the belief that if we send out thoughts of love, forgiveness, and sympathy, we will be backed up by the good Father who is the "Loving, Intelligent Will" - who is the "All and in All"...so I feel that it is important to reitorate that Young taught not to hate.