The Brown Foundation of Houston granted the new library $218,000 and it was named the Herman Brown Free Library in honor of one the founders of the foundation. Other donations were received from Maurine Goodrich, HEB of Burnet, Epsilon Pi Chapter of Epsilon Sigma Alpha sorority of Burnet, Pen Chat Club, and many Burnet citizens. Groundbreaking began in August of 1976 and the sale of the re-purposed Army building to the Burnet Presbyterian Church made it possible to add the mezzanine. A donation from the Moody Foundation of Galveston made it possible to install the elevator. Maurine Goodrich donated additional funds in honor of Robert D. Goodrich, for whom the meeting room is named. A grant and bequest from the J. Frank Dobie Foundation made it possible to begin the genealogy and local history collections. The 9,200 square foot Herman Brown Free Library building was completed in early 1977.

Many updates have been done since 1977. In 2015, with the help of a grant from the Tocker Foundation, new carpet was installed throughout the building, furniture and shelving were updated, and the wooden beams that stretched across the ceiling above the circulation desk were removed to provide an open feel. The most noticeable change is the addition finished in the fall of 2017 and funded by the Friends of The Herman Brown Free Library,  the heirs of Elizabeth Van Gorkom, and a grant from The Tocker Foundation. This 2,800 square foot space houses the Elizabeth Van Gorkom Genealogical Research Center, beautiful new bathrooms, and the Media Lounge. The original genealogy room, The J. Frank Dobie Room, has been re-purposed as a children’s area.