The Burnet County Library was established in 1948 on the second floor of the old Burnet County Jail, located on the north-east corner of the Burnet Square. The library started with $500 worth of books and 1,000 books on loan from the state library. In 1949, the City of Burnet purchased a surplus Army building and had it moved to the land behind the jail. The library was moved into this 4,500 square foot building in 1951. A wing was added in 1962 in memory of the first librarian, Miss Edwin Sue Goree.
Burnet quickly outgrew their small library and began looking for a new location. Gavin Garrett of Lampasas donated the lot where the Herman Brown Library now sits and funds were raised for a new building. The Brown Foundation of Houston granted the library $218,000 for the new library and it was named the Herman Brown Free Library in honor of one the founders of the foundation. Other donations were received from Maurine Goodrich, HEB of Burnet, Epsilon Pi Chapter of Epsilon Sigma Alpha sorority of Burnet, Pen Chat Club, and many Burnet citizens. Groundbreaking began in August of 1976 and the sale of the re-purposed Army building to the Burnet Presbyterian Church made it possible to add the mezzanine. A donation from the Moody Foundation of Galveston made it possible to install the elevator. Maurine Goodrich donated additional funds in honor of Robert D. Goodrich, for whom the meeting room is named. A grant and bequest from the J. Frank Dobie Foundation made it possible to begin the genealogy and local history collections located in the room bearing his name. The 9,200 square foot Herman Brown Free Library building was completed in early 1977.
Many updates have been done since 1977. In 2015, with the help of a grant from the Tocker Foundation, new carpet was installed throughout the building, furniture and shelving were updated, and the wooden beams that stretched across the ceiling above the circulation desk were been removed to provide an open feel. The most noticeable change would be the addition finished in the fall of 2017 that was funded by the Friends of The Herman Brown Free Library, the heirs of Elizabeth Van Gorkom, and a grant from The Tocker Foundation. This 2,800 square foot space houses the Elizabeth Van Gorkom Genealogical Research Center, beautiful new bathrooms, and the Media Lounge. The original genealogy room, The J. Frank Dobie Room, has been re-purposed as a children’s area complete with preschool computer, play table, blocks, puzzles and puppets.