In 1764, New York patents for over 15 thousand acres of land along the Mettowee River were granted to about 30 officers who had served in the French and Indian War. The initial immigrants to settle in Granville were mostly from New England. Granville's population grew slowly until the 1790s. The New England origin of the settlers caused them to favor Vermont's laws of universal suffrage and individual land tenure. In 1782 many residents of Granville participated in the failed attempt to annex Charlotte County (an area encompassing most of Washington County and Vermont's Rutland County) to Vermont.

 Gristmills, sawmills, blacksmith shops, woolen and cotton mills were some of the early industries to be established in Granville. Flax and wool from nearby farms were spun and woven by local families. It was said "the rattle of the loom and the buzz of the spindle could be heard in every house."

With the discovery of slate in the 1850s, Granville experienced a significant growth in population. By the 1890s, local slate companies were actively recruiting Welsh quarrymen. Soon, Italian, Czechoslovakian, Irish and other European families arrived in Granville to work the slate quarries.

The success of the slate industry was reflected in the wealth and influence of the slate barons. Hugh W. Hughs, the so-called "slate king of Granville" arrived from Wales in 1857 with three dollars in his pocket. Thirty years later Hughs owned some of the most productive slate quarries in the region and was making annual sales of over $200,000. His Elmwood Cemetery monument with a life-sized statue of Hughs was said to have cost $12,000 in 1890.

After accumulating considerable wealth in the fur business in New York City, Franklin T. Pember pursued a lifelong vocation as a naturalist, collecting wildlife species from all over the world. In 1901 he built the Pember Opera House, an 800-seat facility said to be "one of the finest&in Northern New York." In 1909, the Pember Library and Museum opened. The museum still houses Pember's vast collection of rare birds, mammals and reptiles.