Housing growth is prevalent in rural areas in the United States and landscape fragmentation is one of its many effects. Since the 1930s, rural sprawl has been increasing in areas rich in recreational amenities. The question is how housing growth has affected landscape fragmentation. We thus tested three hypotheses relating land cover and land ownership to density and spatial pattern of buildings, and examined whether building density or spatial pattern of buildings was a better predictor for landscape fragmentation. Housing locations were mapped from 117 1:24,000-scale USGS topographic maps across northern Wisconsin. Patch-level landscape metrics were calculated on the terrestrial area remaining after applying 50, 100 and 250 m disturbance zones around each building. Our results showed that building density and the spatial pattern of buildings were affected mostly by lake area, public land ownership, and the abundance of coniferous forest, agricultural land, and grassland. A full 40% of the houses were within 100 m of lakeshores. The clustering of buildings within 100 m of lakeshores limited fragmentation farther away. In contrast, agricultural and grassland areas were correlated with higher building density, higher fragmentation, and more dispersed building pattern possible legacies of agricultural settlement patterns. Understanding which factors influence building density and fragmentation is useful for landscape level planning and ecosystem management in northern Wisconsin and areas that share similar social and environmental constraints.