It started as a food for the wealthy
Ice cream consumption in North American colonies is documented as early as the middle of the 18th century, and much earlier are the recipes that describe how it is made. The combination of cream, sugar, and flavorings combined to nearly frozen consistency—ice cream—arrived with the Europeans, despite the fact that there are accounts of indigenous peoples enjoying snow and ice shavings flavored in various ways. Only the wealthy knew about the dish for almost a century. It was pricey. Most Americans at the time considered sugar to be a luxury. Ice was pricey in the summer, even though it was common in the winter.
Ice cream was clearly a favorite food of Jefferson's and many of his fellow Virginians. A large portion of America's wealth was concentrated in and around Philadelphia in the early years of the country's new government under the Constitution. Those few wealthy Americans who could afford to imitate their betters purchased ice creams from confectioners in Philadelphia, Baltimore, and other major cities because wealthy Americans preferred it as a dessert at formal meals. Americans with less money found the combination to be a rare pleasure because they couldn't afford to buy big quantities of sugar and ice.
In the late 18th and early 19th centuries, ice cream recipes and descriptions of the equipment required to prepare it in large quantities first appeared in American cookbooks and recipe collections. In what was then known as the President's House (White House), Jefferson offered ice creams, including a dish in which frozen ice cream was served in a baked crust. So, despite the third president's efforts to popularize the dish, it remained essentially a pleasure for the wealthy rather than a staple food for the average person.