Although early settlers were first attracted to the area that would become known as Sag Harbor in the late 1600s because its meadows offered an abundant supply of salt hay to feed their livestock, they soon saw its potential for a port. That port would soon supplant Northwest Harbor and North Sea.
Shipping was established in Sag Harbor as early as the turn of the 18th century, and there were efforts dating to the early 1740s to build a proper wharf, but the first one was not built until 1761 when Nathan Fordham Jr. and James Foster constructed a modest structure a few hundred feet to the west of the current Long Wharf.
Work on what is now Long Wharf did not begin until 1770 when an association of men from East Hampton and Southampton bought land on either side of the town line and built a structure that was 30 rods (a rod is 5 and a half yards) long and 35 feet wide. It was completed in 1771.