Ten minutes by ferry from lower Manhattan sits a 40-foot-tall red sandstone castle most New Yorkers have never stepped inside. Castle Williams, on Governors Island, began life between 1807 and 1811 as one of a chain of coastal defences protecting New York Harbor, alongside the star-shaped Fort Jay. Once technology moved on, its days as a serious fort faded, and it cycled through roles as troop quarters, a Civil War prison, and later part of what became the largest and most complex Coast Guard installation in the Atlantic Area Command. After the military left in the 1960s, the building survived demolition threats, a brief stint as a youth centre, a 1990s restoration, and finally joined the Governors Island National Monument in 2001.

Visiting is refreshingly low-fuss. The ground floor and open-air courtyard are open for self-guided exploring, with casemates — the thick-walled enclosures once built to shield artillery — now housing exhibits on the fort's harbour-defence history. One casemate has deliberately been left untouched so you can see the original design. Rangers hover nearby for questions, and occasional guided tours (30 to 45 minutes) unlock the upper floors, where casemates became prison cells and barracks. The tour ends on the roof: panoramic skyline and harbour views, plus hulking Civil War cannons left in place because they were simply too heavy to move. Tours run a few times a day, Wednesday to Sunday, from late May to late October.

The castle is only one excuse to spend a day on Governors Island. The 172-acre spot is entirely car-free, laced with pedestrian paths and about 7 miles of bike routes — Blazing Saddles rents bikes, surreys and pedal cars. There's glamping if you want to stay overnight, the QC Spa wellness centre, rotating public art, and events like the Porch Stomp folk festival. Families get Slide Hill, home to New York City's longest slide, plus the Hammock Grove Play Area. History fans can pair Castle Williams with Fort Jay, built in 1797 and also explorable on a self-guided basis.

Practicalities are simple. Ferries leave roughly every 30 minutes from South Ferry in Manhattan year-round; Brooklyn service runs weekends and holidays from late May to early November, and a round trip costs $5. Everything is seasonal — summer Thursdays through Sundays are the busiest days — so check the National Monument calendar before building a trip around a castle tour.

This one suits anyone who likes their sightseeing with a bit of grit: a prison-turned-monument, rooftop cannon sunsets, and a car-free island to wander, all for the price of a coffee.