Our research group was given permission to visit Wakefield, so on Tuesday three of us went to ___ prison to interview the governor.

Explication

The researchers are visiting the building, so the definite article is used. Go to prison without an article means being sent there as a convicted prisoner, which is clearly not what happened to a research group with an appointment.