The site in Cromer was earmarked for closure last year after it saw a downturn in the number of people using it.
Instead, the majority of visitors are choosing to get information about the town online or from other locations like cafés and B&Bs.
Its closure was one of several options put forward to save money at the cash-strapped council, which also included closing some of its public toilets.
At a North Norfolk District Council cabinet meeting this week, the final decision was made to proceed with its closure.
Liz Withington, cabinet member for leisure (Image: Newsquest)
Liz Withington, portfolio holder for leisure, said: “This does not involve us no longer committing ourselves to tourism.
“We need to move this process forward so we can consider the future use of the building and how we can support tourism in other ways.
“With the season coming up, we need to address this with some urgency. ”
The closure of the site was unanimously approved, with Ms Withington adding that there had been a high level of interest from people wanting to lease the building at the Meadow Road car park.
Cromer Pier and seafront (Image: Newsquest)
The council will now investigate how it can continue to provide a tourist information service in the town.
This could involve an enhanced online presence, electronic information boards or physical leaflets at prominent tourist destinations like Cromer Pier.
It also aims to relocate the interactive Deep History Coast display currently housed at the visitor centre, with hopes to move it to the nearby Cromer Museum.
Businesses interested in leasing the visitor centre building will be able to express their interest from Friday, March 28.