Vadsø was established as a municipality on 1 January 1838 (see formannskapsdistrikt). The law required that all cities should be separated from their rural districts, but because of a low population and very few voters, this was impossible to carry out for the municipality of Vadsø in 1838. (See also Hammerfest and Vardø.)
The rural districts of Vadsø were separated from the city in three stages: Nesseby (in 1846), Sør-Varanger (in 1858), and Nord-Varanger (in 1894). The last one was, however, merged back into the city on 1 January 1964.
In the 16th century, the settlement consisted of a fishing village and a church, located on the island of Vadsøya. The settlement later moved to the mainland. Township privilege was granted in 1833, and soon settlers came from Finland and the northern part of Sweden, which suffered from famine. Finnish was rapidly becoming the language of the majority, and this continued for decades. Even today Finnish is still spoken in some households. During the occupation of Norway by Nazi Germany, Vadsø suffered several air raids from the Soviet Union, which bombed Nazi troops. However, there are, unlike most places in Finnmark, a number of 19th century wooden houses preserved close to the city centre, notably the house of Esbensen, built by a Norwegian, and the house of Tuomainen, built by a Finn. On the island of Vadsøya is the airship mast used by Umberto Nobile and Roald Amundsen for their expedition over the North Pole with the airship Norge in 1926, and used again on Nobile's flight with the airship Italia in 1928.
The municipality of Vadsø forms the southern coast of the Varanger Peninsula, which is largely covered by birch forests on this more sheltered side (as opposed to the northern side).