The Colonial Office was relocated to Great Smith Street after World War Two. There had been plans for a large new set of offices on the site of the Old Westminster Hospital. But decolonisation meant that this would never become a reality.
Picture courtesy of geograph.org.uk. This page also shows its precise location.
The Colonial Office Article
Armed Forces | Art and Culture | Articles | Biographies | Colonies | Discussion | Glossary | Home | Library | Links | Map Room | Sources and Media | Science and Technology | Search | Student Zone | Timelines | TV and Film