The Colonial Office was relocated to Great Smith Street after World War Two. There had been plans for a large new set of offices on the site of the Old Westminster Hospital. But decolonisation meant that this would never become a reality.
Picture courtesy of geograph.org.uk. This page also shows its precise location.
The Colonial Office Article