Any building for workers can be easily sold! It's done the same way you sell other buildings and objects at your Station.

If the option to sell is unavailable, just wait until the production at your factories stops. Let the workers come home and get all ready for the big move :)
As soon as the option appears - sell away! Just make sure you still have enough workers in the end ;)