Actually, I imagine that mechanic as working like this:
Cities have base hitpoints, which are upgraded with walls and other buildings.
Units can be "garrisoned" inside the city, which further adds to the cities hit points. They can also be "ungarrisoned", which accordingly reduces the hitpoints and stations the unit outside the city.
Though it would make sense if the unit is lost for good too, that would make for lesser micromanagement - mass-produce units, add to city, get over with it.