Don't worry, the Ancient Totem can't be lost even if you forget it in a temporary location. You'll find it in your Home Storage, sled, or ship.