The Chief has left you a part of the map where you can see the location of the fabric fragments. Check it out! :)