You need to edit the connectors for 3D objects that are placed inside and outside the fabric. For example, a zipper puller. In such cases, assign the connectors somewhere in the middle of the depth of the object, where the depth is the dimension that is perpendicular to the surface of the fabric.
The following images show connectors placed inside a zipper puller and the result in the 3D simulation and ray trace render. Note that the part of the zipper that is pushed inwards naturally collides with the layer below, whether a lower fabric layer or the avatar:
Screen Capture from VStitcher:
Ray trace render from VStitcher