Using data from the Longitudinal Study of Astronomy Graduate Students (LSAGS), we examine factors that affect attrition from physics and astronomy using two different measures of attrition. We have now conducted two rounds of surveys, during 2007-08 and during 2012-13. This analysis focuses on a subset of respondents, all of whom had Ph.D.s in astronomy, astrophysics, or a related field at the time of the second survey. While the imposter syndrome and mentoring affected the likelihood of respondents’ thinking about leaving the field, they did not directly contribute to actually working in a field that was not physics or astronomy. Relationship with graduate advisors and the two-body problem both had significant effects on working in physics or astronomy, as did completing a postdoc. The sex of the respondent had no direct effect on our measures of attrition, but indirectly affected attrition because women were less likely to report positive relationships with graduate advisors and more likely to report two-body problems. This research identifies specific areas of concern that can be addressed by the scientific community to increase the retention of all people, but especially women, in astronomy and astrophysics.