I'm pretty sure there are Fundamentalist Mormons who would say LDS has veered away from the founder's original teachings. Their church started in 1830 and it was only in 1904 they disavowed polygamy and started excommunicating members for it. The split-offs still practice it to this day, but they may use strategies like "religious marriages" to avoid legal problems. I also believe LDS members of standing would call any FLDS or AUB member a "Jack Mormon" even if they were born into their sect.

So not so much "fallen away" as "not following the current LDS orthodoxy".