The problem is with bgsync.cpp here:

if (theirTS < _lastOpTimeFetched) {
    log() << "replSet we are ahead of the primary, will try to roll back" << rsLog;
    theReplSet->syncRollback(r);
    return true;
}

If we're syncing from the primary, then this logic is valid. If we are not, then this member must have applied some ops from the buffer and got ahead of the sync source, which is okay and we don't want to rollback. Probably the right fix is to make sure the buffer is empty before choosing a new sync target.

This is a fairly annoying bug to run into. If the member is in the middle of applying a batch of ops, the rollback will fail (because you're not at minvalid) and put the member into FATAL state. (I think restarting should fix it, though.)