I agree with what CL said and I personally wouldn't worry seeing that there is no resultant problem.
You must remember you are raising and lowering the arm mechanically and if the lifter has any slope be it from wear, misalignment or even a TT out of level, what you are experiencing might happen. My stylus normally returns to the same (or an adjacent) track after a lift and set down but my deck and hence lift mechanism is relatively new and I raise and lower slowly.
Anti-skate forces are quite low so I can't see how the arm would move sideways whilst sitting on a completely level, good condition lift pad at normal VTFs. However, at low VTF settings with a rapid lift, I could imagine the arm being induced to bounce lightly on the pad and perhaps move a little sideways depending on the anti skate force set. Alternatively, I could also imagine the arm/stylus moving slightly sideways on set down once the pad is withdrawn but only if the anti-skate is incorrectly set. Personally I'd check/investigate what I could and yes ideally, it would be nice to have co-incident lift and set down points, but not a big deal if all else is OK.