Both have a phono stage and both reequalize RIAA. But its the NAD's stereo preamp section that is doing such a better job that is making the difference.
What your discovering is why digital AVR's like your Onkyo are not well suited for 2 channel stereo reproduction, even in so called "pure direct" mode. Your older NAD, that was built for 2 channel analog, is blowing it away. Your hearing it yourself.
If your serious about vinyl, ditch the AVR or keep it strictly for car crashes, explosions and battles between giant robots. But not for music.