Both have a phono stage and both reequalize RIAA. But its the NAD's stereo preamp section that is doing such a better job that is making the difference.
What your discovering is why digital AVR's like your Onkyo are not well suited for 2 channel stereo reproduction, even in so called "pure direct" mode. Your older NAD, that was built for 2 channel analog, is blowing it away. Your hearing it yourself.
If your serious about vinyl, ditch the AVR or keep it strictly for car crashes, explosions and battles between giant robots. But not for music.
VPI Traveler V2, Ortofon Rondo Blue, Simaudio Moon 110LP
Technics SL-1400, Audio Technica AT-14sa, Pro-ject Phono Box S
Marantz PM15S2 LE, Oppo 103, Simaudio Moon 100D
Dynaudio Focus 260's & a REL T1 Sub