Well, first off, I'm not speaking from direct experience of your amp, but I own (and have owned) older Marantz amps, and Rega arms and phono stage. However, it might prompt some other thoughts.
The RP3 is a turntable, and it is really the cartridge that influences the sound-match. If you have or are considering the 'standard' combination with the Elys 2, then I think you'd be fine, unles you were prepared to pay as much as the amp (say around £300) for a separate phono stage. That would give a noticeable improvement if your listening room and speakers were up to it. But it is quite a big cost imbalance unless you have plenty of money to spare.
To illustrate this, for the cost of a PM6004 + a £300 phono stage you are getting into Rega and Roksan amp territory, which you might prefer for c £600.
As ever, it's all about the best for the cash, so if you want a simple system, stick with standard PM6004 + RP3/Elys 2, but if you hanker for better, start by hearing the Rega amp with the RP3. Their dealer in Exeter was pretty helpful when I visited them last year, and worth booking a trip.
Hope this helps.