No, you’re right, Inkscape can be a huge pain in the butt in this regard, which is why I was suggesting doing it sort of manually - placing a lot of guidelines and centering things based on these, or at least validating that way. In doing so, you’ll soon discover the FPD has the origin in the top left corner (legacy from originally using Zoner Draw for the panel design, which was page-coordinate based). Moving the origin in FPD might help.
I don’t know if Inkscape can handle it, but you could try and import the .dxf file from the Wiki page into Inkscape.
That’s how I started my layout. Once I imported, I grouped everything together, the used a scale function to size everything to the correct overall size. For the front panel, 232.41 x 163.27 mm; For the Rear, 52.58 x 217.93 mm.
If Inkscape can’t handle .dxf, I could try and give you a direct .dxf import from Illustrator and save it as .svg.