Definitely; I was asking if you HAD to use ports. If you don't need to use ports then use the binary packages (exclusively).
poudriere has been a bit much for me so I'm looking at moving from portmaster on 13.x to trying binary packages on 14.x to see if I can get away with only binary packages.
But whatever you do, hopefully it sorts out your issue.