You are just asking for trouble wanting to setup your own mail server. I am not saying you can’t but the amount of work required to setup, maintain, keep off blacklists, etc is way more work than simply installing something like WP Mail SMTP by WPForms to handle all that for you via a trusted mail server.
If you still want to set up your own mail server, you will probably want to setup a docker container you could setup mail server docker container and hook it up with EE’s network. Maybe this: https://github.com/tomav/docker-mailserver/
P.S. Having done both, I would highly recommend you look into WP Mail SMTP by WPForms as it will make your life in the short run and the long run a lot easier. I use Gmail through it and it works great but they have options for other services and also a manual SMTP mode.