Finally decided on soldering a PCB and when I programmed it, it wouldn't start the program afterwards. So I was calling for help on twitter and got some well meant advices about pulling certain pins that no ESP32 dev board pulled up. It took me a while to notice that I was missing the capacitor in my reset circuit. I didn't check what C6 actually does while debugging, so I thought being so close to the ESP reset pin and resistor, that it was part of it. With a bodged on capacitor it now resets to run the program!

So far it's only a cartridge shaped ESP dev board that can talk to 5V devices, but it's good to know this is programmable!