Let's finally add the LM358 to the mix.

Without Load

It works quite well without any load.

Positive & Negative outputs are within 50mV of the input with a simple voltage follower.

With Load

As expected, with some load, the voltage isn't that great : 3.3V only at 2.4mA, which isn't surprising, given the output of the charge pump.

Conclusion & Improvements

My initial idea didn't work as well as expected. Without load it is perfect, but as soon as some load is added, the voltage drops. 

But the good findings are that the charge pump is the weak link here. So, if it is improved it should be fine.

I'll planning to try 2 improvements:

• Using a LMC555, which is the CMOS version of the NE555 as suggested in the comments by Ken Yap.
• Using a CD4069 CMOS buffer to achieve the same effect than using the LMC555.

Once I'll manage a stable output even with the max expected load from the LM358, I'll aim to miniaturize on a PCB.