It turned out to be a problem in my wirings. 

What I thought was the most logical pinout of my audio socket turned out to be wrong. When a 3.5mm jack is plugged in, the tip of it gets tied to the metallic shielding on a pin 2, what made me initially believe was reserved for the GND, that is for the sleeve of the jack. But no, it's tied to the tip.

There also seems to be an error in the datasheet, bc after inspecting it, it turns out that pin 3 is actually jack sleeve, while pin 1 is its ring. After inspecting the connecter with a more caution this time, I found out that proper way to connect it is :

Pin 1: R (jack ring)
Pin 2: L (tip of the jack)
Pin 3: GND (sleeve of the jack)
Pin 4: NC (metal piece tied to the Pin 2)

Unfortunately, it only improved a little my signal-to-noise ratio. After examining contacts with multimeter for the 100th time, and left without ideas, I fiddled with the jack in and out of the socket. And marvelously, found a position where the sound gets crystal clear!

Crystal clear sound only comes out when:
HPLOUT is tied to the R in+
HPROUT is not connected
GND is tied to the L in+
Additionally, extra noise reduction is achieved when the GND from PB is tied to the PE of the amplifier (that is also tied to amp gnd at one single point)

This is very curious, since the preamp behaves as it is a differential input amplifier and it is not.