I also had the same problems with the Fairphone adapter. The only remedy was to change the adapter. I bought the following but it is currently not available: https://www.digitec.ch/de/s1/product/proxtend-usb-c-35mm-headset-adapter-audio-adapter-16815462. As an alternative, I would also consider a new headset with a USB-C connection. This also works much better than an adapter. Since

Unfortunately, the Fairphone adapter didn't work properly for me either. I didn't notice anything myself, but my conversation partners told me that it was much too quiet and that a hissing noise could be heard in between. I had no other option than to buy an adapter. In my case it was the one from Samsung:
Samsung USB-C Audio Adapter (USB Type C, 3.5mm)

I don't hear any noise at all.
Maybe your module from the Fairphone or the cable socket is damaged.