I drilled out the 4 holes in the bracket by 1mm. Unfortunately there was no other way

A better solution is that you can buy a threading adapter that will step down the size from an M6 to M4. You can find this cheap at Ali Express.