I have the same problem so took my Dremel with a small rotating sanding drum to the adapter and now it works fine. It looks rougher in the pics than it actually is, and it works fine.
I found it was sometimes awkward getting the phone lined up with the connecter so I glued some stiff card under the connector to raise its angle slightly. Pics here -
