It seems to be in three parts. A head that I can unscrew using the flat head, some type of casing that's attached to the longer part going into the wall.
Attaching two pictures. Complete, and without the head.
If it doesn't work with a pair of pliers around the casing, take two nuts and screw them on and let them lock against each other. Then you can use a wrench to unscrew everything by holding onto the innermost nut.
Thanks for the tips. I didn't have enough tools or any nuts at home, but I managed to land a couple of good hits with the hammer so the threads broke on one of the screws. Then I could unscrew it. I wasn't quite as lucky with the other one which broke off, so it's now going to live on in the wall