Thanks for the help everyone. What a wonderful community!
I followed the advice to wait a long time to let the filler dry. It's been almost a month now. I screwed in the holder for the roller blind and it went well without the plugs coming loose.
I put up the roller blind and started pulling the cords and discovered it was just as difficult as before the entire fitting came loose. Spent two hours trying to understand how the mechanism worked and still didn't get it but sprayed in a lot of lubricating oil and after that the mechanism worked much easier. When I afterwards put up the roller blind again, it worked well.
The previous problem must have been that it was poorly oiled and therefore difficult to operate, which resulted in high forces on the wall mount that came loose since it was only plugged into the grout.
As mentioned, it works now, but time will tell how long it lasts. I'll try to provide an update in a couple of months regardless of the outcome.