It really sounds a lot like the first layer is plaster and not any lightweight concrete. What's behind it is harder to determine.
I would investigate the wall construction a bit more before making any definitive decisions. Maybe consult with the board? Maybe post pictures here?