If you don't 'have to' use brick, both gypsum and minerite work.
However, gypsum tends to become porous from the heat, so minerite is probably preferable.
It's safest to consult with the chimney sweep beforehand, but it would be strange if he opposed any of the above options.