If the material is free from contaminants, then you can. Unfortunately, flaky and flaky ballast results in rather dull and rough concrete. When classifying ballast, attention is paid to f-shape, which is the ratio between the shortest cross-section of the ballast grain compared to the longer one.
In addition to grain shape, you should also consider mica content and a lot of other factors. There are certain minerals in some rocks that are not volume stable.
The basic rule is that the materials used for concrete must have been "proven suitable" for use in concrete. If your material comes from a quarry that normally supplies masses for construction work, there should be no problem with that.