What colour is ballast?

On the real railway line, the ballast would have come from a local quarry so the colour would reflect this. The colour will mostly epend on the type of stone common ones are limestone and granite, with colours varying from white to grey and brown.

The colour will also vary over time. Ballast that was laid a long time ago will be heavily weathered while freshly laid ballast will be more vibrant.

Some railways has specific customs such as the use of ash cinders (which would be black) in the NER region.

So as always, make sure you study the prototype if you want to be accurate.

Obviously, if you are just building a railway with no specific region in mind then the choice of colour is yours.