Is ballast grey or brown?

Ballast can be either colour.

If you are creating a layout of a prototype line then you should try and stick as close to the real thing as possible, the stone would have come from a local quarry so the colour of ballast would change from region to region.

So, for the best results do some research on the line you are creating, look at photographs, even pay a visit to the railway in question.

If, however you are building a layout that is not a current or historical line, you can use either colour to suit your preference.