Protestant
According to the original script for the pilot episode, book is garbed as a Protestant minister.
We see, Passing through frame, Shepherd BOOK. He's about sixty,
weathered and worldly, with a quiet kindness in his eyes. Farmer
stock, not a trace of bullshit and a workingman's hands. He drags a
few boxes and suitcases on a sort of wheeled papoose, carries another
suitcase in his hand. His clothes are plain and instantly identify him
as some kind of protestant minister.
Firely: Serenity - Pilot Screenplay
Catholic
That being said, there's pretty good evidence that Book is some manner of Catholic. He apparently took orders at the Southdown Abbey (a cloistered monastery) and he also takes Confession, atypical for a Protestant minister.
Finally Wong said, “You’re still a Shepherd, yes?”
Book nodded. “I am. I’ve left the abbey for the time being, but I
haven’t left the Order.”
Firefly: Big Damn Hero
He refers to himself as a monk (or at least doesn't correct Kaylee referring to him as one).
“Kinda did,” Kaylee said. “What the Shepherd means is that if someone
confessed that he was going to do something really bad, the monks
would tell on him.”
“Yes. We would tell on him,” Book agreed. “Or her.”
And we see him in his cassock in Serenity: The Shepherd's Tale
