Yes, the pipe is what makes the sound. I've got a stock "silencer" on my FMF fatty...because it's marginally more quiet, and I'm too lazy to put the FMF "silencer" back on. FMF has the same connection as stock, in the same location.
If you were just going to get one, I would just get the pipe. While the stock silencer is most likely more restrictive than any aftermarket one, a silencer is a silencer, the pipe is what makes the difference.