Remove it. It is not appropriate to mislead people as to the submission or publication status of your paper. You should also remove any other reference to the conference appearing in the paper (e.g. a logo).
Additionally, you'll have to upload your LaTeX source to arXiv in order to have it posted, including the conference style file, so you should make sure its copyright allows this.
(Even if you really had submitted the paper to that conference, you typically wouldn't acknowledge this in the preprint. The reason is simple: if the paper isn't actually published in that conference, people will be able to infer that it must have been rejected, and this is something you'd probably rather were not publicly known.)
Update: Here is a question from someone who used a template that included the name of Journal X when submitting to Journal Y, and had the paper rejected on suspicion of double submission. While I think that Journal Y was out of line in doing this, it does illustrate that these sort of misunderstandings can have serious consequences.