I work in this field. Yes, you should contact the Editor to ask them what their policy is on submissions that are based on a prior conference publication -- but ask very carefully. See the last two paragraphs of this answer before writing to them. In particular, I suggest contacting the Editor-in-Chief. It's fair game to write and ask them what their policy is, regarding whether they accept journal submissions that are an extension of a conference paper and if so what the requirements are.
I looked through the guidelines for authors for this particular journal, and I have to say, it's not clear to me what the expected policies are. Therefore, I think it's entirely reasonable to contact the Editor-in-Chief to ask for a clarification.
I can't tell you what this journal's policies are, but I can give you my impression of what a common policy in this field is. It's not uncommon for journals to allow publication of a journal version of a paper that was previously published at a conference. Yes, one common rule-of-thumb is that the journal version of the paper should have at least 25% new content. Of course this is a subjective judgement call, but it gives you a rough idea of what to shoot for. Also, the journal version should be "archival-quality" (e.g., the definitive reference; polished, high quality, well executed). But this can potentially vary. There's no requirement that every journal has to subscribe to this particular policy -- it's up to each individual journal to decide on its own policy, and it would be entirely fair for any particular journal to choose something different.
For an example of the 25% requirement, see the ACM's policy for submissions to ACM journals: http://www.acm.org/publications/policies/sim_submissions/
Do note that what matters is not the percentage of words that are new, but rather the percentage of intellectual ideas. We measure the contribution of each paper by its "delta" over previous publications, i.e., by the new intellectual content of the new ideas that have not previously appeared. That can't be measured by counting the number of words that are new.
In journals that do allow you to publish a journal version of work that previously appeared at a conference, I would not expect any requirement that you have to rewrite or rephrase every sentence of the previously published work. (It's not clear what purpose that would serve.) You are of course expected to revise anything that can be improved, but there's no requirement to paraphrase or reword your own writing just to avoid having the same words in the same order.
Your question was a little unclear about how the previous work was previously published. In my experience with reputable conferences in this field, the most common kinds of papers are: "papers published at a conference" and "papers published at a journal". Less commonly, there's "posters presented at a conference" (usually does not come with a paper that's considered published), and occasionally "short 1-page or 2-page paper published at a conference with a poster presented at the conference" (the 2-page paper would be considered published). If you're referring to the latter kind of publication, then I would not have expected that to lead to outright rejection of your journal submission without consideration of its merits. But who knows? It's up to each journal to determine its own policy.
So, given that the publicly available web pages of that journal aren't entirely clear, I think it's perfectly reasonable to email the Editor-in-Chief and ask them what the policy is. You can point to your submission and explain the situation if you want.
However, it's extremely important to avoid conveying any implication that you feel you've been wronged, that their decision was inappropriate, or any other form of criticism. The purpose of this email should be solely to gather information about what the journal's policy is. Once you know their policy, then you can consider what to do. But don't write to the Editor-in-Chief to complain or dispute or appeal the decision. That just sets up an adversarial situation that isn't likely to yield any constructive results. You might want to write a draft email, sit on it for a few days, edit it to remove any hint of disappointment or criticism, then give it to a senior colleague to read, and follow their guidance.
And you can also treat this as a learning experience. If you do decide to submit it elsewhere, you can check what their policies are on this sort of situation before submitting. If it's not clear from their web pages, you can contact the editor in advance to ask.
