If a Jew breaks halacha because he honestly does not believe that the Torah is G-d's law or that he is obligated it follow it, is his nonconformance with the law considered wickedness or ignorance? That is, is he Rasha or Am Ha'aretz?
Assume he knows the laws, just does not believe them to be given by G-d.
This is a really great question. Although I think your binary choice is too limiting, it's an interesting dichotomy*.
As in all things Jewish, the simple one line answer is that "It Depends".
In the modern world, the typical person who does not keep mitzvot because they don't think that they are really Gd's will is an Am Haaretz. This type of person is normally typified by statements such as, "Oh the food is kosher, it's just that the Rabbis are trying to gain political points or get more money.", Or they might say, "Tradition never said anything about watching TV on Shabbat, and it helps me relax, whoever said you couldn't watch TV was just confused." These are traditional Am Ha'aretz points of view because they are looking at a very limited impact of the laws and society and don't understand the larger points. Such as how it affects the community, or what other halachot are being balanced and applied.
If a person lives in a healthy and stable environment within a Charedi community (or a Religious Kibbutz or any other similarly closed off Jewish community)**, and are not personally exposed to some terrible situations which have become popular in the press recently... And such a person does not feel that the Mitzvot are binding, that such a person is a Rasha. Such a person is not really rejecting Gd or the Mitzvot, they are rejecting their community, friends and family. However, instead of figuring out which aspect of their community they are rejecting, they turn around and declare that they are rejecting Gd instead. It is not a lack of education of the inner workings of Torah that causes them to reject Gd but rather an aspect of evil within themselves. If a person really questioned if the community they grew up in was Gd's will, then they would explore other options of what Gd's will might be. The premise for the story of the Kuzari, for example is how a non-wicked person would act. The King in the Kuzari story says that he does not know how to follow Gd's will, so he brings in people from each religion to learn about all paths and then make a decision. If the person rejecting his upbringing also wasn't a Rasha, they would do the same thing, and examine the many other paths of following Gd's will. The important thing in the case of the Rasha, is that the person grew up surrounded by Jews and Halachic life, had a good family, good friends, and a healthy life. To a normal person, this would all be evidence that they are in fact leading a good life and following Gd's will. So to reject all the blessings that Gd has given them, would be acting as a Rasha.
*They could also be an Acher type person, who feels they are doomed and will be punished, but doing the mitzvot doesn't help at all. Or they could just be confused, or growing, or any sort of other things. Rav Kook even argues that such a a person might be acting as a Tzadik if their motives are pure, and eventually they will return to Gd.
** I don't mean to imply that a Charedi community are the only communities that does things correctly, I'm only trying to give an example of a fully encompassing Jewish community where what is written in the books or taught in schools, and what one sees in life are the same thing. A rare thing in the modern world.
I'm not sure exactly what you're getting at here, but here's Rambam, Laws of Teshuva Chapter 3:
ג,יד [ו] ואלו שאין להן חלק לעולם הבא, אלא נכרתין ואובדין, ונידונין על גודל רשעם וחטאתם, לעולם ולעולמי עולמים: המינים, והאפיקורוסים, והכופרים בתורה,
...
ג,יז שלושה הן הכופרים בתורה: האומר שאין התורה מעם ה', אפילו פסוק אחד, אפילו תיבה אחת--אם אמר משה אמרו מפי עצמו, הרי זה כופר בתורה; וכן הכופר בפירושה, והיא תורה שבעל פה, והכחיש מגידיה, כגון צדוק ובייתוס; והאומר שהבורא החליף מצוה זו במצוה אחרת, וכבר בטלה תורה זו, אף על פי שהיא הייתה מעם ה', כגון הנוצריים וההגריים. כל אחד משלושה אלו כופר בתורה.
...
ג,כו במה דברים אמורים שכל אחד מאלו אין לו חלק לעולם הבא, בשמת בלא תשובה. אבל אם שב מרשעו, ומת והוא בעל תשובה--הרי זה מבני העולם הבא, שאין לך דבר שעומד בפני התשובה: אפילו כפר בעיקר כל ימיו, ובאחרונה שב--יש לו חלק לעולם הבא, שנאמר "שלום שלום לרחוק ולקרוב, אמר ה'--ורפאתיו" (ישעיהו נז,יט).
ג,כז כל הרשעים והפושעים והמשומדים וכיוצא בהן שחזרו בתשובה, בין בגלוי בין שחזרו במטמונייות--מקבלין אותן, שנאמר "שובו בנים שובבים" (ירמיהו ג,יד; ירמיהו ג,כב): אף על פי שעדיין שובב הוא, שהרי בסתר בלבד חוזר ולא בגלוי--מקבלין אותו בתשובה.
The following do not have a share in the World to Come; instead they are cut off and lost, brought to justice for their great wickedness and sins eternally: the mins, the apikoruses, those who deny the Torah ... mins are those who reject the existence of G-d ... apikoruses reject that He is involved with, and communicates with, humanity ...
There are three types of "Torah deniers": 1.] one who says the Torah is not from G-d, even one verse or word -- if he says Moses made it up, that's a "Torah denier"; similarly 2.] one who denies its explanation i.e. the Oral Law and contradicts its bearers, such as the leaders of the Sadducees and Boethusians; and 3.] one who claims that the Creator dropped His Word in favor of a different one, and this Torah is obsolete, though it was originally G-d-given -- such as the Christians and Muslims. All these three are "Torah deniers."
... All the above lose their share in the World to Come only if they die without repenting first; even someone who expressly rejected the Existence of G-d their entire life and at the end came back, has a share ...
All the wicked and the like described above who returned to Jewish belief, whether in a public or private fashion, should be accepted by the community.
