I am going to answer this question for kabbalah in general sense it answers most of what you ask about the Arizal as well. Typically there are two ways to deal with kabbalistic teachings: to accept them as divinely inspired or as definitely complimenting what is written in Torah shebichtav (Written Torah; Pentateuch) and to deny them as being fallible teachings. I will address the prior method first:
Kabbalah, or mysticism, is not necessarily a foreign idea to Jewish thought. As an example, the ayin hara (evil eye) is discussed in context of Bereshit 49:22:
בֵּן פֹּרָת יוֹסֵף, בֵּן פֹּרָת עֲלֵי-עָיִן; בָּנוֹת, צָעֲדָה עֲלֵי-שׁוּר.
Yosef is a fruitful branch; a fruitful branch alei ayin by the fountain, its branches run over the wall.
The key phrase here is "alei ayin" which can be translated as "above the eye." The implication being that Joseph, as opposed to others, was immune to ayin hara. A reference to the (evil) eye is also seen in Bamidbar 24:2:
וַיִּשָּׂא בִלְעָם אֶת-עֵינָיו, וַיַּרְא אֶת-יִשְׂרָאֵל, שֹׁכֵן, לִשְׁבָטָיו; וַתְּהִי עָלָיו, רוּחַ אֱלֹהִים.
And Balaam lifted up his eyes, and he saw Israel dwelling tribe by tribe; and the spirit of God came upon him.
Balaam had previously been known for setting curses on the Emorite cities (Bamidbar 21:27), so the implication is that when he lifted his (evil) eye[s] towards Israel, God protected them from any curse or mystical consequence.
Kabbalists see the mystical and esoteric as being very real and as being clearly present within the Written Torah. So when they see kabbalistic works introduced, they do not immediately discard it because they accept that kabbalistic teachings and stories do exist within the Torah.
There are also plenty of references towards mystical concepts in Talmud Bavli like demons, curses, the composition of the soul, et cetera; but, I will forgo this to skip ahead to the most well known kabbalistic work, the Zohar. The narrative goes that the Zohar was composed by many rabbonim over the course of many centures, the most important being the Tanna, R' Shimon bar Yohai. To be more technically correct, kabbalists, for the most part, probably do not affirm the texts legitimate off any unbreakable chain of gedolim that preserved its teachings. It appears that they accept Zohar for some a posteriori reason. The reason itself is not nonsense necessarily. They accept Zohar as legitimate and important because it is clearly important to them today. It seems similar to how Yeshayahu Leibowitz treats halacha. He does not keep halacha because of Har Sinai, he keeps it lishma, for its own sake. I am probably being more confusing the clear, so I will stop here and just say that they have reasons. Later, the Zohar was formally published by Moses de Leon and then published for the Jewish world. This work would be the foundation for many of the Arizal's teachings and the Arizal would spend a large portion of his time to this work.
Now you ask the question, "...why do we not actually follow all of the teachings of the Arizal?" One explanation I've been given with halachic teachings in kabbalistic works it that they are really ad hoc explanations given after the fact. For example, the Zohar speaks about the importance of tzniut, in particular hair covering for women, and says something along the lines of, "[Some] women shave their heads so that not even a single hair shows through their wigs/hair coverings because she will then have good children. But as my friend explained, the Zohar might have just had this explanation to justify the practice in the first place. Or, in other words, the practice was never done for that reason, but that reason was given to justify that practice.
Obviously, I do not think most kabbalists will agree with that statement, they will say that the Zohar and its teachings did not come after the practice but was born at Har Sinai. So, typically, if I am not mistaken, the reason why many halachic sayings are not kept by many kabbalists is because it is an issue of being machmir. Not everyone can be super frum and be machmir on all the minhagim. For example, there is a minhag within Chabad to abstain from sugar on Pesah because the Rebbe asked for a brick of sugar on the holiday and when his students fetched it for him, he broke open the brick and found a single wheat grain in it. Do all Chabadniks keep this minhag? Of course not and, in fact, I actually know a family where the father and mother do not keep the minhag but their young son does. Not everyone can be as machmir as the Rebbe or as Shimon bar Yohai. They are not breaking a halacha, but there is always room to improve and serve out God's law better.
Concerning your other question as to whether or not Torah changed, there are many that argue that the Torah we have today is different from Torah from Har Sinai because of scribal errors (I think there are like 7 differences between Sephardi and Ashkenazi Humashim), but these do not change the meaning of the text all that much. The reason why many Kabbalists can accept what Torah says as true while also accepting a somewhat contradictory kabbalistic teaching is because, in part, due to a midrash Bamidbar Rabbah 13:15:
Torah is learned through 70 faces.
The classical understanding of this is that there are many different ways to express a truth in Torah and that even if they seem to conflict, they actually may not and the problem may be that we are stuck in a mindset of thinking they must conflict. This is sort of similar to how we treat Braitot that seemingly conflict with mishnayot, Gemara typically first attempts to reconcile the two.
I will answer the rest of your question as a catch-all before continuing on with my answer of those that deny kabbalah.
You ask, "Now we need to do these things, but even the greatest of men from previous generations did not?" Kabbalists may say that they actually did kept them, but that keeping of the mystical tradition is never mentioned. This is similar to how we hear midrashim explaining how Jacob never lied and that when he said "Ani Esav Bechorecha" he was actually saying "Ani Yaakov, ve'Esav Bechorecha," along with other midrashim explaining how Avraham fed his guests milk products first and then meat (as to avoid the prohibition of eating meat and dairy in the first meal). Kabbalists are not defeated in this regard. They can foster many explanations that support their belief that hazal and other gedolim kept these mystical teachings without their keeping ever being mentioned.
Now to explain those that deny kabbalah. Those that deny mystical teachings will first and foremost say that there are mystical aspects to Torah but that they are hidden from us and that we cannot lust after it. In Devarim 29:28 it states:
הַנִּסְתָּרֹת--לַיהוָה, אֱלֹהֵינוּ; וְהַנִּגְלֹת לָנוּ וּלְבָנֵינוּ, עַד-עוֹלָם--לַעֲשׂוֹת, אֶת-כָּל-דִּבְרֵי הַתּוֹרָה הַזֹּאת.
The secret things belong unto the LORD our God; but the things that are revealed belong unto us and to our children for ever, that we may do all the words of this law.
The verse here states that knowledge that we can arrive at and are not secret, we can do with as we wish. However, the things which are esoteric, secret, mystical, et cetera are foreign to us and belong to God. It also states in Devarim 18:14:
כִּי הַגּוֹיִם הָאֵלֶּה, אֲשֶׁר אַתָּה יוֹרֵשׁ אוֹתָם--אֶל-מְעֹנְנִים וְאֶל-קֹסְמִים, יִשְׁמָעוּ; וְאַתָּה--לֹא כֵן, נָתַן לְךָ יְהוָה אֱלֹהֶיךָ.
These nations which you are driving out listen to astrologers and diviners. This is not [what God has granted] you.
For those that reject kabbalah, the general idea is that we should stay away from mystical teachings because they will distract us and lead us to idolatry. Which is a main position for many people that reject kabbalah. Rambamists, students of R' Jose Faur, those that follow Yeshayahu Leibowitz, "Rational Jews", Dor Deah (the Yemenites that rejected the Zohar and other kabbalistic works), et cetera hold that many teachings in Zohar and kabbalah are idolatrous in nature. The obvious ones to note are ayin hara, kaparot, sephirot, certain explanations of verses in Torah (e.g. that we should read Bereshit 1:1 word by word so that "Bereshit raba Elokim" literally means "In the beginning created God," implying that God was created.), so on and so forth.
Those that reject kabbalah will say that we do not find these teachings in any detail in Torah or anywhere in Jewish cannon, that we do not see them in Gemara, and that major kabbalistic works like the Zohar were written in the 12th century by Moses de Leon himself and that ever since his work was published, more and more anonymous rabbis added onto the text. For these individuals, kabbalistic works are not midrashim and they are certainly not Torah. They are made up, brought by foreign sources, and should ultimately be rejected because they encourage idolatrous thinking. So they reject kabbalah and avoid many of your questions by stating that, "We have Torah, we have halacha, why do we need these mystical teachings that can only confuse Jewish life and ideology?"
