I think you’re on the right track, but you’re missing a few points. Allow me to quote the entire passage, v. 1-7, in the original Hebrew:
ה֚וֹי הַשַּׁאֲנַנִּ֣ים בְּצִיּ֔וֹן וְהַבֹּטְחִ֖ים בְּהַ֣ר שֹׁמְר֑וֹן נְקֻבֵי֙ רֵאשִׁ֣ית הַגּוֹיִ֔ם וּבָ֥אוּ לָהֶ֖ם בֵּ֥ית יִשְׂרָאֵֽל׃ עִבְר֤וּ כַֽלְנֵה֙ וּרְא֔וּ וּלְכ֥וּ מִשָּׁ֖ם חֲמַ֣ת רַבָּ֑ה וּרְד֣וּ גַת־פְּלִשְׁתּ֗ים הֲטוֹבִים֙ מִן־הַמַּמְלָכ֣וֹת הָאֵ֔לֶּה אִם־רַ֥ב גְּבוּלָ֖ם מִגְּבֻלְכֶֽם׃ הַֽמְנַדִּ֖ים לְי֣וֹם רָ֑ע וַתַּגִּישׁ֖וּן שֶׁ֥בֶת חָמָֽס׃ הַשֹּֽׁכְבִים֙ עַל־מִטּ֣וֹת שֵׁ֔ן וּסְרֻחִ֖ים עַל־עַרְשׂוֹתָ֑ם וְאֹכְלִ֤ים כָּרִים֙ מִצֹּ֔אן וַעֲגָלִ֖ים מִתּ֥וֹךְ מַרְבֵּֽק׃ הַפֹּרְטִ֖ים עַל־פִּ֣י הַנָּ֑בֶל כְּדָוִ֕יד חָשְׁב֥וּ לָהֶ֖ם כְּלֵי־שִֽׁיר׃ הַשֹּׁתִ֤ים בְּמִזְרְקֵי֙ יַ֔יִן וְרֵאשִׁ֥ית שְׁמָנִ֖ים יִמְשָׁ֑חוּ וְלֹ֥א נֶחְל֖וּ עַל־שֵׁ֥בֶר יוֹסֵֽף׃ לָכֵ֛ן עַתָּ֥ה יִגְל֖וּ בְּרֹ֣אשׁ גֹּלִ֑ים וְסָ֖ר מִרְזַ֥ח סְרוּחִֽים׃
Ho, those who are at ease in Zion, and those who are confident on Mount Shomron, you notables of the head of the nations, that the House of Israel comes to them. Cross over Calneh and see, go from there to Greater Hamas, and go down to Gath of the Philistines. Are you better than these kingdoms? Is your border larger than their border? You ward off a day of bad and convene in a session of lawlessness. They lie on ivory beds, lolling on their couches, feasting on lambs of the flock and cows of the stall. They hum snatches of the song, to the tune of the lute; they think themselves musicians like David. They drink from the wine bowl and anoint themselves with oil, but they are not concerned with the ruin of Joseph. Therefore, shortly they will be exiled at the head of the exiles; they shall no more loll at festive meals.
The problem with your understanding of the passage is that it’s point has nothing to do with abstinence. Joseph here (as it often does throughout the Prophets) is a placeholder for the Jewish people at large, particularly the northern Ten Tribes. They were sinning - in particular, the prophet here warns them about assimilation - and, rather than recognizing they were doing anything on, instead relaxed and rejoiced. We’re meant to be different, and we’re not supposed to be like all the nations, so why do these people assimilate?
Targum Yonason translates the third clause in v. 1 as:
מְקַיְמִין שׁוּם בְּנֵיהוֹן כְּשׁוּם בְּנֵי עַמְמַיָא וְכָל דְמִן דְיִסְתַּמְכוּן עֲלֵיהוֹן בֵּית יִשְׂרָאֵל:
who set the names of their children like the names of the children of the nations
And as Rashi comments on v. 2:
עברו כלנה וראו. למה מאסתם בתורתי לדבק בחוקות עכו"ם עברו וראו בעכו"ם החשובים אם טובים הם מן הממלכות האלה שבשומרון ושבירושלים:
Why have you rejected My Torah, to cling with the ways of the idolaters? Go and see in the notable idolatrous [nations] if they’re any better than these kingdoms in Shomron and Jerusalem.
All of these nations mentioned in v. 2 were the largest powers in their times, yet ultimately they too fell. Why do you seek to emulate them? Babylon (here named Calneh) came and went. Antioch (here named Hamas) came and went. Gath came and went. Yet we still stand. Why would you seek to join them?
Yet you close your eyes to the truth and indulge in materialism. In v. 4 they stretch out on ivory beds - Rashi quotes our Sages that this is a euphemism for wife-swapping, a Biblical prohibition (Deuteronomy 24:4). In v. 5 they proclaim that their song, sung for pure enjoyment, is on the level of David’s hymns, crafted for the sake of Heaven.
So the issue at play here is not that they’re indulging in materialism. Whether that’s an issue or not depends entirely on the circumstances and is a topic that will have to wait for its own day. The problem with the people discussed in the Amos passage is not their indulgence in materialism, but their indulgence in actual sins - assimilation, which led to wife-swapping as well.