I don't think @Shimon bM's comment that order doesn't matter is justified. The Rishonim speak at length about the reasons for the order of the Masechtos (see Tosfos beginning of Bava Metzia and Avodah Zarah for example, and the Rambam's intro to the Mishnah etc. etc.). The Gemara even discusses whether the order bears Halachic repercussions (Bava Kama 102a, Avodah Zarah 7a).
@DoubleAA is correct that the real question is why the printers of the Shas changed the order, since it is clear that the 'true' order is the order of the Mishnah (& Tosefta and Rambam etc.). His suggestion that it had to do with size considerations is hard to swallow. RH and Beitzah have the same number of Dapim, why should printers have preferred grouping Yoma-Sukkah-Beitzah to grouping Rosh Hashanah-Yoma-Sukkah? It seems more likely that the theme of Tishrei Chagim seemed more logical for them (and for those who want to study the current holiday and will find it easier to find all of Tishrei in a single vol.)
The big question is still what determined the 'true' order of the Masechtos, and why Beitzah precedes Rosh Hashanah. I saw a fascinating reference regarding the 'true' order (i.e. which is more correct) of Masechtos at https://dafyomi.co.il/yoma/yo-in-biblio.htm . (It is worth becoming familiar with that site - there are answers and information there for everything related to Talmud and Mishnah.) In brief, they cite Rav R. Margoliyos (Yesod ha'Mishnah va'Arichasah) who points out that in every Seder, the order of the Masechtos is from larger to smaller - not in number of Dapim, but in number of Perakim. If so, it follows that the 'true' order must be Yoma, Sukah, Beitzah, RH (8,5,5,4 Perakim)!
This theme follows through the entire Shas with a single exception - the beginning of Zeraim (where the Masechtos reduce in size from Berachos to PEah to Demai (9,8,7), but jump up with Kilayim Shevi'is Terumos (9,10,11) before returning to the normal pattern with Ma'asros etc. (5,4,4,3,3). Since no simple answer has yet been suggested for that anomaly, the Rambam is justified in suggested his (sometimes forced) reasons for why each Maseches follows the previous one!