This is called Al-Alif Al-Qaseera (الألف القصيرة) that goes on top of Al-Alif Al-Maqsoora (الألف المقصورة) — which is drawn looking like a ya' (ى) — to show that the pronunciation of the ya' is like that of alif.
Traditionally, علىٰ (to mean "on") used to be written as علا (to mean "exalt" as in Qur'an 28:4):

This caused confusion in distinguishing the meaning of the words, so it was replaced by Al-Alif Al-Maqsoora (على). But then, this introduced a problem of how to distinguish the pronunciation.
To distinguish the pronunciation, dots were used on top of or below letters to denote the different ligatures, as shown in the document below from a mushaf from North Africa. This way of writing ligatures has been partially re-adopted for ya' (ي) in the modern times, with two dots under it to replace kasra (the only tradition I am aware of that continue till today from this way of writing).

Later on, Al-Alif Al-Qaseera was adopted to distinguish the pronunciation.
Note this Al-Alif Al-Qaseera does not get added when the following letter is sakin as in this verse for موسى:

whereas, if the letter following is not sakin, then موسى will adopt Al-Alif Al-Qaseera on top:

As you may have noticed, that Al-Alif Al-Qaseera is also used to replace (badal) the actual alif in a lot of words, typically for beautification of scripture, and is only used this way in scribing the Qur'an (which has different rules in scribing).
In addition, it may be used to differentiate different types of recitation as in the following example, which can be recited as malik or maalik:

For further information on this topic, you may refer to books on the topic (e.g., رسم المصحف دراسة لغوية تاريخية).