In The Original Series (and Enterprise), Starfleet warp cores are horizontal:
Beginning with Star Trek: The Motion Picture and continuing through the films and later television series (e.g. The Next Generation), warp cores are vertical:
What advantage does a vertical warp core have over a horizontal one?
Out of universe, I would expect that the reason was that 'the rule of cool' kicked in - a tall, vertical warp core gives an imposing concept of power of great magnitude.
In universe, though, Memory Alpha explains the change:
In the 23rd century, the warp core was not situated in the main engineering. The main warp reaction occurred in a dilithium crystal converter assembly which consisted of two flattened rounded nodules situated directly in front of the warp plasma conduits to the warp engines, which were behind a large metal grate. (Star Trek: The Original Series; ENT: "In a Mirror, Darkly, Part II") By 2270, most Federation warp cores were redesigned to consist of a large warp core unit in the secondary hull with matter and antimatter channeling into the core through vertical conduits, with the resulting energy directed to the nacelles through a horizontal conduit leading out from the rear of the core. (Star Trek: The Motion Picture)
(Source)
I have bolded the important aspects of this excerpt. From it, we learn that there was a change in warp technologies in the Federation, or so we assume anyway. It seems that the warp core was re-designed into a larger warp core rather than two nodules, probably creating more power. This concept that it allowed for greater power is supported by the recallibration of the warp system. As per aramasi's answer to this question, we see the conversion:
OWF = Old Scale Warp Factor
NWF = New Scale Warp Factor
C = Speed in multiples of the speed of light
D/LY = Days per Light year
NWF: 1 2 3 4 5 6 7 8 9
C: 1 10.1 38.9 101.6 213.7 392.5 656.1 1024.0 1516.4
OWF: 1 2.2 3.4 4.7 6.0 7.3 8.7 10.1 11.5
D/LY: 365 36.2 9.4 3.6 1.7 0.9 0.6 0.4 0.2
Note that warp factor 8.7 on the old scale is about warp factor 7 on the new scale. Now consider that the Constitution class had a maximum safe speed of about warp 8 on the old scale (source) which is about warp 7 on the new system, and a cruising speed of warp 6, which is warp 5 on the new scale. Compare this then to the Enterprise after a refit with the vertical warp core: it had a new maximum scale of warp 12 (over warp 9 on the new scale) and a cruising speed of warp 8 (about warp 6.5 on the new scale) (source).
So, all this supports the idea that this new design had more power than the old horizontal design. But, it probably needed to be bigger and, considering a starship is far taller than it is wide, there's more room to install a longer warp core by installing it vertically than horizontally!
The Constitution class didn't really have a warp "core." It had 2 discrete intermix chambers, one vertical, one horizontal:
(Lower level of engineering shown in ST:TMP)
(Upper level view of engineering shown in ST VI)
The NX-class likely has a horizontal warp core because the ship is small and has few decks.
Later, larger starships have very long vertical warp cores that facilitate mechanical ejection in case of a warp core breach. This isn't possible with earlier designs. Running the core through multiple decks might also be a more convenient layout that having it occupy a ton of space on one or two decks.
It's unclear where the warp cores are located in most shuttles, but they're likely hidden either above or beneath the cabin. Some could very well be horizontally oriented or non-tubular.
So they could be ejected out of the bottom of the ship with the antimatter containment pods. If they wanted to eject horizontally it would either have to come out under the shuttle bay (where craft my also be trying an emergency escape), or through the deflector, which again isn't great
The Enterprise A, warp core is vertically integrated. The long pipe you're staring at is the main EPS conduit that splits into a Y shape and channels power to the warp nacelles. The NX class warp reactor is horizontal thanks to it's compact design, as ships got taller, cores got longer. Another reason would be that artificial gravity would allow the core to get dumped in the event of an emergency, it's easier to drop the core than to push it out from the rear.
