11

Parallel computation is frequently modeled using a deterministic local rate of computation, latency overhead, and network bandwidth. In reality, these are spatially variable and non-deterministic. Studies such as Skinner and Kramer (2005) observe multi-modal distributions, but performance analysis seems to always use either deterministic or Gaussian distributions (not just inaccurate, it's inconsistent due to positive probability of negative latency).

Have higher-fidelity statistical models been developed? Do any account for cross-correlation in local compute/memory, latency, and bandwidth variability?

Jed Brown
  • 25,650
  • 3
  • 72
  • 130

1 Answers1

3

From the Computer Science perspective I do not think that make sense to make an general statistical model for memory access time (latency) and memory bandwidth.

It does make sense to create an statistical model for a algorithm. That is because each algorithm has a specific memory access pattern, the memory access patterns are relevant to the cache hierarchy, e.g. a algorithm with high data locality will take advantage from low level caches benefiting of really fast memory access times while other algorithms will have to go all the way to the RAM (or even worst the swap memory) and have extremely slow access times.

The general purpose values are given from the architecture point of view, you can check your architecture and search the access time from a given core to a given memory location (let's say L3 cache). Be aware that the recent architectures are Non Uniform Memory Access NUMA which will make your job a bit harder.

RSFalcon7
  • 159
  • 8