Micro-Threads (multi Core) - Background

Background

As microprocessors are becoming faster, mainly because of the cores being added every few months, memory latency gap is becoming wider. Memory latency was few cycles in 1980 and it is reaching nowadays almost 1000 cycles. If the micro-processor has enough cores and hopefully they are not sending requests to the main memory at the same time, there will be partial aggregate hiding of memory latency. Some cores might be executing while others are waiting for memory response. This is not the best situation for multi-core processors. High performance computing experts are striving to keep all cores busy all the time. So, if each core is kept busy all the time, a complete utilization of the whole micro-processor is possible. Creating software based threads won't solve the problem for one obvious reason. Context switching threads to main memory is much expensive operation when compared to memory latency. For example, in Cell Broadband Engine context switching any of the core's thread takes 2000 micro-seconds in best cases. Some software techniques like double or multi-buffering may solve the memory latency problem. However, they can be used in regular algorithms, where the program knows where is the next data chunk to retrieve from memory; in this case it sends request to memory while it is processing previously request data. However, this technique won't work if it the program does not know the next data chunk to retrieve from memory. In other words, it won't work in combinatorial algorithms, such as tree spanning or random list ranking. In addition, multi-buffering assumes that memory latency is constant and can be hidden by statically. However, reality shows that memory latency changes from application to another. It depends on the overall load on microprocessor's shared resources, such as the rate of memory requests shared cores interconnections.