Parallel Processing Access Patterns

Parallel processing refers to the structure when a single program is running by multiple CPU cores at the same time. Depending on what the program does and how it works, this could lead to massive gains in speed. Especially if computation tasks could be separated & executed in not blocking way. *(like with java `ForkJoinPool` approach)* Writing the programs to run in parallel requires different thought processes than sequential programming.The first thing to do is to figure out what exactly can be separated & parallelized. If 1st part of the program depends on the results of 2nd part, then the first part should guarantee *happens-before* relation to the 2nd one. On the other hand, most programs have large sections that can run without reference to one another and then their outputs can be combined. <a name='more'></a> ## Access Patterns There few common approaches how to deal with the case, when multiple processors need to access the same piece of data. ---- ### Exclusive Read Exclusive Write (EREW) **Only One Processor can access a given memory location in the same time.** This is the simplest solution and obviously the safest and the slowest one. ---- ### Concurrent Read Exclusive Write (CREW) **Any Number of Processors can read the given memory location simultaneously, but only one processor can write to location at the same time.** This is quite convenient and safe solution to prevent interference, cause resource will be locked during the write. *(like with java ` ReentrantReadWriteLock` class)* ---- ### Concurrent Read Concurrent Write (CRCW) **Any Number of Processors can read and write the given memory location simultaneously.** This is the fastest way, but at the same time it requires additional attention & careful programming to achieve consistent results. ---- ## Maximizing Parallelism Efficiency Three part optimization challenge: * **Minimize the total number of computations** * **Minimize the length of non parallelizable sections** * **Minimize the overhead of processor-processor communication** So, to deal with it properly you have to: * **Find the parallelism**. Separate the independent parts that can act simultaneously. * **Load balancing**. Ideally all available processors should be utilized at all the time with no waiting. *Note: The increasing number of threads itself doesn't mean the performance increasing. The amount of utilized processors does...* ---- ## see Also * [Oracle Tuts - The try-with-resources Statement](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) * [Keynote: Exceptions in Action](http://bloomlab.blogspot.com/2015/01/keynote-exceptions-in-action.html) * [Keynote: Understand Java Collections](http://bloomlab.blogspot.com/2015/05/keynote-understand-java-collections.html) * [Keynote: Threads like trains:)](http://bloomlab.blogspot.com/2015/05/keynote-threads-like-trains.html) * [Keynote: Shadowing (Hiding fields)](http://bloomlab.blogspot.com/2015/01/keynote-shadowing-hiding-fields.html) * [Keynote: Initialization Order flow](http://bloomlab.blogspot.com/2014/12/keynote-initialization-order-flow.html)