module ChefUtils::ParallelMap
This module contains ruby refinements that adds several methods to the Enumerable class which are useful for parallel processing.
Public Instance Methods
The flat_each
method is tightly coupled to the usage of parallel_map
within the ChefFS implementation. It is not itself a parallel method, but it is used to iterate through the 2nd level of nested structure, which is tied to the nested structures that ChefFS returns.
This is different from Enumerable#flat_map because that behaves like map.flatten(1) while this behaves more like flatten(1).each. We need this on an Enumerable, so we have no Enumerable#flatten method to call.
- [ 1, 2 ], [ 3, 4
-
].flat_each(&block) calls block four times with 1, 2, 3, 4
- [ 1, 2 ], [ 3, 4
-
].flat_map(&block) calls block twice with [1, 2] and [3,4]
# File lib/chef-utils/parallel_map.rb, line 82 def flat_each(&block) map do |value| if value.is_a?(Enumerable) value.each(&block) else yield value end end end
This has the same behavior as parallel_map
but returns the enumerator instead of the return values.
@return [Enumerable] the enumerable for method chaining
# File lib/chef-utils/parallel_map.rb, line 61 def parallel_each(pool: nil, &block) return self unless block_given? parallel_map(pool: pool, &block) self end
Enumerates through the collection in parallel using the thread pool provided or the default thread pool. By using the default thread pool this supports recursively calling the method without deadlocking while using a globally fixed number of workers. This method supports lazy collections. It returns synchronously, waiting until all the work is done. Failures are only reported after the collection has executed and only the first exception is raised.
(0..).lazy.parallel_map { |i| i*i }.first(5)
@return [Array] output results
# File lib/chef-utils/parallel_map.rb, line 42 def parallel_map(pool: nil) return self unless block_given? pool ||= ChefUtils::DefaultThreadPool.instance.pool futures = map do |item| Concurrent::Future.execute(executor: pool) do yield item end end futures.map(&:value!) end