How to Use Shared Variables in Scipy's Differential Evolution with Multiple Workers

Learn how to pass shared variables in Scipy's differential evolution function when using multiple workers, and track experiment numbers effectively.
---
This video is based on the question https://stackoverflow.com/q/74768283/ asked by the user 'First Python' ( https://stackoverflow.com/u/20754491/ ) and on the answer https://stackoverflow.com/a/74768473/ provided by the user 'Majid' ( https://stackoverflow.com/u/1668003/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: How to pass shared variable in scipy differential evolution with worker 2

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Use Shared Variables in Scipy's Differential Evolution with Multiple Workers

In the realm of optimization, the scipy.optimize library in Python offers various powerful tools, one of which is the differential_evolution function. However, if you ever find yourself needing to keep track of shared variables, such as an experiment counter with multiple workers, you might run into some challenges.

The Problem

When using differential_evolution, many users typically start with a single worker, and things work perfectly fine. For example:

[[See Video to Reveal this Text or Code Snippet]]

But increasing the number of workers can cause the shared variable (the experiment counter in this case) to miscount and produce inconsistent results. For instance, if you set workers=2, you might see outputs that all show the experiment number as 1, rather than incrementing correctly across multiple threads.

Example Output with Workers 1:

[[See Video to Reveal this Text or Code Snippet]]

Example Output with Workers 2 or 3:

[[See Video to Reveal this Text or Code Snippet]]

As you can see, the output is not as expected with multiple workers.

The Solution

To solve this problem, we can utilize thread-safe mechanisms provided by the multiprocessing module in Python. Specifically, we will implement a Counter class that uses a RawValue to maintain a shared integer and a Lock to ensure that increments to that variable are synchronized.

Step-by-Step Implementation

Import Required Libraries:
Ensure that you import the necessary libraries for your code.

[[See Video to Reveal this Text or Code Snippet]]

Create a Thread-Safe Counter:
We define a Counter class that encapsulates a RawValue integer and a Lock.

[[See Video to Reveal this Text or Code Snippet]]

Manage Experiment Info:
We can create a second class, ExpermientInfo, that utilizes our Counter.

[[See Video to Reveal this Text or Code Snippet]]

Define Your Objective Function:
This function will now call the increment method on each call to track the experiment number.

[[See Video to Reveal this Text or Code Snippet]]

Set Up Your Differential Evolution Call:
Now you can call differential_evolution using multiple workers.

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

By incorporating a thread-safe counter, you can efficiently pass and track shared variables in Scipy's differential_evolution function, even when utilizing multiple worker threads. This method not only enhances the accuracy of experiment tracking but also promotes safer concurrent programming practices.

Now you can confidently proceed with your optimization tasks without the worry of losing track of your experiment numbers. Happy coding!