Skip to content

ipyparallel and pymoo doesn't work #848

Description

@patdub

Hello all,
I am trying to use ipyparallel with a module for Genetic Algorithm (pymoo) and for that, I need to pass the class name 'MyProblem' to the clusters (don't know really why clusters are not aware but I have also the same problem with function defined in the main program). When I pass the name using 'dview["MyProblem"] = MyProblem' I am facing a recursing error.

This is the simplified code below which reproduce the error. Do you have any idea ?

import numpy as np
import ipyparallel as ipp
from pymoo.algorithms.moo.nsga2 import NSGA2
from pymoo.core.problem import ElementwiseProblem
from pymoo.optimize import minimize

# Very simplified problem
class MyProblem(ElementwiseProblem):

    def __init__(self):
        super().__init__(n_var = 1,
                         n_obj = 1,
                         n_constr = 0,
                         xl=np.array([0]),
                         xu=np.array([1]))

    def _evaluate(self, x, out, *args, **kwargs):
        
        # Fitness functions (values to minimize)
        f1 = -x[0]
        out["F"] = [f1] 
        
# Start the clusters
nb_core = 16
rc = ipp.Cluster(n=nb_core).start_and_connect_sync()
rc.wait_for_engines(n=nb_core)
dview = rc[:]

# Send class def to all cluster for further use
dview["MyProblem"] = MyProblem

Thanks in advance for your help.
Patrick.

Activity

  1. prodhype commented on Dec 24, 2023

    @prodhype

    This seems related to child classes that call super() and appears to be a bug in the pickling/canning process.

    import ipyparallel as ipp
    
    class A:
        def __init__(self):
            self.x = 1
    
    class B(A):
        def __init__(self):
            pass
    
    class C(A):
        def __init__(self):
            super().__init__()
    
            
    # Start the clusters
    rc = ipp.Cluster(n=2).start_and_connect_sync()
    rc.wait_for_engines(n=2)
    dview = rc[:]
    
    dview["A"] = A
    dview["B"] = B
    dview["C"] = C  # <-- RecursionError: maximum recursion depth exceeded
  2. minrk commented on Jan 5, 2024

    @minrk
    Member

    You might try dview.use_cloudpickle() if you have more complex local objects to push. Otherwise, it's often more robust to just define classes remotely instead of pushign them if you can, e.g. via view.run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions