Repository navigation
More user-friendly errors and automatic restarts in case of engines crashing due to OOM #695
Description
Activity
I think it's hard to do this in general such that it fits in the base class, but Launchers have two relevant methods:
_log_outputwhich is called on stop. This is what logs the mpi errors. You can override this in your custom Launcher to do further processing/parsing of the output to change what's logged by default instead of or in addition to the current MPI output- Launcher.on_stop allows registering arbitrary stop callbacks. example notebook.
If you already have a custom launcher, you can combine these to add
self.on_stop(self.custom_log_message)at the end of.start()to always add your own custom stop handlers.Sorry, missed that part. Automatic restart could possibly also be achieved through the on_stop callback. The question becomes whether it makes sense to restart the same engine set vs starting a new one. Restarting in-place would probably feel cleaner, but likely would also make debugging more challenging (e.g. losing handles on the logs for the crashed engines). Starting a new engine set is simpler, because you only need to call
cluster.start_engines(n).I think it's reasonable for restart-on-fail to be a built-in feature for Engine[Set]Launcher, but it should be possible now via on_stop.
The errors we report in case of OOM and Segmentation-Fault are now much better, but I was wondering is there a way to make them more "user-friendly"?