client.watchdog

The watchdog module allows the execution of functions under special surveillance.

class soulsai.distributed.client.watchdog.ClientWatchdog(watched_fn: Callable, minimum_samples_per_minute: int, external_args: Any)

Watchdog to surveil the client sampling function.

The watchdog starts an observation thread that periodically checks if the client’s main script is still running as expected. If this is not the case, it restarts the script.

start()

Start the watchdog thread and execute the watched function.

Restart the function in case the watchdog thread determined a failure. If the function exits nominally without intervention from the watchdog thread, the watchdog shuts down.

class soulsai.distributed.client.watchdog.WatchdogGauge(sync_value: Synchronized, update_time: float = 60.0)

A simple gauge to measure the sample rate.

inc(amount: int = 1)

Increment the gauge by a given amount.

On each increment, the gauge checks if more than update_time seconds have passed since the last time update. If so, it updates the shared value with the current sample rate and resets the counter and timer.

Parameters:

amount – The amount to increment the gauge by.