telemetry_node.telemetry_node

The main telemetry node module.

class soulsai.distributed.server.telemetry_node.telemetry_node.TelemetryNode(config: SimpleNamespace)

The telemetry node receives telemetry from the training node to track the training progress.

Samples are received from Redis and added to the telemetry history. Every update_interval message, the results are plotted and saved both to a figure and to a json file.

If live monitoring is enabled, the node also uses a GrafanaConnector to expose the data to a Grafana instance running within the same network.

run()

Run the telemetry node.

Receives telemetry messages from clients via Redis and adds them to the training statistics. The statistic lists are shared by reference with the GrafanaConnector (if enabled). An update of the lists in the main loop therefore automatically updates the stats for the GrafanaConnector as well.

Note

Telemetry messages and therefore the stats are not guaranteed to be in order.

shutdown(_)

Shut down the telemetry node.