telemetry_node.connectors#

The connector module allows us to expose the data from the telemetry node to external services.

The module contains a file connector that saves the data to a json file and plots the current stats, a Grafana connector that exposes the data to a Grafana instance, and a Weights and Biases connector that sends the data to a Weights and Biases project.

class soulsai.distributed.server.telemetry_node.connectors.TelemetryConnector#

Abstract class for telemetry connectors.

Connectors are used to expose the telemetry data. This allows us to save the same telemetry data to the disk, sync it with a Grafana instance, and send it to a Weights and Biases project at the same time using a unified API.

start()#: Start the telemetry connector.

abstract update(data: dict)#

Update the data dictionary.

Parameters:: data – Data dictionary.

stop()#: Stop the telemetry connector.

class soulsai.distributed.server.telemetry_node.connectors.FileStorageConnector(config: dict)#

File storage connector to save the telemetry data to a json file and plot the current stats.

The connector saves the telemetry data and, if enabled, the current plots to files on the disk.

update(data: dict)#

Update the save files and the plot.

Parameters:: data – Data dictionary.

class soulsai.distributed.server.telemetry_node.connectors.GrafanaConnector(config: dict)#

Custom connector to expose the telemetry node as Grafana data source.

Grafana enables users to define custom data sources by setting up a server that responds to predefined queries. This class is a small wrapper around a flask server that allows the telemetry node to respond to Grafana queries using the current data.

start(host: str = '0.0.0.0', port: int = 80)#

Start the Grafana connector server.

Parameters:

host – Server address.
port – Server port.

update(data: dict)#

Update the data dictionary.

Parameters:: data – Data dictionary.

stop()#: Stop the Grafana server.

index() → str#

Index response.

Returns:: An OK string to enable the data source in Grafana.

search() → str#

Search response.

Returns:: The available data keys.

query() → str#

Query response.

In case we have more samples than the maximum requested data points, we evenly space the samples over the whole history.

Returns:: The data points for the requested target.

class soulsai.distributed.server.telemetry_node.connectors.WandBConnector(config: dict)#

Custom connector to send telemetry data to Weights and Biases.

Note

This connector requires the Weights and Biases API key to be stored in a file at /run/secrets/wandb_api_key. When running docker compose, secret files under config/secrets are mounted to /run/secrets in the container.

start()#: Start the telemetry connector.

update(data: dict)#

Upload the new stats to Weights and Biases.

Parameters:: data – Data dictionary.

stop()#: Stop the connector and finish the WandB run.

telemetry_node.connectors

Contents

telemetry_node.connectors#