Federated learning FAQ

An FL client can join the FL training any time. As long as the participating FL clients are still within the maximum number of clients, the client can join. The newly joined client will get the current round of global model for training and will contribute to the current global model.

No, federated learning training does not require for FL clients to open their network for inbound traffic. The server never sends uninvited requests to clients but only responds to client requests.

Yes, the administrator command start_mgpu client <gpu number> <client name> can be used to start training on FL clients with different numbers of GPUs when starting training. It is possible for different clients to be training with different numbers of GPUs. Note that training on the same client may not work if docker containers are started with the flag –net=host.

The federated learning clients are identified by a dynamically generated FL token issued by the server during runtime. When an FL client first joins an FL training, it first needs to send a login request to the FL server. During the login process, the FL server and client need to exchange SSL certificates for bi-directional authentication. Once the authentication is successful, the FL server sends an FL token to the client. The FL client will use this FL token to identify itself for all following requests for the global model and all model updating operations.

Yes. The FL clients are identified by FL token, not machine IP. Each FL client will have its own FL token as well as instance name, which is the client name that must be used for issuing specific commands to that client.

Yes, you can start multiple instances of FL clients from the same client packages. Each FL client will be identified by its unique instance names, for example: “flclient1”, “flclient1_1”, “flclient1_2”, etc. The instance name must be used for issuing specific commands to that client from the admin tool.

Federated learning clients will send a heartbeat call to the FL server once every minute. If an FL client crashes and the FL server does not get a heartbeat from that client for 10 minutes (can be set with “heart_beat_timeout” in the server’s config json), the FL server will remove that client from the training client list.

Yes, an FL client can join or quit in the middle of the FL training at any time. The client will pick up the global model at the current round of the server to participate in the FL training. When quitting, the FL server will automatically remove the FL client after it quits and no heartbeat is received for the duration of the “heart_beat_timeout” configured on the server. If using an admin tool, it is recommended to use the “abort” and “shutdown” commands to gracefully stop the clients.

When an FL client passes authentication, it can request the current round of the global model and starts the FL training right away. There is no need to wait for other clients. Once the client finishes its own training, it will send the update to the server for aggregation. However, if the server does not receive enough updates from other clients, the FL server will not start the next round of FL training. The finished FL client will be waiting for the next round’s model.

The FL server begins model aggregation after accepting updates from the minimum number of FL clients required and waiting for “wait_after_min_clients” configured on the server. The updates that are received after this will be discarded. All the clients will get the next round of the global model to start the next round FL training.

The FL client always asks the server for the current round of training. If the server is not ready, the FL client will wait. The client will only stop if the server becomes unreachable. The FL client can also be killed with the admin tool issuing a “shutdown” command or by ctrl-C on the client itself, although this is not recommended because the server will wait for the duration of “heart_beat_timeout” before knowing that the client has stopped.

There are two scenarios for the FL server crashing during the FL training. If the server crashes when the FL client is trying to connect to the server for model exchange, the FL client will continue to attempt connecting to the server for up to 30 seconds. If the server is still down after that, FL client will shut itself down. If the server crashes during the FL client model training, as long as the server restarts before the FL client attempts model updating, it will have no impact to the FL clients.

When restarting the FL server, you can find the previous training round number from the previous log. Then you can choose to train from scratch or continuously using previous training model.

No, there is no need to have GPU on the server side for the FL server to deploy. However, certain handlers may require GPUs. To disable GPUs on the server, include the following in the shell script that runs the server:

            

            
export CUDA_VISIBLE_DEVICES=
        

Depending on the configuration of config_fed_server.json which controls which port the gRPC is deployed to, the FL server network needs to open that port for outside clients to reach the FL server. For example, with the following configurations, the AWS instance needs to open 8002 port for TCP traffic to reach the AWS instance:

            

            
"service": {
    "target": "ec2-3-81-120-62.compute-1.amazonaws.com:8002",
    "options": [
        ["grpc.max_send_message_length",    1000000000],
        ["grpc.max_receive_message_length", 1000000000]
    ]
}
        

Currently, federated learning does not support load balancing between multiple FL servers.

Use the MMAR_CKPT option in server_train.sh to continue FL training using a pre-trained model.

The FL server always runs from the “start_round” to “num_rounds”. The FL server will stop the training when the current round meets “num_rounds”.

As in Clara Train, for running classification training, DATASET_JSON must be specified in an environment file the FL Server MMAR has access to, and specifically, the “label_format” in the classification Datalist JSON file must be configured properly according to the data.

Yes, the steps to set up the AWS FL server instance are:

• Set up an AWS instance with a minimum of t2.2xlarge instance type

• run:

  Copy

  Copied!

              
  
              
  docker login nvcr.io
  
  docker pull nvcr.io/nvidian/dlmed/clara-train-sdk:v3.1-EA1
          

• Set up the ROOT certificate and the server SSL certificate (make sure to use the AWS instance name to create the SSL server certificate, e.g, ec2-3-99-123-456.compute-1.amazonaws.com)

• Create the client certificate using the ROOT certificate to sign

• Configure config_fed_server.json and config_fed_client.json properly

• Start the server and clients

For more information on running Clara on a Cloud Service Provider, see Clara Train - Getting started with a Cloud Service Provider.

You can edit the MMAR for each individual client on your desktop, then upload and deploy to each individual client with the admin tool. Each client can run with its own MMAR configuration.

Yes, you can re-use the same “run_number” as previously used. The “run_number” serves as an FL training workspace. The FL training logs, such as tensorboard, training stats, etc, are stored within the same “run_number” workspace.

The admin can issue a command to abort the FL client training: abort client client_name. If the command is issued without the client_name, then the command will be sent to all the clients. Because of the nature of model training, it may take a little time for the FL client to completely stop. Use the “check_status client client_name” command to see if the client status is “stopped”.

The admin commands to the clients pass through the server. If for some reason the command is delayed by the network, or if the client command takes a long time to process, the admin console will experience a delay for the response. The default timeout is 10 seconds. You can use the “set_timeout” command to adjust the command timeout. If this timeout value is set too low, the admin command may not reach the client to execute the command.

Yes, you can upload different MMARs to the server and clients to train different models. Make sure to use the “run_number” to keep your trained models in different run spaces without confusing the models. The FL system only completes when the admin issues the “shutdown” command or ctrl-C is used to end the process.

If you want to change the code you have already loaded in a custom component, it is recommended that you add a version number or change the class name slightly. Python does not load new code definitions with the same class name and by default Python does not allow the loaded modules to be removed. With a version or altered name, Python will be able to treat the code as new and load it from the sys.path.

Sometimes if you are trying to check status of the client and the server is already busy transferring the model and does not have extra bandwidth for the command, the command may time out. In that case, please wait and try again.

Cross site validation is opt-out by default. To opt in, you must set “cross_validate”: true in config_fed_client.json and configure the cross_site_validation settings.

config_validation_ckpt.json is the config used for validating models on our data. The MMAR requires to have this file available.

Use the admin commands “validate all” or “validate <client_1> <client2>” to retreive the results.

If some client is not participating in cross_site_validation OR an error occurs during validation, you will see empty results for that section. Please use the logs to retreive the error.

In some cases, cross site validation may get stuck. This is because the server sometimes doesn’t know when (or if ever) a model will become available. In these cases, please use the “abort <client>” to stop cross site validation manually.

This is the intended behavior. If a client is aborted, it will transition to cross site validation phase (if participating). To completely abort, call “abort client” command again.