Fallback to Offline Experiment

While there are times when you want to run an Experiment offline, more often than not you need an offline experiment because of issues with internet connectivity. Defaulting to OfflineExperiment now happens automatically should Comet detect prolonged issues with your connection. Your Experiment will keep logging data to a file, and if connectivity is restored while the Experiment is still running, the data will be uploaded to Comet automatically. If the experiments finishes before connectivity is restored, you will have a local zip file that you can upload to Comet.

In order to customize the directory that these zip files will write to in the case of offline fallback, set the COMET_OFFLINE_DIRECTORY environment variable. Otherwise, it will default to .cometml-runs.

At the end of your run, the Experiment summary provides instructions for uploading the run to the Comet UI.

COMET INFO: To upload this offline experiment, run:
    comet upload /content/.cometml-runs/54338792a0324a39bc4ce26ae148e97e.zip

To automate this upload process, you can also use the set_offline_zip_uploader method to specify a callback function that will be invoked to upload the offline ZIP archive created if a connectivity issue occurs.