Comet releases¶

The following is a history of released Comet versions. It does not list everything that was changed in a release, but does mention the highlights and all public-facing additions, changes, and deprecations.

December 2023 release¶

Backend Version: 3.7.111

Frontend Version: 5.62.5

Optimizer: 2.0.22

Added

• Experiment Management: Added a search panels option in the Project Panels page. In addition, it is now possible to update the x-axis and chart smoothing options for all panels through the general dashboard settings.
• Experiment Management: Grouping of experiments is now supported in bar charts. Note: Only the experiments in scope (ie. in the sidebar) are used to determine metrics per group.
• Experiment Management: Metadata logged alongside an image is now displayed in the UI.
• Experiment Management: It is now possible to move experiments to a different workspace.
• Experiment Management: Add endpoints to get a model's version history.
• Experiment Management: Network system metrics are now displayed in the system metrics tab of the single experiment page.
• Experiment Management: If a video is logged using log_asset method in the SDK, it will now be marked as a video.
• LLM: Prompts can now be deleted through the UI.
• LLM: Averages of the duration and user_feedback columns are now displayed when prompts are grouped.
• MPM: Segments are now supported in the Number of Predictions chart in the Model Performance tab.
• MPM: Added a toggle to view either the percentage of predictions or the total number of predictions in each category for categorical distributions.
• MPM: Added more actions in chart tooltips.
• MPM: It is now possible to validate events synchronously instead of asynchronously by specifying the parameter ?enforceValidation=True to the mpm/events and mpm/events/batch endpoints.
• MPM: Alert descriptions are now added to the alert email.
• MPM: The variable nb_predictions can now be used in custom metrics.

Changed

• Experiment Management: The experiment Name no longer has a max width limit.
• Experiment Management: Images are now downloaded from S3 using a pre-signed URL.
• Experiment Management: Fixed mis-match between number of users in a workspace and the list of members in a workspace due to the way deleted users are counted in the chargeback report.
• Experiment Management: If an experiment that has crashed is resumed, it will no longer be marked as crashed.
• MPM: Removed the Total Number of Predictions section in each chart.
• Model Registry: Delete button in the model registry is now displayed for workspace members, only workspace owners can delete models in the model registry.
• Model Registry: Disabled the change status button for users that are not members of the workspace.
• Model Registry: Added throttling to model download endpoint. This is disabled by default for on-premise deployments.
• Deployment: Usage and chargeback reports are now enabled by default for on-prem customers.

Fixed

• Experiment Management: A tooltip is now displayed in the Experiment Table if a value was truncated.
• Experiment Management: Fixed issues with the rendering of image groups in the single experiment graphics tab.
• Experiment Management: Fixed issues with code highlighting on small screens.
• Experiment Management: The order of panels is fixed after searching for a panel.
• Experiment Management: Long legend keys no longer take up the whole width of the chart when exporting a panel to JPEG.
• Experiment Management: Updated Python Panels so they now work on air-gapped environments.
• LLM: Chain trace is now displayed for chains with single span.
• Deployment: Fixed issue causing Can't call rollback when autocommit=true errors to be displayed in the Comet logs. This issue did not have any user impact but lead to confusion when reviewing logs.
• MPM: Fixed the computation of drift metric for categorical features when using training distributions.
• MPM: Fixed issues with uploading training distributions for datasets with numerical features.
• MPM: Fixed the tooltip in the feature distribution graph.

DB Migrations

• The table sso_login_url was added
• The table webhook_url_status was added
  

Feature toggles

• The feature toggle anonymous-mode now defaults to on.
• The feature toggle move-experiments-to-ws now defaults to on.

Environment Variables

The following variables were added:

The following environment varaibles where added in the YAML configuration files:

The following environment varaibles where updated in the YAML configuration files:

Oct 2023 release¶

Backend Version: 3.7.25

Frontend Version: 5.54.19

Optimizer: 2.0.22

Added

• Experiment Management: It is now possible to group metrics by parameter in the line chart.
• Experiment Management: Disk utilization metrics are now available in the system tab.
• Experiment Management: Images logged to an experiment can now be grouped and sorted based on image metadata.
• Experiment Management: Images can now be sorted based on the time they were logged.
• Experiment Management: Videos can now be logged and are available in both the graphics tab in the single experiment view and as part of the Video panel.
• LLM: User feedback can be updated through the LLM sidebar.
• LLM: Prompt inputs, outputs and metadata can now be viewed with a YAML editor rather than just JSON making it easier to read long prompts.
• MPM: Model Fairness can now be tracked for all classification models tracked in MPM.
• MPM: Previously for numerical features we would only display percentiles over time, you can now view the shape of the distribution for the entire time period selected or a given hour / day.
• MPM: The alert description is now added to the MPM alert email.
• MPM: Macro-average precision and recall are now computed by default for all classification models.

Changed

• MPM: Support for multi-output models is now on by default
• Deployment: Health check was added for project-columns sql triggers.
• SDK: Rate limit error messages where updated to include more information about how many events can be sent per throttling window.

Fixed

• Experiment Management: Fixed styling issues in the single experiment page for non-logged in users.
• Experiment Management: When editing a panel, it is not longer moved to the botton of the dashboard but instead retains it's position.
• Experiment Management: Fixed alignment issues with bounding boxes and masks for certain images.
• Experiment Management: Fixed race condition when logging parallel experiments to a project that did not previously exist. This lead to duplicate projects being created.
• LLM: Login modal is no longer displayed when viewing a public LLM project.
• LLM: Fixed styling issues with long Span names when viewing chain traces.
• MPM: Fixed issues with sorting models by name.

DB Migrations

-

Feature toggles

• The model-production-monitoring-segments-new-panel toggle was removed.
• The model-production-monitoring-models-pipeline toggle now defaults to ON.
• The model-production-monitoring-numerical-distribution toggle now defaults to ON.
• The model-production-monitoring-output-metrics toggle now defaults to ON.
• The video-panel toggle now defaults to ON.

Environment Variables

The following variables were added:

• MAX_VIDEOS_IN_VIDEO_PANEL
• MAX_EXPERIMENT_IN_VIDEO_PANEL
• MIN_VIDEO_SEARCH_PHRASE_LEN
• VIDEO_FILE_MAX_SIZE_IN_MB_FOR_THUMBNAILING
• UPLOAD_VIDEO_MAX_SIZE_MB
• USE_CACHE_CONTROL_IN_MEDIA
• CACHE_CONTROL_MAX_AGE_SEC
• GROUP_CHART_MAX_EXPERIMENTS_PER_GROUP
• GROUP_CHART_MAX_GROUP_CHARTS
• PROJECT_COLUMNS_CACHE_HEALTHCHECK_ENABLED
• RECALCULATE_EXPERIMENT_COUNTS
• MPM_ALLOW_MULTI_OUTPUT
• MPM_INPUT_DRIFT_SEARCH_MODELS_ENABLED
• MPM_PDF_CHART_BINS_NUMBER
• RECALCULATE_EXPERIMENT_COUNTS

The following variables were removed:

• MPM_MAX_SEGMENT_SLOTS_PER_MODEL
• MPM_NEW_ENGINE_MIGRATION_BATCH_SIZE
• MPM_NEW_ENGINE_MIGRATION_ITERATION_RANGE_DAYS
• MPM_NEW_ENGINE_MIGRATION_AUTH_KEY
• MPM_NEW_ENGINE_MIGRATION_SLEEP_SECS
• MPM_NEW_ENGINE_MIGRATION_QUERY_TIMEOUT
• MPM_NEW_ENGINE_MIGRATION_REOPEN_CONN_AFTER

Sept 2023 release¶

Backend Version: 3.7.0

Frontend Version: 5.52.5

Optimizer: 2.0.22

Added

• Experiment Management: Updated all chart tooltips to simplify the comparison on multiple experiments
• Experiment Management: Allow users to move projects between different workspace in the UI
• Experiment Management: Allow users to sort images based on the time at which they were logged
• Experiment Management: Filters and sorting defined in the data panel are now saved in project, experiment and report views
• Experiment Management: Preload previous / next images in the image gallery viewer to improve performance when navigating between images
• LLM: Users can now provide feedback for prompt response using thumbs up / down
• LLM: Improve UI/UX for long texts to analyze the data in the LLM projects
• MPM: Model fairness metrics are now available for all classification models.
• MPM: Alerts can now be sent to Slack and MS Teams channels
• MPM: AUC metric is computed by default for all classification models that have ground-truth labels

Fixed

• Experiment Management: Throttling by system metrics is now separate from metrics throttling. If system metrics are throttled, metrics will not be throttled.
• Experiment Management: Fixed issue where the SDK throws an error that the experiment key already exists (due a BE issue) on creating a new experiment.
• Experiment Management: Fixed compatibility issue with MySQL 8 when grouping experiments.
• Experiment Management: Fixed compatibility issue with MariaDB v10.
• Experiment Management: Fixed edge-case issue with accessing public projects when not logged in.
• Experiment Management: Fixed searching projects when the user is not in the first page of projects
• Experiment Management: Fixed decimal precision slider in the Metrics Experiment Page if the experiment has metric with negative values.
• Experiment Management: Fixed issue where parallel Coordinates Chart would draw some lines outside the chart if the parameter was null
• Reports: Fixed issue when panel ordering was not maintained when a report was downloaded as a PDF
• MPM: Custom metrics can now have an uppercase as the first character.
• MPM: Fixed issue with sorting by model ame in the models page.
• MPM: Removed filter by pipeline name in the models page that was not supported.

DB Migrations

• Updated schema in log_data_html and experiment_metrics_params_other.
• Remove or orphan records in team_members and organization_members.
• change DB character-set to utf8mb4 and collation to utf8mb4_unicode_ci.

Feature toggles

• Updated the feature toggle feature-move-project-to-workspace to default to on
• Updated the feature toggle model-production-monitoring-fairness to default to on

Aug 2023 release¶

Backend Version: 3.6.79

Frontend Version: 5.50.13

Optimizer: 2.0.22

Added

• Experiment Management: Added support for grouping of experiments in the line chart visualization when multiple metrics are added to the panel.
• Experiment Management: Network metrics are now logged by default to the Comet UI and available in the single experiment page.
• Experiment Management: OR filter conditions are now supported for all filters
• LLM: Prompt chains can now be viewed in the LLM sidebar
• Model Registry: Webhook payloads can now be configured to match the format expected by Github Actions
• MPM: Released a new aggregation engine that increases support throughput to over 5000 predictions per second.
• MPM: MAPE, MAE, RMSE, MAX_ERROR metrics are now available for all regression models.

Changed

• Experiment Management: log_env_details and log_git_metadata are now controlled separately in the Comet SDK.
• Experiment Management: The preview URI is now displayed for all remote assets.
• LLM: Prompt sidebar has been redesigned for easier analysis

Fixed

• Experiment Management: The table view (cards or list) is now saved in the project list page instead of resetting to the default when navigating away from this page.
• Experiment Management: Fixed performance issues related to get all the members of an organization.
• Experiment Management: Fixed edge-cases around the column-name caching changes released in the previous version.
• Experiment Management: Fixed issue related to edit the thumbnail for a custom panel.
• Experiment Management: The filter outlier toggle has been fixed in the single experiment page.
• Model Registry: Disabled download button for remote models.
• MPM: Support negative alert thresholds for model drift.
• MPM: Fixed UI crash when creating alerts under certain conditions.

DB Migrations

• Created a new table for the LLM feature: experiment_chains.
• Created a new table license_tokens for licensing needs.
• Created a new table query_filters which will be used as part of an upcoming feature.
• Created new tables for the project columns caching:
  • experiment_metrics_params_other_trigger_ins
  • experiment_metrics_params_other_trigger_upd
  • experiment_metrics_params_other_trigger_del
  • runs_to_experiments_trigger_ins
  • runs_to_experiments_trigger_upd
  • runs_to_experiments_trigger_del

Feature toggles

• Added feature toggle for model-registry-webhooks-github which defaults to on
• Added feature toggle for feature-move-project-to-workspace which defaults to off
• Added feature toggle for model-production-monitoring-fairness which defaults to off

July 2023 release¶

Release date: July 11, 2023

Backend Version: 3.6.40

Frontend Version: 5.46.10

Optimizer: 2.0.22

LLMOps

We are excited to announce that we have released a brand new set of LLM functionality, which includes new LLM projects that provide tailored views specifically for prompt engineering. Additionally, there is a new open-source SDK available at this GitHub repository.

These LLM projects allow data scientists to log prompts, responses, and any other metadata (such as usage data or examples). They can then easily analyze these prompts and responses in Comet.

Added

• MPM: Added support for creating alerts on percentiles for numerical features.
• MPM: Added support for the new metric Mean Absolute Percentage Error.
• Experiment Management: Added support for remote models, allowing you to track your models even when their files are stored remotely, such as in AWS S3.
• Experiment Management: Added support for nested hyperparameters in the Python EM SDK.
• Experiment Management: Added a button in the Experiment Graphics tab to quickly compare images across experiments using the Image Panel.
• Experiment Management: Added support for searching experiments by name in the Project page.
• Experiment Management: Added the possibility to share Project views across the installation. This needs to be enabled by your Comet Deployment Engineer.

Changed¶

• Experiment Management: We now store up to 150Mb of output for each Experiment. Previously, we stored only 100Mb of output per Experiment.
• DevOps: DB Migrations logs are now hidden at server startup.

Removed

• MPM: Removed the full-screen option for all charts.

Fixed

• Experiment Management: Project columns do not include columns from archived experiments anymore.
• Experiment Management: Fixed uploading of images and media in notes, both for projects and experiments.
• Experiment Management: Fixed rounding of metrics when using the CSV Export feature.
• Various fixes for the Confusion Matrix, Model Registry, Project panels.

Performance¶

• Project columns are now cached, which will make the Project page load faster.
• Support for direct upload to S3 from the Python SDK has been added. This feature needs to be enabled by your Comet Deployment Specialist.
• Support for downloading and visualizing images directly from S3 has been added, which can lead to faster performance with big or numerous images. This feature also needs to be enabled by your Comet Deployment Specialist.
• The performance of logging an Artifact has been improved.

DB Migrations

• Created a new table for the LLM feature: experiment_chain.
• Created a new table license_tokens for licensing needs.
• Created a new table query_filters which will be used as part of an upcoming feature.
• Created new tables for the project columns caching:
  • experiment_metrics_params_other_trigger_ins
  • experiment_metrics_params_other_trigger_upd
  • experiment_metrics_params_other_trigger_del
  • runs_to_experiments_trigger_ins
  • runs_to_experiments_trigger_upd
  • runs_to_experiments_trigger_del

May 2023 release¶

Release date: May 11, 2023

Backend Version: 3.5.80

Frontend Version: 5.40.11

Optimizer: 2.0.20

Added

• Experiment Management: Added 2 new featured panels (LLMOps History and LLMOps Playground) for analyzing LLM prompts and chains.
• Experiment Management: Added the ability to download all assets in a folder from the Comet UI.
• Experiment Management: Added system metrics and confusion metrics tabs when diffing multiple experiments.
• Experiment Management: Simplified the model registration process from the single experiment page by adding a Register model button.
• Experiment Management: Annotations can now be logged using the exp.log_image function and can be viewed in the graphics data and the image panel.
• MPM: Introduced a throttling mechanism, it is disabled by default for all on-prem installations.
• Model Registry: Added a UI for configuring model webhooks.
• Artifacts: When logging Artifacts from Snowflake, you can now view the SQL that was used to create it as well as a sample of the Snowflake dataset.

Changed

• MPM: MPM Segments are now controlled through a new dropdown instead of the sidebar for better usability. -

Fixed

• MPM: Drift metrics are now computed when requested rather than being pre-computed leading to greatly reduced load on the MPM database.

DB Migrations

• Created two new tables multipart_upload_metadata, multipart_upload_tags which will be used as part of an upcoming feature that optimizes the upload of large artifacts and assets.

• In order to update to this version, you will need to update the Minio versions to:

  • minio : minio/minio: RELEASE.2022-10-02T19-29-29Z
  • minio client : minio/mc: RELEASE.2022-10-22T03-39-29Z

March 2023 release¶

Release date: March 22, 2023

Backend Version: 3.4.90

Frontend Version: 5.34.11

Optimizer: 2.0.20

Added

• Experiment Management: Added support for TSV files in the Data Panel.
• Experiment Management: Added option to filter experiment so that only the experiments without a specific tag are shown.
• Experiment Management: Introduced support for image annotations in the single experiment page. You can now log bounding boxes and view them from the graphics tab. More information on this feature can be found in the documentation.
• Model Registry: Introduced the Model Approval process used the manage the status of a model. Workspace members will need to request a change in status for a model that will be approved by a workspace admin. More information on this feature can be found in the documentation.
• Model Registry: Introduced a new BI event for on-premise customers related to registering a model to the Model Registry. This BI event is only reported if BI event reporting is enabled for your environment.
• Configurations: Added the option to disable the creation of personal workspace during signup. This is controlled by the Comet environment variable PERSONAL_WORKSPACE_CREATION, please contact your Comet deployment engineer for more information.

Changed

• Experiment Management: Increase the default configuration value for the maximum number of metrics an experiment can log. The limit was increased from 200 to 50,000.
• Experiment Management: When adding a panel in the single experiment page, only the metrics and parameters for this experiment will be shown in the different dropdowns. Previously all metrics and parameters from the project were shown.

Fixed

• Experiment Management: In the Bar Chart Panel, the option to display the min for a metric was missing, this has been fixed.
• Experiment Management: Fixed an issue that meant you could not change the decimal precision when diffing multiple experiments.
• Experiment Management: Fixed UI crash when a user added 2 or more metrics to a line chart and tried to enable grouping in the chart editor.
• Experiment Management: Fixed UI crash when a user created a bar chart with a metric of type float.
• Experiment Management: Fixed performance issues when many code files are logged to an experiment.
• Experiment Management: When diffing two experiments, if one experiment had images but not the other, we would display the images for both experiemnts.
• Reports: Fixed issue leading to charts not rendering when a report was opened from a direct link rather than from the reports list page.

DB Migrations

• Created new table for MPM called mpm_organization_metadata
• Introduced a new unique key in the registry_model_item_pending_status table
• Added a new index to the billing_records table
• Drop width and height columns from the images_metadata
• Updated constraints on the registry_model_history table

February 2023 release¶

Release date: February 8, 2023

Backend Version: 3.4.41

Frontend Version: 5.30.7

Optimizer: 2.0.20

Added

• Experiment Management: Display aggregate metrics for metrics when using groups in the Experiment Table.
• Experiment Management: Show error in the Comet UI if an experiment fails because of an exception in the user script. Each experiment now has the value hasCrashed which can be used for sorting and filtering.
• Experiment Management: Allow the name column to be resized in the Experiment Table page.
• Reports: Reports are now embeddable as an iframe, the iframe code can be accessed using the share feature in the reports page.

• Admin Dashboard: Added more granular workspace permissions. Each user in a workspace is now a Workspace Admin or a Workspace Member, in addition each Workspace Member can be given additional permissions related to inviting users to the workspace and setting projects as public or private.

• Configurations: Comet now adds additional S3 metadata information (user_name, experiment_key, workspace_name, project_name,actual_file_name`) to objects stored in the Comet S3 bucket. This is disabled by default and can be enabled through an environment variable.

Changed

• Experiment Management: API and lineage tab are now hidden for Model Registry item that have been automatically created from MPM.
• Experiment Management: In the single Experiment page, the Charts and Panels tabs have been merged. The functionality that was previously available in the charts tab has been moved to the Panels tab.
• Experiment Management: When using a recent Python SDK, the System Metrics tab in the Single Experiment page will now display CPU and GPU metrics related to the training run rather than system metrics. Checkout the documentation for more details.
• Experiment Management: Improve responsiveness of the UI for projects with many Experiments.
• Artifact: Downloading an Artifact that contains remote assets from the UI is now disabled as it was never supported.

Fixed

• MPM: Fixed timezone issue in the MPM UI leading to recent data not being displayed.
• Experiment Management: Fixed a bug where the timestamp of a Metric could be wrong when displaying the minimal or maximal value.
• Experiment Management: Fixed issue with pinning experiment when using the Group By feature.
• Experiment Management: Fix a bug in the Customize Column modal when the selection list didn't show up.
• Reports: Fixed a number of issues in Reports.

DB Migrations

• Added new table for Model Registry Approval.
• Added new table for MPM Pipeline.
• Added a new table for MPM Training Distributions feature.
• Update table for supporting comments in the Model Registry history.

Security

• Authentication: The minimum size for Comet username is now configurable through the environment variable USER_NAME_MIN_LENGTH.
• Authentication - Add extra password validation for short passwords. Comet now also checks if the username is in the password, if there are more than x consecutive characters or if a character is repeated more than x times in a row. All these additional checks are configurable and can be disabled.
• Optimizer - Added support for reading the REDIS_AUTH token from the AWS Secrets Manager.

December 2022 release¶

Release date: December 26, 2022

Backend version: 3.3.64

Frontend version: 5.25.15

Optimizer version: 2.0.17

Added

• MPM: Add support for Segments
• MPM: Add support for debugger tab
• Admin Dashboard: Add support for batch user invite
• Experiment Management: Add new builtin-panel, the Data Panel to visualize structured data
• Experiment Management: Add support for new CPU and GPU System Metrics

Changed

• The Artifact Lineage page now also displays registered models.
• Updates to the Registry Model page which now includes a history tab as well as the ability to track changes to training metrics between model versions
• Single Registry Model versions can now be moved to another Registry model and/or another Workspace.
• The image panel functionality has been expanded to allow to zoom in or out.
• HTML content, logged through Experiment.log_html is now limited to 50MB
• Multiple improvements to Reports

DB Migrations

• Added new table (registry_model_item_notes) for Model Registry
• Added new tables (workspace-user-permission, organization_setting) for workspace permissions features (not used yet)

November 2022 release¶

Release date: November 3, 2022

Backend version: 3.3.8

Frontend version: 5.22.21

Optimizer version: 2.0.7

Added

• Package the Kubeflow, Vertex and Metaflow visualization panels as Featured panels.
• When using the Metaflow Comet Panel, the default project view will now automatically show the relevant Metaflow related panels and group experiment from the same Metaflow run together.
• Redesign of model registry page to allow user to customize table columns with data from experiment management.
• The image panel functionality has been expanded to allow a preview of images, displaying them at their full size.
• Introduced support for Custom Metrics in MPM allowing to track model or domain specific metrics

Changed

• It is now possible to diff up to 4 experiments at once from the Comet UI.
• Workspace names can now be 80-characters long, previously they were limited at 28 characters.
• It is now possible to select multiple y-axis when x-axis is a metric.

Fixed

• Fix an edge-case when custom panels stops refreshing when the browser page has been opened for several hours.
• Fix a bug when editing a custom panel leads to the panel to being duplicated.
• Fix scrolling issues in Workspaces tab in Admin dashboard.
• Fix display of the Embeddings Project link button in the Experiment Assets page.
• [Image Panel] Fix image panel slider when too many steps are logged

DB Migrations

• Adding columns and index to MySQL table registry_model_history for supporting model-registry history tab feature
• Adding Continuous Aggregates in TimeScaleDB for supporting MPM segments feature:
• Added hourly_features_segment_n where n is 0,1,2,...,9
• Added hourly_predictions_segment_n where n is 0,1,2,...,9

Security

• CVE-2022-42889: Upgrade version of apache commons-text dependency to non-vulnerable version 1.10.0.