AIStor Server publishes cluster and node metrics using the Prometheus Data Model. You can use any scraping tool to pull metrics data from AIStor Server for further analysis and alerting. This page covers metrics version 3, which has additional metrics and endpoints. Existing deployments can continue to use version 2 metrics. Version 3 endpoints The base endpoint, without an added path, returns cluster, node, and per-bucket statistic metrics from /cluster/usage/buckets.

AIStor supports publishing bucket notification events to a Redis service endpoint. Prerequisites The AIStor mc command line tool This procedure uses the mc command line tool for certain actions. See the mc Quickstart for installation instructions. Add a Redis endpoint to an AIStor Server The following procedure adds a new Redis service endpoint for supporting bucket notifications in an AIStor Server.

The AIStor Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the AIStor deployment to take advantage of the server-side processing power without constraints of the local machine where you run the AIStor Client. The keyrotate batch job type cycles the SSE-S3 or SSE-KMS keys for encrypted objects on an AIStor deployment. The YAML configuration supports filters to restrict key rotation to a specific set of objects by creation date, tags, metadata, or kms key. You can also define retry attempts or set a notification endpoint and token.

AIStor supports hot-swapping failed drives with new healthy drives. Healing AIStor detects and heals those drives without requiring any node or deployment-level restart. AIStor healing occurs only on the replaced drive(s) and in most cases has minimal or negligible impact on deployment performance.

If an AIStor node suffers complete hardware failure (for example, loss of all drives, data, etc.), the node begins healing operations once it rejoins the deployment. AIStor healing occurs only on the replaced hardware and does not typically impact deployment performance. AIStor healing ensures consistency and correctness of all data restored onto the drive. Changed in RELEASE.2025-02-04T00-52-01Z If multiple drives within the same erasure set require healing, AIStor prioritizes the first drive that begins healing within the erasure set. Other drives in the erasure set wait until healing finishes on the first drive before the next drive starts healing.

This page provides an overview of AIStor Server deployment architectures from a production perspective. A production AIStor Server deployment consists of at least 4 hosts with homogeneous storage and compute resources. Each AIStor Server host in this pool has matching compute, storage, and network configurations These hosts make up a server pool, where the AIStor Server presents the aggregated compute, memory, and storage as a single resource to clients. Each pool consists of one or more erasure sets

This section provides documentation for operating and maintaining AIStor Object Store deployments. Use this documentation for Day 2 and ongoing operational tasks in production environments, including monitoring, scaling, troubleshooting, and managing production AIStor deployments. You can find detailed explanations for core concepts that underpin AIStor. Understanding topics like erasure coding, distributed architecture, and data protection mechanisms provide a foundation for effectively using your cluster. AIStor provides comprehensive monitoring capabilities to maintain reliability and performance. Use metrics, logging, audit trails, and healthcheck endpoints to track deployment health and identify performance bottlenecks.

The following procedure adds a server pool to an existing AIStor cluster running on Kubernetes infrastructure. Each pool expands the total available storage capacity of the cluster with new nodes and persistent volumes while maintaining the overall availability of the cluster. Pay particular attention to the parity (EC:M) value of the existing cluster and the new pool. MinIO strongly recommends seeking an architecture review from MinIO Engineering through SUBNET to analyze potential performance impacts before committing to a new pool for your cluster.

Metrics and alerts AIStor Server publishes point-in-time metrics using the Prometheus Data Model. You can use any scraping tool which supports that data model to pull those metrics into a database for populating historical views, performing query/analysis of metrics data, or creating alerts on preferred data points.

AIStor supports publishing audit logs to one or more configured Apache Kafka receivers. AIStor send an event for each API operation to the receiver for processing and storage. The receiver is responsible for correctly processing events, including returning 200OK or similar success messages upon receipt of the event. AIStor cannot recover events that were successfully sent but not correctly stored on the receiver. You can configure a new Kafka audit endpoint using either environment variables or runtime configuration settings. If you configure both, AIStor uses the environment variables.