Minio / S3 Service (TrueNAS-13.0-U3.1) Metrics

[Richard Durso]

I have Minio S3 Service working. It is configured to expose metrics endpoint and Prometheus is scraping it. I can see metrics within Prometheus UI.

I'm trying to get the S3 Service build in dashboard to fetch metrics from an external Prometheus. The TrueNAS Core S3 service doesn't have native support it seems for providing the two environment variables needed to enable this MINIO_PROMETHEUS_URL and MINIO_PROMETHEUS_JOB_ID.

MINIO_PROMETHEUS_URL=https://k3s.[REDACTED].com/prometheus/
MINIO_PROMETHEUS_JOB_ID=minio-job-truenas

So I'm poking around...

First I can CURL from TrueNAS to the remote Prometheus endpoint:

root@truenas[~]# curl -XGET -G 'https://k3s.[REDACTED].com/prometheus/api/v1/label/__name__/values' --data-urlencode 'match[]={__name__=~".+", job="minio-job-truenas"}'

{"status":"success","data":["ALERTS","ALERTS_FOR_STATE","count:up0","count:up1","minio_bucket_objects_size_distribution","minio_bucket_replication_received_bytes","minio_bucket_usage_object_total","minio_bucket_usage_total_bytes","minio_cluster_disk_free_inodes","minio_cluster_nodes_offline_total","minio_cluster_nodes_online_total","minio_inter_node_traffic_errors_total","minio_inter_node_traffic_received_bytes","minio_inter_node_traffic_sent_bytes","minio_node_disk_free_bytes","minio_node_disk_total_bytes","minio_node_disk_used_bytes","minio_node_go_routine_total","minio_node_ilm_expiry_pending_tasks","minio_node_ilm_transition_active_tasks","minio_node_ilm_transition_pending_tasks","minio_node_ilm_versions_scanned","minio_node_scanner_bucket_scans_finished","minio_node_scanner_bucket_scans_started","minio_node_scanner_directories_scanned","minio_node_scanner_objects_scanned","minio_node_scanner_versions_scanned","minio_s3_requests_errors_total","minio_s3_requests_inflight_total","minio_s3_requests_rejected_auth_total","minio_s3_requests_rejected_header_total","minio_s3_requests_rejected_invalid_total","minio_s3_requests_rejected_timestamp_total","minio_s3_requests_total","minio_s3_requests_waiting_total","minio_s3_time_ttfb_seconds_distribution","minio_s3_traffic_received_bytes","minio_s3_traffic_sent_bytes","minio_software_commit_info","minio_software_version_info","minio_usage_last_activity_nano_seconds","promhttp_metric_handler_requests_in_flight","promhttp_metric_handler_requests_total","scrape_duration_seconds","scrape_samples_post_metric_relabeling","scrape_samples_scraped","scrape_series_added","up"]}#

I tried exporting the variables within /usr/local/etc/rc.d/minio such as:

: ${minio_prometheus_url="https://k3s.[REDACTED].com/prometheus/"}
: ${minio_prometheus_job="minio-job-truenas"}
...
# Insert these for Minio
export MINIO_PROMETHEUS_URL=${minio_prometheus_url}
export MINIO_PROMETHEUS_JOB_ID=${minio_prometheus_job}

pidfile="/var/run/${name}.pid"
procname="/usr/local/bin/minio"
command="/usr/sbin/daemon"
command_args="-f -t ${name} ${minio_syslog_output_flags} -p ${pidfile} /usr/bin/env ${minio_env} ${procname} ${minio_config_flag} -S \"${mi
nio_certs}\" --quiet server --address=\"${minio_address}\" ${minio_console_address_arg} ${minio_disks} ${minio_args}"
...

Restarted the S3 service and the Minio Dashboard detects the Environment variables are set, and now displays:

We can't retrieve advanced metrics at this time
MinIO Dashboard will display basic metrics as we couldn't connect to Prometheus successfully.

Please try again in a few minutes. If the problem persists, you can review your configuration and confirm that Prometheus server is up and running.

The MinIO / Tools / Logs:

API: SYSTEM
Time: 19:12:06:0
DeploymentID: 2a488c37-8e1b-401a-beda-b22910187c88
Error: readlink /proc/self: no such file or directory (*fs.PathError)
internal/logger/logonce.go:52:logger.(*logOnceType).logOnceIf()
internal/logger/logonce.go:92:logger.LogOnceIf()
cmd/metrics-v2.go:891:cmd.getMinioProcMetrics.func1()
cmd/metrics-v2.go:224:cmd.(*MetricsGroup).RegisterRead.func1.1()
cmd/metrics-v2.go:236:cmd.(*MetricsGroup).Get()
cmd/metrics-v2.go:1737:cmd.populateAndPublish()
cmd/metrics-v2.go:1702:cmd.ReportMetrics.func1()

Not sure what else to try, don't see any options for more verbose logging.