Skip to content

Add prom metrics to cloud's feature docs #848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Piroddi wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add prom metrics to cloud's feature docs #848

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d93bcf8
Add prom metrics to cloud's feature docs
Piroddi Apr 2, 2026
fa082cc
removed internal metrics
Piroddi Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 125 additions & 1 deletion docs/HyperIndex/Hosted_Service/hosted-service-features.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ This feature is only available for blockchain indexers deployed with version 2.2
Stay informed about your indexer's health and performance with our integrated alerting system. Configure multiple notification channels and choose which alerts you want to receive.

:::info Version Requirement
This feature is only available for blockchain indexers deployed with version 2.24.0 or higher.
This feature is only available for indexers deployed with version 2.24.0 or higher.
:::

### Notification Channels
Expand All @@ -104,6 +104,130 @@ Configure one or multiple notification channels to receive alerts:
- **Telegram**
- **Email**

## Monitoring

:::info Version Requirement
This feature is only available for indexers deployed with version 3.0.0 or higher.
:::

Monitor your indexer's performance and health using standard Prometheus metrics. Metrics are exposed at your deployment's endpoint under `/hyperindex/metrics`, making it easy to integrate with your existing monitoring stack (Prometheus, Grafana, Datadog, etc.).

**Endpoint:**
```
<your-endpoint-url>/hyperindex/metrics
```
Comment on lines +116 to +118
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add language identifier to fenced code block.

The code block should specify a language identifier for proper syntax highlighting and markdown compliance.

📝 Proposed fix 
 **Endpoint:**
-```
+```plaintext
 <your-endpoint-url>/hyperindex/metrics

</details>

As per coding guidelines, fenced code blocks should have a language specified (markdownlint MD040).

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
**Endpoint:**
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)

[warning] 116-116: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/HyperIndex/Hosted_Service/hosted-service-features.md` around lines 116 -
118, The fenced code block containing "<your-endpoint-url>/hyperindex/metrics"
in hosted-service-features.md lacks a language identifier; update the opening
fence to include a language (e.g., change ``` to ```plaintext) so the block
reads as a labeled fenced code block (keep the existing closing ``` unchanged)
to satisfy markdownlint MD040.


### Available Metrics

#### Progress & Sync Status

| Metric | Type | Description |
|--------|------|-------------|
| `envio_progress_block` | gauge | Latest block number processed and stored in the database. Labeled by `chainId`. |
| `envio_progress_events` | gauge | Number of events processed and reflected in the database. Labeled by `chainId`. |
| `envio_progress_ready` | gauge | Whether the chain is fully synced to the head (`1` = synced). Labeled by `chainId`. |

#### Event Processing

| Metric | Type | Description |
|--------|------|-------------|
| `envio_processing_seconds` | counter | Cumulative time spent executing event handlers during batch processing. |
| `envio_processing_handler_seconds` | counter | Cumulative time spent inside individual event handler executions. Labeled by `contract` and `event`. |
| `envio_processing_handler_total` | counter | Total number of individual event handler executions. Labeled by `contract` and `event`. |
| `envio_processing_max_batch_size` | gauge | Maximum number of items to process in a single batch. |

#### Entity Preloading

| Metric | Type | Description |
|--------|------|-------------|
| `envio_preload_seconds` | counter | Cumulative time spent preloading entities during batch processing. |
| `envio_preload_handler_seconds` | counter | Wall-clock time spent inside individual preload handler executions. Labeled by `contract` and `event`. |
| `envio_preload_handler_seconds_total` | counter | Cumulative time spent in preload handlers (can exceed wall-clock time due to parallel execution). Labeled by `contract` and `event`. |
| `envio_preload_handler_total` | counter | Total number of individual preload handler executions. Labeled by `contract` and `event`. |

#### Storage

| Metric | Type | Description |
|--------|------|-------------|
| `envio_storage_write_seconds` | counter | Cumulative time spent writing batch data to storage. |
| `envio_storage_write_total` | counter | Total number of batch writes to storage. |
| `envio_storage_load_seconds` | counter | Time spent loading data from storage. Labeled by `operation`. |
| `envio_storage_load_total` | counter | Number of successful storage load operations. Labeled by `operation`. |
| `envio_storage_load_size` | counter | Cumulative number of records loaded from storage. Labeled by `operation`. |

#### Data Source & Fetching

| Metric | Type | Description |
|--------|------|-------------|
| `envio_fetching_block_range_seconds` | counter | Cumulative time spent fetching block ranges. Labeled by `chainId`. |
| `envio_fetching_block_range_total` | counter | Total number of block range fetch operations. Labeled by `chainId`. |
| `envio_fetching_block_range_events_total` | counter | Cumulative number of events fetched across all block range operations. Labeled by `chainId`. |
| `envio_fetching_block_range_size` | counter | Cumulative number of blocks covered across all fetch operations. Labeled by `chainId`. |
| `envio_source_request_total` | counter | Number of requests made to data sources. Labeled by `source`, `chainId`, and `method`. |
| `envio_source_known_height` | gauge | Latest known block number reported by the data source. Labeled by `source` and `chainId`. |

#### Indexing Pipeline

| Metric | Type | Description |
|--------|------|-------------|
| `envio_indexing_known_height` | gauge | Latest known block number reported by the active indexing source. Labeled by `chainId`. |
| `envio_indexing_concurrency` | gauge | Number of executing concurrent queries to the chain data source. Labeled by `chainId`. |
| `envio_indexing_max_concurrency` | gauge | Maximum number of concurrent queries allowed. Labeled by `chainId`. |
| `envio_indexing_buffer_size` | gauge | Current number of items in the indexing buffer. Labeled by `chainId`. |
| `envio_indexing_buffer_block` | gauge | Highest block number fully fetched by the indexer. Labeled by `chainId`. |
| `envio_indexing_idle_seconds` | counter | Time the indexer source syncing has been idle. A high value may indicate a bottleneck. Labeled by `chainId`. |
| `envio_indexing_partitions` | gauge | Number of partitions used to split fetching logic. Labeled by `chainId`. |
| `envio_indexing_addresses` | gauge | Number of addresses indexed on chain (static and dynamic). Labeled by `chainId`. |

#### Reorgs & Rollbacks

| Metric | Type | Description |
|--------|------|-------------|
| `envio_reorg_detected_total` | counter | Total number of reorgs detected. |
| `envio_reorg_detected_block` | gauge | Block number where a reorg was last detected. |
| `envio_reorg_threshold` | gauge | Whether indexing is currently within the reorg threshold. |
| `envio_rollback_enabled` | gauge | Whether rollback on reorg is enabled. |
| `envio_rollback_total` | counter | Number of successful rollbacks on reorg. |
| `envio_rollback_seconds` | counter | Total time spent on rollbacks. |
| `envio_rollback_events` | counter | Number of events rolled back on reorg. |

#### Effect API

| Metric | Type | Description |
|--------|------|-------------|
| `envio_effect_call_seconds` | counter | Processing time taken to call the Effect function. Labeled by `effect`. |
| `envio_effect_call_total` | counter | Cumulative number of resolved Effect function calls. Labeled by `effect`. |
| `envio_effect_active_calls` | gauge | Number of Effect function calls currently running. Labeled by `effect`. |
| `envio_effect_cache` | gauge | Number of items in the effect cache. Labeled by `effect`. |
| `envio_effect_queue` | gauge | Number of effect calls waiting in the rate limit queue. |

#### System Info

| Metric | Type | Description |
|--------|------|-------------|
| `envio_info` | gauge | Information about the indexer. Labeled by `version`. |
| `envio_process_start_time_seconds` | gauge | Start time of the process since Unix epoch in seconds. |

### Example: Scraping with Prometheus

Add the following to your `prometheus.yml` to scrape your indexer's metrics:

```yaml
scrape_configs:
- job_name: "envio-indexer"
metrics_path: "/hyperindex/metrics"
static_configs:
- targets: ["<your-endpoint-host>"]
```

### Example: Key Metrics to Watch

- **`envio_progress_ready`** and **`hyperindex_synced_to_head`** — confirm your indexer is caught up to the chain head
Copy link
Copy Markdown
Member

@DZakh DZakh Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not mention hyperindex_synced_to_head

Copy link
Copy Markdown
Contributor Author

@Piroddi Piroddi Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

- **`envio_progress_block`** — track indexing progress over time
- **`envio_processing_handler_seconds`** — identify slow event handlers by contract and event
- **`envio_indexing_idle_seconds`** — a high value may indicate the data source sync is a bottleneck
Copy link
Copy Markdown
Member

@DZakh DZakh Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be Querying seconds - idle means that the data source is not queried.

Copy link
Copy Markdown
Contributor Author

@Piroddi Piroddi Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow. Do you want to rename this metric?

- **`envio_reorg_detected_total`** — monitor for chain reorganizations

## Zero-Downtime Deployments

Update your blockchain indexer without any service interruption using our seamless deployment system with static production endpoints.
Expand Down