> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pgw.sipmicro.services/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape pgw metrics with Prometheus

> Scrape the Prometheus metrics that pgw serves on its admin listener, and look up each metric, its type, and its labels.

Use the metrics to see how many devices are registered, what share of pushes fail, and how long apps take to wake. pgw serves them in Prometheus format at `/metrics` on the admin listener, with no API key.

The admin listener accepts connections only from the pgw host by default, so run Prometheus or a scraping agent on that host.

Start with these four metrics:

| Metric | What it tells you |
| - | - |
| `pgw_bindings` | How many devices pgw can wake. A sudden drop means that apps can't register. |
| `pgw_pushes_total` | Pushes by provider and result. A growing `error` count means that a push service refuses or fails your pushes. |
| `pgw_invite_outcomes_total` | How calls end: delivered, timed out, or cancelled. |
| `pgw_upstream_up` | Whether each PBX answers pgw's keepalive. |

## Scrape the metrics

**1. Check the endpoint**

```bash theme={null}
curl -s http://127.0.0.1:8080/metrics | grep ^pgw_
```

```text theme={null}
pgw_app_connections{transport="tls"} 0
pgw_app_connections{transport="wss"} 0
pgw_app_connections_refused_total 0
pgw_bindings 0
...
pgw_invite_outcomes_total{result="delivered"} 1
...
pgw_registrations_total{result="ok"} 1
pgw_registrations_total{result="rejected"} 0
pgw_registrations_total{result="upstream_error"} 0
...
pgw_tls_cert_not_after_timestamp_seconds 1.826322758e+09
...
pgw_upstream_up{upstream="hTA160K5dFVJf-42dGDCrA"} 1
...
```

Some labeled series, such as those of `pgw_pushes_total`, appear only after pgw counts their first event.

**2. Add a scrape job**

Add a job for pgw to the Prometheus configuration. If you changed `admin_listen`, use its address as the target.

```yaml theme={null}
scrape_configs:
  - job_name: pgw
    static_configs:
      - targets: ["127.0.0.1:8080"]
```

## Metric reference

Besides the following metrics, pgw exports the Go runtime metrics (`go_*`), the process metrics (`process_*`), and `go_build_info`.

### Registrations

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_registrations_total` | Counter | `result`: `ok`, `rejected`, `upstream_error` | REGISTER requests from apps. `upstream_error` means that the PBX didn't answer or answered `503` or `408`, that in managed mode it refused pgw's stored credentials or rejected the registration, or that it dropped pgw's Path on an upstream with `strict_path`. A `401` challenge, or another PBX refusal that pgw relays to the app unchanged, isn't counted. |
| `pgw_bindings` | Gauge | | Registered devices. |
| `pgw_register_auth_failures_total` | Counter | `reason`: `pbx_refused`, `wrong_password`, `unknown_user`, `disabled_user`, `username_mismatch`, `ambiguous_credentials`, `bearer_invalid`, `bearer_subject_mismatch` | Refused credentials that count toward a ban of their source address. `pbx_refused` means the PBX refused an app's REGISTER in passthrough mode. The other reasons come from managed mode: a wrong password, a user that the upstream doesn't have or has turned off, a digest username that isn't the To user, more than one set of credentials, a Bearer token that fails its checks, or a valid token for another user. The app gets the same answer for each. A refresh over a registered connection isn't counted. |
| `pgw_register_bans_total` | Counter | | Source addresses that pgw banned for too many refused credentials. |
| `pgw_register_banned_total` | Counter | | REGISTER requests that pgw answered with `503` because their source address was banned. |
| `pgw_register_auth_failures_dropped_total` | Counter | | Refused credentials that pgw couldn't count toward a ban, because it already tracks 65,536 source addresses. |

### Calls

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_invite_outcomes_total` | Counter | `result`: `delivered`, `forbidden`, `not_found`, `merged`, `busy`, `timeout`, `cancelled`, `answered_elsewhere`, `unavailable`, `push_failed`, `error` | Calls from the PBX, by how they ended. Each device counts on its own, so a call to a user with three devices counts three times. `forbidden` means the source address isn't in any upstream's `invite_source_cidrs`. `answered_elsewhere` means another device of the user answered, or the PBX cancelled the call with `Reason: SIP;cause=200`. |
| `pgw_invites_refused_total` | Counter | `reason`: `per_binding_cap`, `global_cap`, `call_push_limit` | Calls that pgw refused because it already held `max_held_per_binding` calls for the device or `max_held_invites` calls in total, or because the device reached `call_push_rate_per_binding`. |
| `pgw_bucket_depth` | Gauge | | Calls held now while their apps wake. |
| `pgw_hold_seconds` | Histogram | `outcome`: `delivered`, `timeout`, `cancelled`, `push_failed`, `shutdown`, `push_limited` | How long pgw held a call, by how the hold ended. `push_limited` means the device reached `call_push_rate_per_binding`, so pgw answered `486 Busy Here` without a push. |
| `pgw_app_requests_total` | Counter | `method`: `INVITE`, `MESSAGE`, `SUBSCRIBE`, `OPTIONS`, `REFER`, `PUBLISH`, `INFO`, `NOTIFY`, `BYE`, `UPDATE`, `PRACK`, or `other`; `result`: `relayed`, `forbidden`, `rate_limited`, `error` | Requests that apps sent toward their PBX, such as calls, messages, and requests inside a call. ACK isn't counted, nor a request refused with `483` or `420` for its Max-Forwards or Proxy-Require. `forbidden` means the connection had no registration, a request inside a call came on another connection than the registration's, or the Request-URI carried `pn-*` parameters. |

### Pushes

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_pushes_total` | Counter | `provider`: `apns`, `fcm`, `webpush`; `type`: `call`, `missed`, `refresh`, `test`; `result`: `ok`, `dead`, `error`, `skipped`, `backoff`, `limited` | Push attempts. See [Push results](#push-results). |
| `pgw_push_latency_seconds` | Histogram | `provider` | Time until the push service answered a push. |
| `pgw_push_queue_depth` | Gauge | `provider`; `lane`: `call`, `refresh` | Pushes waiting for a free worker. The `call` lane holds call, missed-call, and test pushes. Each lane holds up to `push_queue_size` pushes. A call lane that stays near `push_queue_size` means the push service is slow or `push_workers` is too low. |
| `pgw_push_queue_skipped_total` | Counter | `provider`; `lane` | Queued pushes that pgw dropped because the caller had given up or the call had already ended. |
| `pgw_push_breakers_open` | Gauge | `provider`: `apns`, `fcm`, `webpush` | Apps whose push service failed 5 times in a row, so pgw holds their pushes back. For `webpush`, each pair of a web app and a browser push service counts once. |
| `pgw_push_event_queue_depth` | Gauge | | Push events waiting to be stored. A value that stays high means the database is slow. |
| `pgw_push_events_dropped_total` | Counter | `reason`: `queue_full`, `store_error`, `shutdown` | Push events that pgw didn't store. The pushes themselves were sent. `queue_full` means the database fell behind, `store_error` that a write failed, and `shutdown` that pgw stopped before it could store them. |

#### Push results

| Result | Meaning |
| - | - |
| `ok` | The push service accepted the push. |
| `dead` | The push service reported the push token as invalid. |
| `error` | The push failed. This includes a call push that pgw couldn't queue because the call lane was full. |
| `skipped` | pgw didn't send the push: a non-call push to an iPhone that registered no remote token, a missed-call or refresh push to a browser, which gets only call and test pushes, or a call push whose call ended while the push waited in the queue. |
| `backoff` | pgw held the push back because the push service was failing, was rate-limiting the app or the device, or had asked pgw to wait with `Retry-After`, or because half of `push_workers` were already waiting on the same browser push service. |
| `limited` | pgw didn't send a missed-call or test push because the device reached `call_push_rate_per_binding`. |

### Upstreams

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_upstream_requests_total` | Counter | `method`: `REGISTER`; `class`: `1xx` through `6xx`, or `other` | Final responses from the PBX to app REGISTER requests that pgw forwarded in passthrough mode. pgw's own managed mode registrations aren't counted. |
| `pgw_upstream_register_seconds` | Histogram | | Time the PBX took to answer an app's REGISTER that pgw forwarded in passthrough mode. |
| `pgw_upstream_up` | Gauge | `upstream`: the upstream's ID | `1` when the PBX answered pgw's recent OPTIONS, `0` after it missed 3 in a row. Any answer counts, even an error status. Only upstreams whose `keepalive_seconds` isn't `0` have it. |
| `pgw_upstream_keepalive_failures_total` | Counter | `upstream` | OPTIONS to the PBX that got no answer within 5 seconds. |
| `pgw_upstream_auth_failures_total` | Counter | `upstream` | Managed registrations that the PBX refused after pgw sent the stored password. The cause is a changed password, or an AOR with no room for another contact, which some PBXs answer with `403`. |
| `pgw_upstream_path_missing_total` | Counter | `upstream` | Registrations whose answer from the PBX didn't keep pgw's Path. Such a PBX sends calls straight to the app, past pgw, so the app gets no call push. |

### Managed users and Bearer tokens

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_user_source_lookups_total` | Counter | `upstream`; `result`: `found`, `missing`, `unavailable`, `cached` | Lookups of a managed upstream's users, in pgw's users or in Kamailio's database. `missing` means the user doesn't exist or is turned off. `unavailable` means pgw couldn't read the user, so the app got `500` with `Retry-After: 5`. `cached` counts Kamailio subscribers that pgw read in the last 60 seconds and didn't query again. |
| `pgw_user_source_lookup_seconds` | Histogram | `upstream` | Time Kamailio's database took to answer a subscriber query. A query gives up after 2 seconds. |
| `pgw_kamailio_domains` | Gauge | `upstream` | Domains in the last list that pgw read from Kamailio's domain table. |
| `pgw_kamailio_domain_load_failures_total` | Counter | `upstream` | Failed reads of Kamailio's domain table. pgw keeps the last list that it read. |
| `pgw_bearer_verifications_total` | Counter | `upstream`; `result`: `valid`, `invalid`, `subject_mismatch`, `keys_unavailable` | Bearer tokens on app REGISTER requests. `keys_unavailable` means pgw couldn't fetch the issuer's keys, so the app got `500` with `Retry-After: 5` and nothing counted toward a ban. |
| `pgw_bearer_key_fetch_failures_total` | Counter | `upstream` | Failed fetches of the token issuer's keys from `bearer.jwks_url`. pgw keeps the keys that it last fetched, and after a failure fetches again at most once a minute. |

### Connections and internals

| Metric | Type | Labels | Description |
| - | - | - | - |
| `pgw_app_connections` | Gauge | `transport`: `tls`, `wss` | App connections open now. |
| `pgw_app_connections_refused_total` | Counter | | App connections that pgw closed at once because `app_max_connections` connections were open. |
| `pgw_tls_cert_not_after_timestamp_seconds` | Gauge | | When the app listener's certificate expires, in Unix seconds. |
| `pgw_sip_log_dropped_total` | Counter | | SIP messages that the SIP message log left out because writing fell behind, for example when a log collector reads standard output slowly. pgw never delays SIP handling to wait for the log. |
| `pgw_panics_total` | Counter | | Internal errors that pgw recovered from. The affected request or push failed, with `500` when a request still expected an answer, and pgw kept running. |

## Next steps

> [card] [Set up alerts](https://docs.pgw.sipmicro.services/alerts.md)
>
> Copy PromQL expressions that alert on these metrics.

> [card] [Read the logs](https://docs.pgw.sipmicro.services/logs.md)
>
> Find the log line behind a counter that moved.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.