Skip to content

fix(spanner): respect universe_domain and client_options in metrics exporter - #19172

Open
sakthivelmanii wants to merge 6 commits into
mainfrom
fix/spanner-metrics-exporter-universe-domain
Open

sakthivelmanii wants to merge 6 commits into
mainfrom
fix/spanner-metrics-exporter-universe-domain

Conversation

@sakthivelmanii

Copy link
Copy Markdown
Contributor

Problem

  • CloudMonitoringMetricsExporter always connected to monitoring.googleapis.com, ignoring client_options.universe_domain and GOOGLE_CLOUD_UNIVERSE_DOMAIN.
  • Clients with a custom client_options.api_endpoint still exported built-in metrics to monitoring.googleapis.com.
  • Client.__init__ passed raw project and credentials instead of resolved self.project and self._credentials, causing fallback to ambient ADC.

Solution

  • Pass resolved self.project, self._credentials, and filtered monitoring client_options to _initialize_metrics and CloudMonitoringMetricsExporter.
  • Skip built-in metrics export when a custom non-default api_endpoint is configured.
  • Configure MetricServiceClient via a transport callable so universe_domain and credentials validation work natively, re-scope Spanner-only scoped credentials to monitoring.write, and avoid ADC project fallback when explicit credentials are provided.
  • Add unit tests for both sync and async clients and the metrics exporter.

…xporter

## Problem
- `CloudMonitoringMetricsExporter` always connected to `monitoring.googleapis.com`, ignoring `client_options.universe_domain` and `GOOGLE_CLOUD_UNIVERSE_DOMAIN`.
- Clients with a custom `client_options.api_endpoint` still exported built-in metrics to `monitoring.googleapis.com`.
- `Client.__init__` passed raw `project` and `credentials` instead of resolved `self.project` and `self._credentials`, causing fallback to ambient ADC.

## Solution
- Pass resolved `self.project`, `self._credentials`, and filtered monitoring `client_options` to `_initialize_metrics` and `CloudMonitoringMetricsExporter`.
- Skip built-in metrics export when a custom non-default `api_endpoint` is configured.
- Configure `MetricServiceClient` via a transport callable so `universe_domain` and credentials validation work natively, re-scope Spanner-only scoped credentials to `monitoring.write`, and avoid ADC project fallback when explicit credentials are provided.
- Add unit tests for both sync and async clients and the metrics exporter.
@sakthivelmanii
sakthivelmanii requested a review from a team as a code owner November 21, 2026 21:13

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates both the synchronous and asynchronous Spanner clients to propagate client_options (such as universe_domain and api_endpoint) to the built-in metrics exporter, skipping metrics export when a custom endpoint is configured. Feedback on the changes highlights that google.api_core.client_options is not explicitly imported in either client file, which could lead to a NameError. Additionally, it is recommended to use getattr when accessing attributes from the caller-supplied client_options object to prevent potential attribute errors.

@olavloite olavloite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, assuming that the choice to disable monitoring when any custom endpoint has been set is intentional.

not _get_spanner_emulator_host()
and not emulator_host
and not isinstance(credentials, AnonymousCredentials)
and not has_custom_endpoint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This disables monitoring export for any custom endpoint (e.g. also for something like spanner.us-central1.rep.googleapis.com). Is that intentional?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, skipping by default is intentional so custom/PSC/regional endpoints do not silently egress credentials and metrics to global monitoring.googleapis.com. Added SPANNER_MONITORING_HOST support (matching Go/Java) so callers using a custom api_endpoint can explicitly opt in to a monitoring endpoint.

@olavloite olavloite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, a few requests on test coverage and async/sync test parity:

  1. tests/unit/_async/test_client_extra.py is missing async counterparts for two tests added in tests/unit/test_client.py (test_initialize_metrics_universe_propagation_and_option_filtering and test_client_custom_api_endpoint_skips_metrics_exporter), and test_initialize_metrics_custom_api_endpoint_skips_exporter only passes a ClientOptions instance, leaving the from_dict branch in google/cloud/spanner_v1/_async/client.py (_initialize_metrics) uncovered.
  2. tests/unit/test_metrics_exporter.py doesn't yet test requires_scopes=True or monitoring / monitoring.write scoped credentials, and uses bare assertRaises(ValueError) without checking the error message.

Suggested patch:

diff --git a/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py b/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
index f8b648f2e0e..11111111111 100644
--- a/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
+++ b/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
@@ -281,36 +281,121 @@ class TestClientExtra(unittest.IsolatedAsyncioTestCase):
 
         from google.cloud.spanner_v1._async import client as MUT
 
+        custom_endpoints = [
+            {"api_endpoint": "private-spanner.internal:443"},
+            ClientOptions(api_endpoint="private-psc.internal:443"),
+            ClientOptions(api_endpoint="spanner.us-central1.rep.googleapis.com"),
+            ClientOptions(
+                api_endpoint="spanner.googleapis.com",
+                universe_domain="my-universe.com",
+            ),
+            ClientOptions(api_endpoint="spanner.googleapis.com@evil.example"),
+        ]
+        for client_options in custom_endpoints:
+            with (
+                mock.patch(
+                    "google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
+                    False,
+                ),
+                mock.patch(
+                    "google.cloud.spanner_v1._async.client._get_spanner_emulator_host",
+                    return_value=None,
+                ),
+                mock.patch(
+                    "google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
+                ) as mock_exporter,
+                mock.patch(
+                    "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
+                ) as mock_set_provider,
+                mock.patch(
+                    "google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
+                ),
+            ):
+                MUT._initialize_metrics(
+                    "project",
+                    self.credentials,
+                    client_options=client_options,
+                )
+                mock_exporter.assert_not_called()
+                mock_set_provider.assert_called_once()
+                self.assertIsInstance(
+                    mock_set_provider.call_args[0][0],
+                    MUT.metrics.NoOpMeterProvider,
+                )
+
+    def test_initialize_metrics_universe_propagation_and_option_filtering(self):
+        from google.api_core.client_options import ClientOptions
+
+        from google.cloud.spanner_v1._async import client as MUT
+
+        creds = mock.Mock()
+        client_options = ClientOptions(
+            universe_domain="my-universe.com",
+            quota_project_id="quota-proj",
+            scopes=["https://www.googleapis.com/auth/spanner.admin"],
+            api_audience="https://spanner.my-universe.com",
+            api_key="fake-api-key",
+        )
         with (
             mock.patch(
                 "google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
                 False,
             ),
             mock.patch(
                 "google.cloud.spanner_v1._async.client._get_spanner_emulator_host",
                 return_value=None,
             ),
             mock.patch(
                 "google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
             ) as mock_exporter,
             mock.patch(
-                "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
-            ) as mock_set_provider,
+                "google.cloud.spanner_v1._async.client.PeriodicExportingMetricReader"
+            ),
+            mock.patch("google.cloud.spanner_v1._async.client.MeterProvider"),
+            mock.patch(
+                "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
+            ),
             mock.patch(
                 "google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
             ),
         ):
             MUT._initialize_metrics(
-                "project",
-                self.credentials,
-                client_options=ClientOptions(api_endpoint="private-psc.internal:443"),
+                "my-project",
+                creds,
+                client_options=client_options,
             )
-            mock_exporter.assert_not_called()
-            mock_set_provider.assert_called_once()
-            self.assertIsInstance(
-                mock_set_provider.call_args[0][0],
-                MUT.metrics.NoOpMeterProvider,
-            )
+
+        mock_exporter.assert_called_once()
+        _, kwargs = mock_exporter.call_args
+        self.assertEqual(kwargs["project_id"], "my-project")
+        self.assertIs(kwargs["credentials"], creds)
+        monitoring_opts = kwargs["client_options"]
+        self.assertIsInstance(monitoring_opts, ClientOptions)
+        self.assertEqual(monitoring_opts.universe_domain, "my-universe.com")
+        self.assertIsNone(monitoring_opts.api_endpoint)
+        self.assertIsNone(monitoring_opts.scopes)
+        self.assertIsNone(monitoring_opts.api_audience)
+        self.assertIsNone(monitoring_opts.quota_project_id)
+        self.assertIsNone(monitoring_opts.api_key)
+
+    def test_client_custom_api_endpoint_skips_metrics_exporter(self):
+        import os
+
+        from google.cloud.spanner_v1._async import client as MUT
+
+        creds = build_scoped_credentials()
+        with (
+            mock.patch.dict(os.environ, {"SPANNER_DISABLE_BUILTIN_METRICS": "false"}),
+            mock.patch(
+                "google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
+                False,
+            ),
+            mock.patch("google.cloud.spanner_v1._async.client.metrics"),
+            mock.patch(
+                "google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
+            ) as mock_exporter,
+            mock.patch(
+                "google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
+            ),
+        ):
+            MUT.Client(
+                project="my-project",
+                credentials=creds,
+                client_options={"api_endpoint": "private-psc.internal:443"},
+            )
+            mock_exporter.assert_not_called()
 
     def test_client_passes_resolved_project_credentials_and_universe_domain(self):
         import os
diff --git a/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py b/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
index b3df9b5514e..11111111111 100644
--- a/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
+++ b/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
@@ -641,6 +641,59 @@ class TestMetricsExporter(unittest.TestCase):
             _, kwargs = mock_create_channel.call_args
             self.assertIs(kwargs["credentials"], monitoring_creds)
 
+        def test_init_preserves_unscoped_and_monitoring_scoped_credentials(self):
+            """Verify unscoped and monitoring-scoped credentials are not re-scoped."""
+
+            class _FakeScopedCredentials(Credentials, Scoped):
+                def __init__(self, scopes, requires_scopes=False):
+                    super().__init__()
+                    self._scopes = scopes
+                    self._requires_scopes = requires_scopes
+
+                @property
+                def requires_scopes(self):
+                    return self._requires_scopes
+
+                @property
+                def scopes(self):
+                    return self._scopes
+
+                def with_scopes(self, scopes, default_scopes=None):
+                    return _FakeScopedCredentials(tuple(scopes), requires_scopes=False)
+
+                def refresh(self, request):
+                    pass
+
+            unscoped_creds = _FakeScopedCredentials(scopes=None, requires_scopes=True)
+            monitoring_write_creds = _FakeScopedCredentials(
+                scopes=("https://www.googleapis.com/auth/monitoring.write",),
+                requires_scopes=False,
+            )
+            monitoring_rw_creds = _FakeScopedCredentials(
+                scopes=("https://www.googleapis.com/auth/monitoring",),
+                requires_scopes=False,
+            )
+
+            for creds in (
+                unscoped_creds,
+                monitoring_write_creds,
+                monitoring_rw_creds,
+            ):
+                with patch(
+                    "google.cloud.spanner_v1.metrics.metrics_exporter."
+                    "MetricServiceGrpcTransport.create_channel",
+                    return_value=MagicMock(),
+                ) as mock_create_channel:
+                    CloudMonitoringMetricsExporter(
+                        project_id=PROJECT_ID,
+                        credentials=creds,
+                    )
+
+                _, kwargs = mock_create_channel.call_args
+                if creds.requires_scopes:
+                    self.assertEqual(
+                        kwargs["credentials"].scopes,
+                        (
+                            "https://www.googleapis.com/auth/cloud-platform",
+                            "https://www.googleapis.com/auth/monitoring",
+                            "https://www.googleapis.com/auth/monitoring.write",
+                        ),
+                    )
+                else:
+                    self.assertIs(kwargs["credentials"], creds)
+
         def test_init_uses_credentials_project_id_without_adc(self):
             """Verify explicit credentials project_id is used without falling back to ADC."""
             creds = AnonymousCredentials()
@@ -675,7 +728,9 @@ class TestMetricsExporter(unittest.TestCase):
                     return_value=MagicMock(),
                 ),
             ):
-                with self.assertRaises(ValueError):
+                with self.assertRaisesRegex(
+                    ValueError, "project_id could not be determined."
+                ):
                     CloudMonitoringMetricsExporter(
                         project_id=None,
                         credentials=creds,
@@ -703,7 +758,9 @@ class TestMetricsExporter(unittest.TestCase):
                 "google.cloud.spanner_v1.metrics.metrics_exporter.google.auth.default",
                 return_value=(AnonymousCredentials(), None),
             ):
-                with self.assertRaises(ValueError):
+                with self.assertRaisesRegex(
+                    ValueError, "project_id could not be determined."
+                ):
                     CloudMonitoringMetricsExporter(
                         project_id=None,
                         client=mock_client,

After applying, please run ruff check --select I --fix and ruff format on the modified test files and re-run pytest.

@sakthivelmanii sakthivelmanii added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 22, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants