Repository navigation
fix(spanner): respect universe_domain and client_options in metrics exporter - #19172
sakthivelmanii wants to merge 6 commits into
Conversation
…xporter ## Problem - `CloudMonitoringMetricsExporter` always connected to `monitoring.googleapis.com`, ignoring `client_options.universe_domain` and `GOOGLE_CLOUD_UNIVERSE_DOMAIN`. - Clients with a custom `client_options.api_endpoint` still exported built-in metrics to `monitoring.googleapis.com`. - `Client.__init__` passed raw `project` and `credentials` instead of resolved `self.project` and `self._credentials`, causing fallback to ambient ADC. ## Solution - Pass resolved `self.project`, `self._credentials`, and filtered monitoring `client_options` to `_initialize_metrics` and `CloudMonitoringMetricsExporter`. - Skip built-in metrics export when a custom non-default `api_endpoint` is configured. - Configure `MetricServiceClient` via a transport callable so `universe_domain` and credentials validation work natively, re-scope Spanner-only scoped credentials to `monitoring.write`, and avoid ADC project fallback when explicit credentials are provided. - Add unit tests for both sync and async clients and the metrics exporter.
There was a problem hiding this comment.
Code Review
This pull request updates both the synchronous and asynchronous Spanner clients to propagate client_options (such as universe_domain and api_endpoint) to the built-in metrics exporter, skipping metrics export when a custom endpoint is configured. Feedback on the changes highlights that google.api_core.client_options is not explicitly imported in either client file, which could lead to a NameError. Additionally, it is recommended to use getattr when accessing attributes from the caller-supplied client_options object to prevent potential attribute errors.
olavloite
left a comment
There was a problem hiding this comment.
LGTM, assuming that the choice to disable monitoring when any custom endpoint has been set is intentional.
| not _get_spanner_emulator_host() | ||
| and not emulator_host | ||
| and not isinstance(credentials, AnonymousCredentials) | ||
| and not has_custom_endpoint |
There was a problem hiding this comment.
This disables monitoring export for any custom endpoint (e.g. also for something like spanner.us-central1.rep.googleapis.com). Is that intentional?
There was a problem hiding this comment.
Yes, skipping by default is intentional so custom/PSC/regional endpoints do not silently egress credentials and metrics to global monitoring.googleapis.com. Added SPANNER_MONITORING_HOST support (matching Go/Java) so callers using a custom api_endpoint can explicitly opt in to a monitoring endpoint.
olavloite
left a comment
There was a problem hiding this comment.
LGTM, a few requests on test coverage and async/sync test parity:
tests/unit/_async/test_client_extra.pyis missing async counterparts for two tests added intests/unit/test_client.py(test_initialize_metrics_universe_propagation_and_option_filteringandtest_client_custom_api_endpoint_skips_metrics_exporter), andtest_initialize_metrics_custom_api_endpoint_skips_exporteronly passes aClientOptionsinstance, leaving thefrom_dictbranch ingoogle/cloud/spanner_v1/_async/client.py(_initialize_metrics) uncovered.tests/unit/test_metrics_exporter.pydoesn't yet testrequires_scopes=Trueormonitoring/monitoring.writescoped credentials, and uses bareassertRaises(ValueError)without checking the error message.
Suggested patch:
diff --git a/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py b/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
index f8b648f2e0e..11111111111 100644
--- a/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
+++ b/packages/google-cloud-spanner/tests/unit/_async/test_client_extra.py
@@ -281,36 +281,121 @@ class TestClientExtra(unittest.IsolatedAsyncioTestCase):
from google.cloud.spanner_v1._async import client as MUT
+ custom_endpoints = [
+ {"api_endpoint": "private-spanner.internal:443"},
+ ClientOptions(api_endpoint="private-psc.internal:443"),
+ ClientOptions(api_endpoint="spanner.us-central1.rep.googleapis.com"),
+ ClientOptions(
+ api_endpoint="spanner.googleapis.com",
+ universe_domain="my-universe.com",
+ ),
+ ClientOptions(api_endpoint="spanner.googleapis.com@evil.example"),
+ ]
+ for client_options in custom_endpoints:
+ with (
+ mock.patch(
+ "google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
+ False,
+ ),
+ mock.patch(
+ "google.cloud.spanner_v1._async.client._get_spanner_emulator_host",
+ return_value=None,
+ ),
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
+ ) as mock_exporter,
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
+ ) as mock_set_provider,
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
+ ),
+ ):
+ MUT._initialize_metrics(
+ "project",
+ self.credentials,
+ client_options=client_options,
+ )
+ mock_exporter.assert_not_called()
+ mock_set_provider.assert_called_once()
+ self.assertIsInstance(
+ mock_set_provider.call_args[0][0],
+ MUT.metrics.NoOpMeterProvider,
+ )
+
+ def test_initialize_metrics_universe_propagation_and_option_filtering(self):
+ from google.api_core.client_options import ClientOptions
+
+ from google.cloud.spanner_v1._async import client as MUT
+
+ creds = mock.Mock()
+ client_options = ClientOptions(
+ universe_domain="my-universe.com",
+ quota_project_id="quota-proj",
+ scopes=["https://www.googleapis.com/auth/spanner.admin"],
+ api_audience="https://spanner.my-universe.com",
+ api_key="fake-api-key",
+ )
with (
mock.patch(
"google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
False,
),
mock.patch(
"google.cloud.spanner_v1._async.client._get_spanner_emulator_host",
return_value=None,
),
mock.patch(
"google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
) as mock_exporter,
mock.patch(
- "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
- ) as mock_set_provider,
+ "google.cloud.spanner_v1._async.client.PeriodicExportingMetricReader"
+ ),
+ mock.patch("google.cloud.spanner_v1._async.client.MeterProvider"),
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.metrics.set_meter_provider"
+ ),
mock.patch(
"google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
),
):
MUT._initialize_metrics(
- "project",
- self.credentials,
- client_options=ClientOptions(api_endpoint="private-psc.internal:443"),
+ "my-project",
+ creds,
+ client_options=client_options,
)
- mock_exporter.assert_not_called()
- mock_set_provider.assert_called_once()
- self.assertIsInstance(
- mock_set_provider.call_args[0][0],
- MUT.metrics.NoOpMeterProvider,
- )
+
+ mock_exporter.assert_called_once()
+ _, kwargs = mock_exporter.call_args
+ self.assertEqual(kwargs["project_id"], "my-project")
+ self.assertIs(kwargs["credentials"], creds)
+ monitoring_opts = kwargs["client_options"]
+ self.assertIsInstance(monitoring_opts, ClientOptions)
+ self.assertEqual(monitoring_opts.universe_domain, "my-universe.com")
+ self.assertIsNone(monitoring_opts.api_endpoint)
+ self.assertIsNone(monitoring_opts.scopes)
+ self.assertIsNone(monitoring_opts.api_audience)
+ self.assertIsNone(monitoring_opts.quota_project_id)
+ self.assertIsNone(monitoring_opts.api_key)
+
+ def test_client_custom_api_endpoint_skips_metrics_exporter(self):
+ import os
+
+ from google.cloud.spanner_v1._async import client as MUT
+
+ creds = build_scoped_credentials()
+ with (
+ mock.patch.dict(os.environ, {"SPANNER_DISABLE_BUILTIN_METRICS": "false"}),
+ mock.patch(
+ "google.cloud.spanner_v1._async.client._metrics_monitor_initialized",
+ False,
+ ),
+ mock.patch("google.cloud.spanner_v1._async.client.metrics"),
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.CloudMonitoringMetricsExporter"
+ ) as mock_exporter,
+ mock.patch(
+ "google.cloud.spanner_v1._async.client.SpannerMetricsTracerFactory"
+ ),
+ ):
+ MUT.Client(
+ project="my-project",
+ credentials=creds,
+ client_options={"api_endpoint": "private-psc.internal:443"},
+ )
+ mock_exporter.assert_not_called()
def test_client_passes_resolved_project_credentials_and_universe_domain(self):
import os
diff --git a/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py b/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
index b3df9b5514e..11111111111 100644
--- a/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
+++ b/packages/google-cloud-spanner/tests/unit/test_metrics_exporter.py
@@ -641,6 +641,59 @@ class TestMetricsExporter(unittest.TestCase):
_, kwargs = mock_create_channel.call_args
self.assertIs(kwargs["credentials"], monitoring_creds)
+ def test_init_preserves_unscoped_and_monitoring_scoped_credentials(self):
+ """Verify unscoped and monitoring-scoped credentials are not re-scoped."""
+
+ class _FakeScopedCredentials(Credentials, Scoped):
+ def __init__(self, scopes, requires_scopes=False):
+ super().__init__()
+ self._scopes = scopes
+ self._requires_scopes = requires_scopes
+
+ @property
+ def requires_scopes(self):
+ return self._requires_scopes
+
+ @property
+ def scopes(self):
+ return self._scopes
+
+ def with_scopes(self, scopes, default_scopes=None):
+ return _FakeScopedCredentials(tuple(scopes), requires_scopes=False)
+
+ def refresh(self, request):
+ pass
+
+ unscoped_creds = _FakeScopedCredentials(scopes=None, requires_scopes=True)
+ monitoring_write_creds = _FakeScopedCredentials(
+ scopes=("https://www.googleapis.com/auth/monitoring.write",),
+ requires_scopes=False,
+ )
+ monitoring_rw_creds = _FakeScopedCredentials(
+ scopes=("https://www.googleapis.com/auth/monitoring",),
+ requires_scopes=False,
+ )
+
+ for creds in (
+ unscoped_creds,
+ monitoring_write_creds,
+ monitoring_rw_creds,
+ ):
+ with patch(
+ "google.cloud.spanner_v1.metrics.metrics_exporter."
+ "MetricServiceGrpcTransport.create_channel",
+ return_value=MagicMock(),
+ ) as mock_create_channel:
+ CloudMonitoringMetricsExporter(
+ project_id=PROJECT_ID,
+ credentials=creds,
+ )
+
+ _, kwargs = mock_create_channel.call_args
+ if creds.requires_scopes:
+ self.assertEqual(
+ kwargs["credentials"].scopes,
+ (
+ "https://www.googleapis.com/auth/cloud-platform",
+ "https://www.googleapis.com/auth/monitoring",
+ "https://www.googleapis.com/auth/monitoring.write",
+ ),
+ )
+ else:
+ self.assertIs(kwargs["credentials"], creds)
+
def test_init_uses_credentials_project_id_without_adc(self):
"""Verify explicit credentials project_id is used without falling back to ADC."""
creds = AnonymousCredentials()
@@ -675,7 +728,9 @@ class TestMetricsExporter(unittest.TestCase):
return_value=MagicMock(),
),
):
- with self.assertRaises(ValueError):
+ with self.assertRaisesRegex(
+ ValueError, "project_id could not be determined."
+ ):
CloudMonitoringMetricsExporter(
project_id=None,
credentials=creds,
@@ -703,7 +758,9 @@ class TestMetricsExporter(unittest.TestCase):
"google.cloud.spanner_v1.metrics.metrics_exporter.google.auth.default",
return_value=(AnonymousCredentials(), None),
):
- with self.assertRaises(ValueError):
+ with self.assertRaisesRegex(
+ ValueError, "project_id could not be determined."
+ ):
CloudMonitoringMetricsExporter(
project_id=None,
client=mock_client,After applying, please run ruff check --select I --fix and ruff format on the modified test files and re-run pytest.
Problem
CloudMonitoringMetricsExporteralways connected tomonitoring.googleapis.com, ignoringclient_options.universe_domainandGOOGLE_CLOUD_UNIVERSE_DOMAIN.client_options.api_endpointstill exported built-in metrics tomonitoring.googleapis.com.Client.__init__passed rawprojectandcredentialsinstead of resolvedself.projectandself._credentials, causing fallback to ambient ADC.Solution
self.project,self._credentials, and filtered monitoringclient_optionsto_initialize_metricsandCloudMonitoringMetricsExporter.api_endpointis configured.MetricServiceClientvia a transport callable souniverse_domainand credentials validation work natively, re-scope Spanner-only scoped credentials tomonitoring.write, and avoid ADC project fallback when explicit credentials are provided.